DEV Community: Ryan Bethel

OAuth Authentication

Ryan Bethel — Fri, 16 Jun 2023 19:42:01 +0000

More and more of our lives take place online. With more services come more passwords and usernames to keep track of. OAuth lets users authenticate to a new service using a previously established identity on another service. This final post in our Authentication series shows how to use OAuth with Enhance apps to authenticate new users.

Authentication can be intimidating. Not necessarily because it is hard to understand, but more because it feels like the consequences of making a mistake are high. This is one of the reasons that people choose OAuth. It allows you to rely on another service (i.e., GitHub, Google, etc.) to establish a user's identity and store the relevant information for their account. While that is true and beneficial, it is important to understand how this process works so that your interface with the 3rd party provider, and the portion of user details transferred to your app, remain secure.

TL;DR

OAuth is a specification that allows a user to establish their identity for one site by logging in on a third-party site. That third-party provider is trusted to vouch for the user while only providing the account details that the user approves to be shared. The rest of this post and sample code uses GitHub as the example third-party provider. If a user visits the site example.com to create a new account. They are saying “You don’t know me, but GitHub does and they will vouch for me.” Example.com then redirects the user to GitHub where they are asked to log in to prove their identity. After that GitHub sends the user back to example.com with a code that can be used to verify successful login to GitHub. The example.com server can then send this code/token to GitHub and receive verification that that is a legitimate GitHub account. Example.com will then use the GitHub username to identify the new account.

GitHub OAuth Keys

Using OAuth requires setting up the 3rd party provider to work with your app. For GitHub, you need to do the following:

Sign in to GitHub
Navigate to Settings: Click on your profile photo in the top right corner and select 'Settings' from the dropdown menu.
Go to Developer Settings: Scroll down the user settings sidebar and click on 'Developer settings'.
Create a New OAuth App: Click on 'OAuth Apps' in the left sidebar and then click on the 'New OAuth App' button.
Fill in the App Details:
- 'Authorization callback URL': The URL where users are sent after they authorize with GitHub. It can be the main URL for the whole application.
Register the App
Copy the Client ID and Client Secret into environment variables

OAUTH_CLIENT_ID=key
OAUTH_CLIENT_SECRET=secret
OAUTH_AUTHORIZE_URL=https://github.com/login/oauth/authorize
OAUTH_TOKEN_URL=https://github.com/login/oauth/access_token
OAUTH_USERINFO_URL=https://api.github.com/user

Make sure to keep the Client Secret a secret. If it is compromised anyone can impersonate your app. In addition to the API keys, create environment variables for the OAuth endpoints. The example above includes the endpoints and corresponding environment variables for GitHub.

If there are several environments for your app (i.e., staging and production) they will each need to have a separate OAuth app setup. These cannot share keys because the app configuration tells GitHub the URL it is allowed to redirect users back to. If your staging environment is located at https://staging.example.com and your production app is at https://example.com each will need separate API keys.

Local development without GitHub API keys

Local development is an important part of developing apps rapidly. Waiting for the deployment process can take minutes. This will drastically slow you down if you deploy your app to test every minor change. It is possible and often desirable to set up full OAuth even for your testing environment to reduce the difference between these environments. But to start with, it is nice to skip the GitHub dashboard clicking and get right to building. If no environment variables and Keys for OAuth are provided, the authentication code uses a mock endpoint to simulate the 3rd party provider. This mock endpoint uses /_mock-oauth. This code is not critical to the authentication. It can be removed. It is important to note that this mock behavior provides no real security and should not be substituted for a trusted 3rd party provider.

GitHub Login Button

The first step to logging in with OAuth is to set up a link to serve to the user that will initiate the authentication.
The function below will generate the necessary login URL.

const isLocal = process.env.ARC_ENV === 'testing'
const useMock = !process.env.OAUTH_CLIENT_ID || !process.env.OAUTH_CLIENT_SECRET
const domain = isLocal ? process.env.DOMAIN_NAME || 'http://localhost:3333' : process.env.DOMAIN_NAME
let urls
if (isLocal && useMock) {
  urls = {
    authorizeUrl: `${domain}/_mock-oauth/login`,
    redirectUrl: `${domain}/oauth`,
  }
} else {
  urls = {
    authorizeUrl: process.env.OAUTH_AUTHORIZE_URL,
    redirectUrl: `${domain}/oauth`,
  }
}
export default function loginHref({ redirectAfterAuth = '/', newRegistration = '' }) {

  let oauthState = { redirectAfterAuth }
  if (newRegistration) oauthState = { redirectAfterAuth, newRegistration };

  const redirectUrlPart = urls.redirectUrl
    ? `&redirect_uri=${encodeURIComponent(urls.redirectUrl)}`
    : ''
  return `${urls.authorizeUrl}?client_id=${process.env.OAUTH_CLIENT_ID
  }${redirectUrlPart}${redirectAfterAuth
    ? `&state=${encodeURIComponent(
      JSON.stringify(oauthState)
    )}`
    : ''
  }`
}

It uses the DOMAIN_NAME and OAUTH environment variables to create the required URL. It also points authentication to the mock routes if the app is local and not configured with a provider. The function can be imported in any API handler route, as shown below, to create this “Log in with GitHub” URL.

import loginHref from '../../auth-shared/login-href.mjs'

OAuth log in

When login is initiated using the link to GitHub, the flow falls under the control of GitHub. If that GitHub login is successful, GitHub returns a code token to the user and redirects them to the server. The code below shows the handler for that endpoint. If that code is in the request as a query parameter, the server will attempt to verify it with GitHub.

import { getAccounts } from '../../../models/accounts.mjs'
import verifyOauth from '../../../auth-shared/oauth-verify-code.mjs'

export async function get(req) {
  const { redirectAfterAuth:sessionRedirectAfterAuth = '' } = req.session
  const { query: { code, state } } = req

  if (!code){
    return {
      location: '/login'
    }
  }

  if (code) {
    let redirectAfterAuth
    try {
      if (state) {
        const parseState = JSON.parse(state)
        redirectAfterAuth = parseState.redirectAfterAuth
      }
    // eslint-disable-next-line no-empty
    } catch (e) { }
    const redirect = redirectAfterAuth || sessionRedirectAfterAuth || '/'
    try {
      const oauthAccount = await verifyOauth(code)
      if (!oauthAccount.oauth.github) throw Error('user not found')
      const accounts = await getAccounts()
      const appUser = accounts.find(a =>
        a.provider?.github?.login === oauthAccount?.oauth?.github?.login &&
        a.authConfig?.loginWith?.github
      )
      const { password: removePassword, ...sanitizedAccount } = appUser ||
        {}
      const accountVerified = appUser?.verified?.phone ||
        appUser?.verified?.email

      if (!appUser) {
        return {
          location: '/login'
        }
      }
      if (!accountVerified){
        return {
          session: {
            redirectAfterAuth: redirect,
            unverified:sanitizedAccount
          },
          location: '/verify'
        }
      }
      return {
        session:{authorized:sanitizedAccount},
        location: redirect
      }

    } catch (err) {
      return {
        statusCode: err.code,
        body: err.message
      }
    }

  }
}

The function to verify the code token on GitHub is shown below. It first makes a POST request to exchange that code for a longer lived token. The server then uses that token to make a GET request for the profile information that the user has approved. In this example, only the login property is kept from the profile to identify the user account on your website.

import tiny from 'tiny-json-http'
const isLocal = process.env.ARC_ENV === 'testing'
const useMock = !process.env.OAUTH_CLIENT_ID || !process.env.OAUTH_CLIENT_SECRET
const domain = isLocal ? process.env.DOMAIN_NAME || 'http://localhost:3333' : process.env.DOMAIN_NAME
let urls
if (isLocal && useMock) {
  urls = {
    authorizeUrl: `${domain}/_mock-oauth/login`,
    redirectUrl: `${domain}/oauth`,
    tokenUrl: `${domain}/_mock-oauth/token`,
    userInfoUrl: `${domain}/_mock-oauth/user`,
  }
} else {
  urls = {
    authorizeUrl: process.env.OAUTH_AUTHORIZE_URL,
    redirectUrl: `${domain}/oauth`,
    tokenUrl: process.env.OAUTH_TOKEN_URL,
    userInfoUrl: process.env.OAUTH_USERINFO_URL,
  }
}

export default async function oauth(code) {
  const data = {
    code,
    client_id: process.env.OAUTH_CLIENT_ID || '',
    client_secret: process.env.OAUTH_CLIENT_SECRET || '',
    redirect_uri: urls.redirectUrl,
  }
  let result = await tiny.post({
    url: urls.tokenUrl,
    headers: { Accept: 'application/json' },
    data
  })
  let token = result.body.access_token
  let userResult = await tiny.get({
    url: urls.userInfoUrl,
    headers: {
      Authorization: `token ${token}`,
      Accept: 'application/json'
    }
  })

  const {login, ...rest} = userResult.body
  return {
    oauth: { github: {login} }
  }
}

Registration

New users registering with OAuth do not need to provide a username and password. In the example app they provide a display name and an email address or phone number as a means of resetting their account.

Other OAuth providers

OAuth is an industry-standard specification. Most of this example code will work for any provider. There are however slight variations in the token exchange where the code is verified in exchange for a token to access user profiles. Tests are especially important when reimplementing this code with other providers.

Try it out

The code for the full example can be found at github.com/enhance-dev/enhance-auth. Try out the deployed example using GitHub OAuth.

Next Step

This is the final post in the authentication series. The full example shows a complete solution that can be modified for most websites. Looking ahead there are a few features that would be nice to add. Passkeys are a popular topic with browser vendors promoting them as the password killer of the future. We will add support for Passkeys when they become widely supported. In the meantime, Pull Requests are welcome at github.com/enhance-dev/enhance-auth.

Authentication with Magic links

Ryan Bethel — Wed, 14 Jun 2023 13:26:29 +0000

Websites should be secure. They should also be easy to use. These requirements seem at odds with many development decisions. Magic links are a popular authentication method that uses email (or phone) links to log in directly to a site. They are a good compromise between security and ease of use for most applications. This fourth installment of the series shows how to use magic links with Enhance and discusses some of the tradeoffs.

TL;DR

All of the pieces are already in place to log in with magic links. The previous posts covered registering a new user and verifying email address or phone number by sending links or codes. All that remains is to:

Add a login form that asks for just the email or phone number to log in.
Send a unique link to that email or a code to that phone number using the same process as verification.
Once the link is clicked or the code is entered, the session is updated and the user is logged in.

Magic Link Login

One of the best features of this login is that it requires only a single input field.

After clicking on the link received by email, the server verifies the link. If it is valid, the session is updated to log the user in. They will then immediately see the following screen confirming they are authenticated.
That is it.

The code for this login handler is below. This endpoint serves the form for entering an email address with an ordinary GET request. A POST request with that email will trigger a link to be sent by email. That link will return here with a token as a query parameter which can be verified. If that token is valid, the associated account is added to the session, and the user is logged in.

// /app/api/login/magic-link.mjs
import sendLink from '../../auth-shared/send-email-link.mjs'
import db from '@begin/data'
import { getAccounts } from '../../models/accounts.mjs'

export async function get(req) {
  const token = req.query?.token
  const { problems, login, ...newSession } = req.session
  if (problems) {
    return {
      session: newSession,
      json: { problems, login }
    }
  }

  if (!token) return

  if (token){
    const verifySession = await db.get({ table: 'session', key: token })
    const {linkUsed}=verifySession
    const linkExpired = verifySession?.ttl < Date.now()
    if (!verifySession || linkUsed || linkExpired) { 
      return { location:'/login/link-expired' } 
    }

    await db.set({
      ...verifySession, 
      table: 'session', 
      key: token, 
      linkUsed: true  
    })

    let accounts, account
    try {
      accounts = await getAccounts()
      account = accounts.find(i => i.email === verifySession.email && 
        i.authConfig?.loginWith?.email)
    }
    catch (e) {
      console.log(e)
    }

    if (!account) { return {location:'/login/magic-link'} }

    const accountVerified = account?.verified?.email
    const { password: removePassword, ...sanitizedAccount } = account

    if (!accountVerified) {
      return {
        session: {
          redirectAfterAuth:verifySession?.redirectAfterAuth || '/', 
          unverified: { ...sanitizedAccount } 
        },
        location: '/verify'
      }
    }
    return {
      session: { authorized: { ...sanitizedAccount } },
      location: verifySession?.redirectAfterAuth || '/'
    }
  }

}

export async function post(req) {
  const session = req?.session
  const {email} = req.body
  const { redirectAfterAuth = '/' } = session

  if (!email){
    return {
      session: {...session, problems:{form:'No Email Address'}},
      location: '/login/magic-link'
    }
  } 

  if (email) {
    const accounts = await getAccounts()
    const account = accounts.find(a => a.email === email && 
      a.verified?.email && 
      a.authConfig?.loginWith?.email)
    if (account) { 
      await sendLink({ email, subject:'Enhance Auth Login Link',
        linkPath:'/login/magic-link', redirectAfterAuth })
    }
    return {
      location: '/login/wait-link'
    }
  } 
}

Magic code/SMS

As discussed in the previous post, SMS messages are not as secure as email addresses. For this reason, allowing users to log in with just a phone number is also less secure. There are situations where this might be a reasonable tradeoff. It may be fine, for example, as a login to a survey to ensure that each user only submits one response.
The form below is the log in by phone example.

In this case, a one-time code is sent to the phone number and a new form is presented in the browser for entering that code. After the one-time code is submitted, the user is logged in and receives the same confirmation message as in the magic link.

Email only streamlined registration

One advantage of the magic link is a more streamlined login experience. In some cases, you might want to streamline registration also. An email (or phone number) might be the only piece of information needed from the user.
There are two options for this. First is to remove everything from the registration form except for the email or phone number. Another option is to drop the registration altogether. If a new email is submitted for login, it can be registered as a new user then.

Try it out

The code for the full example can be found at github.com/enhance-dev/enhance-auth. Try out the deployed example using magic link login.

Next Step

The next post in this series will cover how to do social login. It is another popular option that can streamline logging in down to a single click.

Verify email and phone for new accounts

Ryan Bethel — Wed, 14 Jun 2023 13:15:50 +0000

Having a verified way to communicate with account owners is critical for most applications. It is the best way to recover account access if a password is lost or forgotten. In our previous authentication posts, we covered Why you should roll your own auth and Authentication for a username and password flow. This third installment of the series focuses on account verification with an email or phone number.

TL;DR

An email or phone number is given at registration.
Using SendGrid, an email is sent with a verification link.
Using Twilio, a one time code is sent as an SMS message.
The account is verified as soon as one of those links/codes is used.
At login, if the user forgets their password, they can click “forgot password”.
They will receive a password reset link to recover the account.

Now, let's deep-dive into each of these components.

Lego not Ikea

Think of this authentication series as more like Lego, less like Ikea. Ikea furniture has detailed instructions for only one outcome, Lego gives flexibility to choose the parts you want for different outcomes. Although we include both phone and email options, often, email is enough. SMS has some vulnerabilities that make it less secure than email. These are building blocks; use only what you need.

Twilio and SendGrid

Software projects require constant build versus buy decisions. This series started by making the case that you should build your own authentication rather than buy it from a third party. But here, we use SendGrid and Twilio for email and phone messages. Email deliverability is a significant challenge. The likelihood of your messages being seen by users is much higher with a reputable email service. And the only practical way to send SMS messages is to connect to the network through a third party.

Both these services have a limited free tier so you can try them more easily. Note that for Twilio this demo uses their Verify API. It is built for this purpose and it handles the logic of sending and verifying the one time codes.

Your mileage may vary. But in general, the cost and complexity of adding third party dependencies only grows with time, so it is still best to avoid them when possible.

Environment variables

Configuring the services requires five environment variables. For SendGrid, an API key is needed as well as the email address that will be used as sender for messages. Twilio has an API token and an account service ID. The SMS messages come from a Twilio verified number. The test phone number is where the SMS messages will be sent during local development. That means you can register accounts with fake phone numbers and the SMS code will all be sent to this phone number.

# /.env
SENDGRID_API_KEY=key
TRANSACTION_SEND_EMAIL=me@example.com
TWILIO_API_ACCOUNT_SID=key
TWILIO_API_TOKEN=key
SMS_TEST_PHONE=+15555551111

Local development without API keys

Local development is valuable for testing business logic and iterating quickly. This demo is written so that the code functions in local dev even with no API keys. Without them, the app will print the email validation link and SMS one time codes to the console. This makes it easier to add authentication to an app during the earliest stages of development. Then when the app is ready to be deployed, real API keys can be added.

Email verification

When a user registers with an email address, a unique verification link is sent. When the link is clicked a success message confirms that their account is verified. The database entry for that account is updated to reflect that the account has been verified.

The handler below is on the /verify route. After submitting registration, the unverified user info is added to the session and redirected here. This endpoint initiates the verification of email, phone, or both.

// /app/api/verify/index.mjs
import {sendCode} from '../../auth-shared/sms-code-verify.mjs'
import sendLink from '../../auth-shared/send-email-link.mjs'

export async function get(req){
  const session = req.session
  const { unverified, authorized, redirectAfterAuth = '/', ...restSession } = session
  let phone,email,phoneVerified,emailVerified
  if (unverified) {
    phone = unverified.phone
    email = unverified.email
    phoneVerified = unverified.verified?.phone
    emailVerified = unverified.verified?.email
    const verifyPhone = phone && !phoneVerified
    const verifyEmail = email && !emailVerified
    if (verifyEmail){
      // send verification link
      await sendLink({
        email,
        redirectAfterAuth,
        subject:'Enhance Auth Verify Email Link',
        linkPath:'/verify/email'
      })
      if (!verifyPhone){
        return {
          location: '/verify/waiting-email'
        }
      }
    }
    if (verifyPhone){
      // send SMS code
      const { smsVerify, unverified, authorized} = req.session

      const serviceSid = await sendCode({
        phone,
        friendlyName:'Enhance Auth Verify Phone'
      })

      const newSession = { ...req.session }
      newSession.smsVerify = {otp:{ serviceSid }}

      return {
        session: newSession,
        location: '/verify/phone'
      }
    }
  } else if (authorized) {
    phone = authorized.phone
    email = authorized.email
    phoneVerified = authorized.verified?.phone
    emailVerified = authorized.verified?.email
    return {
      json:{
        verifyPhone:phone && !phoneVerified,
        verifyEmail:email && !emailVerified
      }
    }
  }
}

The function below sends the verification link using the SendGrid API. It sets a reference to the token being sent with some meta data so that when the link is clicked the server can continue verifying the account. A key difference with the email link verification vs. the phone is that when the email verification link is clicked, it opens in a new browser window, and therefore starts with a new session.

// /app/auth-shared/send-email-link.mjs
import crypto from 'crypto'
import db from '@begin/data'
import sgMail from '@sendgrid/mail'

export default async function sendLink({ email, subject='', text, html, linkPath, redirectAfterAuth = '/', newRegistration = false }){
  const isLocal = process.env.ARC_ENV === 'testing'
  const requiredEnvs = process.env.TRANSACTION_SEND_EMAIL && process.env.SENDGRID_API_KEY
  const domain = process.env.DOMAIN_NAME || 'http://localhost:3333'
  const verifyToken = crypto.randomBytes(32).toString('base64')
  const link = `${domain}${linkPath}?token=${encodeURIComponent(verifyToken)}`

  const ttl = Date.now() + 60*60*1000
  await db.set({ table: 'session', key: verifyToken, verifyToken,
    email, redirectAfterAuth, newRegistration, linkUsed:false, ttl})


  // Local Development Testing Setup
  if (isLocal) {
    console.log(`${subject}: ${link}`)
  }

  if (requiredEnvs) {
    sgMail.setApiKey(process.env.SENDGRID_API_KEY)
    let toEmail = email
    if(isLocal) toEmail = process.env.TRANSACTION_SEND_EMAIL;
    const msg = {
      to: toEmail,
      from: `${process.env.TRANSACTION_SEND_EMAIL}`,
      subject,
    }

    if (text) {
      msg.text = text(link)
    } else if (html) {
      msg.html = html(link)
    } else {
      msg.text = `${subject}: ${link}`
    }

    try {
      await sgMail.send(msg)
    } catch (e) {
      console.error(e)
      console.error(e.response.body.errors)
    }
  } else {
    console.log('TRANSACTION_SEND_EMAIL and SENDGRID_API_KEY needed to send')
  }

  return verifyToken
}

Once email verification is initiated, the server redirects to the /verify/email route. The handler for that route is below. It checks the status of the verification and shows a success or waiting message.

// /app/verify/email.mjs
import db from '@begin/data'
import { getAccounts, upsertAccount } from '../../models/accounts.mjs'

/**
 * @type {import('@enhance/types').EnhanceApiFn}
 */
export async function get(req) {
  const token = req.query?.token
  const { authorized, unverified, ...newSession } = req.session

  if (token) {
    const verifySession = await db.get({ table: 'session', key: token })
    const { linkUsed } = verifySession
    const linkExpired = verifySession?.ttl < Date.now()

    if (!verifySession || linkUsed || linkExpired) return  { location: '/verify/expired' };

    await db.set({ ...verifySession, table: 'session', key: token, linkUsed: true })
    let accounts = await getAccounts()
    let account = accounts.find(acct => verifySession.email === acct.email)

    if (!account) return { location: '/login' };

    account.verified = account.verified ? {...account.verified , email:true} : {email:true}
    account = await upsertAccount({ ...account })
    return {
      session: {},
      location: '/verify/success-email'
    }

  } else if (unverified) {
    let accounts = await getAccounts()
    let account = accounts.find(acct => unverified.email === acct.email )
    const accountVerified = account.verified?.email

    if (!account) return {location: '/login'};
    if (accountVerified) {
      return {
        session: {},
        location: '/verify/success-email'
      }
    }

    const { redirectAfterAuth = '/' } = req.session

    await sendLink({ email: account.email, subject:'Enhance Auth Verify Email Link', linkPath:'/verify/email', redirectAfterAuth })

    return {
      session: {},
      location: '/verify/waiting-email'
    }

  } else if (authorized) {
    if (authorized.verified?.email===true){
      return {
        location: '/verify/success-email'
      }
    } else {
      return { location: '/verify/success-email' }
    }

  } else {
    return {
      location: '/login'
    }
  }
}

Phone verification

When an account is registered with a phone number, the unverified account info is added to the session and the browser is redirected to /verify. A one time code is sent to the phone number, and the user is then prompted to enter this code.

The function below uses the Twilio Verify API to send the one time code. Note that if you are in a local development environment and the environment variables are not present, the API is bypassed and a code is sent to the console.

// /app/auth-shared/sms-code-verify.mjs
import twilio from "twilio"
const accountSid = process.env.TWILIO_API_ACCOUNT_SID
const authToken = process.env.TWILIO_API_TOKEN
const isLocal = process.env.ARC_ENV === 'testing'
const requiredEnvs = (process.env.TWILIO_API_ACCOUNT_SID && process.env.TWILIO_API_TOKEN)

export async function sendCode({phone,friendlyName}){
  let service
  if (requiredEnvs){
    const toPhone = isLocal ? process.env.SMS_TEST_PHONE : '+1'+phone.replace('-','')
    const client = twilio(accountSid, authToken)
    service = await client.verify.v2.services.create({
      friendlyName,
    });
    await client.verify.v2.services(service.sid).verifications.create({
      to: toPhone,
      channel: 'sms',
    });

    if (!process.env.SMS_TEST_PHONE) console.log('Warning: SMS messages will be sent to phone numbers unless SMS_TEST_PHONE is set');
  } else {
    console.log('Missing required environment variables')
    if (isLocal){
      console.log('Use simulated One Time Password "123456" for testing')
      service = {sid:'simulated-testing'}
    }
  }
  return service.sid
}

export async function verifyCode({phone, serviceSid, smsCode }){
  let verification, status
  if (requiredEnvs){
    const toPhone = isLocal ? process.env.SMS_TEST_PHONE : '+1'+ phone.replace('-','')
    const client = twilio(accountSid, authToken)
    verification= await client.verify.v2
      .services(serviceSid)
      .verificationChecks.create({ to: toPhone, code: smsCode })
    status = verification.status
  } else {
    console.log('Missing required environment variables')
    if (isLocal){ status = smsCode === '123456' ? 'approved' : false }
  }
  return status
}

The following is the handler for /verify/phone. This handles the forms for entering the code and success messages once verified.

// /app/api/verify/phone.mjs
import { sendCode, verifyCode } from "../../auth-shared/sms-code-verify.mjs"
import { getAccount, upsertAccount } from "../../models/accounts.mjs"

export async function get(req) {
  const { redirectAfterAuth = '/' } = req.session
  const { smsVerify, unverified, authorized } = req.session
  const {otp } = smsVerify || {}
  const phoneVerified = authorized?.verified?.phone || unverified?.verified?.phone

  if (!authorized && !unverified) {
    return {
      location: '/login'
    }
  }
  if (phoneVerified) {
    return {
      location:'/verify/success-phone'
    }
  }
  return {
    json: { otpSent: !!(otp?.serviceSid) },
  }
}

export async function post(req) {
  const { otpCode, request } = req.body
  const { smsVerify, unverified, authorized} = req.session
  const { otp } = smsVerify || {}
  const phoneVerified = authorized?.verified?.phone || unverified?.verified?.phone
  const phone= authorized?.phone || unverified?.phone

  if (request && phone) {

    const serviceSid = await  sendCode({phone, friendlyName:'Enhance Auth Verify Phone'})
    const newSession = { ...req.session }
    newSession.smsVerify = {otp:{ serviceSid }}

    return {
      session: newSession,
      location: '/verify/phone'
    }
  }
  if (otpCode) {
    const { serviceSid } = otp

    const status = await verifyCode({phone, serviceSid, smsCode:otpCode})

    if (status === 'approved') {
      let { smsVerify, unverified, authorized, ...newSession } = req.session
      let key = authorized?.key || unverified?.key
      let account = await getAccount(key)
      let match = account?.phone === phone
      let verified = account?.verified
      if (!match) {
        return {
          session: {},
          location: '/login'
        }
      }

      if (verified) {
        account.verified.phone = true
      } else {
        account.verified = {phone:true}
      }

      if (match) {
        let result = await upsertAccount(account)
        let {password:removePassword, ...newAccount} = result

        const redirectAfterAuth = req.session.redirectAfterAuth || '/'
        return {
          session: { ...newSession, authorized:newAccount },
          location: redirectAfterAuth
        }
      }
    }
  }
  return {
    location: '/verify/phone'
  }
}

Forgot Password

With a verified account, users can recover if a password is lost or forgotten. On the login screen there is a “Forgot password” link that initiates this process. After entering their phone or email address, a link or one time code is generated.

Next steps

Now that we have a fully verified account we have the option to allow users to login directly with just their email or phone number. We will cover using these magic links in the next post. To try out this code here is an example deployed on Begin. You can register an account to try resetting your password. Note that in this deployed example accounts are automatically deleted from the database after a day. The code for the full example can be found on GitHub at https://github.com/enhance-dev/enhance-auth.

Authentication for a Username and Password flow

Ryan Bethel — Tue, 30 May 2023 19:57:22 +0000

Passwords are the oldest form of authentication for shared systems. There are many alternatives now that offer advantages, but sometimes a password is still the best fit. This post explains how to set up password authentication for Enhance.

This is the second part of our series on authentication. The goal is to build a complete authentication system one piece at a time. The first post, Why you should roll your own auth, makes a case for not outsourcing authentication to a third party. Now we cover username/password authentication with secure sessions.

TL&DR

Before anyone can login an account needs to be created.

The registration form takes input for a new account.
This input is validated by a registration endpoint.
And stored as a new account in the database.

To login:

The username and password are entered in a login form.
Posted to the server login endpoint.
Checked against the database for a matching account.
If a matching account is found, the details are stored in a user session.

Now let's break down each of those components.

Secure Sessions

Secure sessions are the basis for authentication. Enhance uses HTTP only cookies for sessions because this is the most secure approach. It ensures no sensitive user information is stored on the client. The cookie is sent with each request to the server. The server can respond with the same cookie or modify it and send it that back with the response. In this way the session persists as long as the user interacts with the site, or it is cleared by the server. Refer to the Bulletproof Sessions post for more details on Enhance sessions. After a user logs in, their account name and other details are attached to the session and used later to authenticate them when they make restricted requests. This session data is securely encrypted so that it cannot be read or modified except on the server. A long secret key is stored in an environment variable ARC_APP_SECRET to secure the sessions.

Register New Accounts

Before anyone can login a new account needs to be created. This is done with a plain HTML form sending a POST to the server. Here is a deployed example. Either a phone number or email is required to verify the account and as a recovery mechanism if the password is lost or forgotten. The email and phone verification will be covered in the next blog post.

Validation

The markup for the registration form is shown below (or in the source repo on GitHub). To properly validate form data requires both client and server validation. Client validation is the fastest way to catch small errors, but some things like uniqueness must be checked on the server. Client validation should not be trusted because it can be bypassed easily. If the server-side validation fails, the problems are returned along with the previous values entered so the user can pick up where they left off and fix any errors. The details of this validation loop are explained in the Progressively enhancing form submission with web components post.

// /app/pages/register/username.mjs
export default function register({ html, state }) {
  const { problems, register } = state.store

  return html`
<enhance-page-container>
  <nav-menu></nav-menu>
  <main>
    <h1 class="mb1 font-semibold text2">Register a new Account</h1>
    <enhance-form action="/register/username" method="post">
      <div class="${problems?.form ? 'block' : 'hidden'}">
        <p>Found some problems!</p>
        <ul>${problems?.form}</ul>
      </div>
      <enhance-text-input
        label="Display Name"
        id="displayName"
        name="displayName"
        type="text"
        pattern="^[a-zA-Z0-9_\-]*$"
        maxlength="30"
        errors="${problems?.displayName?.errors}"
        value="${register?.displayName || ''}"
        required
      >
      </enhance-text-input>
      <enhance-text-input
        label="Username"
        id="username"
        name="username"
        type="text"
        pattern="^[a-zA-Z0-9_\-]*$"
        maxlength="30"
        errors="${problems?.username?.errors}"
        value="${register?.username || ''}"
        description="Numbers, letters, and underscores"
        required
      >
      </enhance-text-input>
      <enhance-text-input
        label="Password"
        id="password"
        name="password"
        type="password"
        minlength="8"
        errors="${problems?.password?.errors}"
        description="At least 8 characters"
        required
      >
      </enhance-text-input>
      <enhance-text-input
        label="Confirm Password"
        id="confirmPassword"
        name="confirmPassword"
        type="password"
        minlength="8"
        errors="${problems?.confirmPassword?.errors}" 
        required
      >
      </enhance-text-input>
      <enhance-text-input
        label="Email"
        id="email"
        name="email"
        type="email"
        value="${register?.email || ''}"
        errors="${problems?.email?.errors}"
      >
      </enhance-text-input>
      <enhance-text-input
        label="Phone Number"
        id="phone"
        name="phone"
        type="tel"
        pattern="[0-9]{3}-[0-9]{3}-[0-9]{4}"
        errors="${problems?.phone?.errors}"
        value="${register?.phone || ''}"
        description="Format: 123-456-7890"
      >
      </enhance-text-input>
      <enhance-submit-button style="float: right">
        <span slot="label">Register</span>
      </enhance-submit-button>
    </enhance-form>
  </main>
</enhance-page-container>
`}

Storing Passwords

When the form is submitted a post request is sent to the server. One of the most critical parts of password authentication is properly handling passwords. They need to be stored so they can be verified at login. But they should also be stored so that if the account data is breached, the original passwords cannot be easily discovered. The solution is to store a hashed and salted version of the password. The hash is a one-way function that is very difficult and time-consuming to reverse.
See the OWASP cheatsheet on hashing-vs-encryption for more information. For hashing and salting, bcrypt is used with ten salting passes. As an additional precaution if there are form validation problems, the password is not returned to the front end with the other form inputs. It needs to be entered again to resubmit. Below is the POST handler for creating a new account (or refer to the source repo on GitHub).


// /app/api/register/username.mjs
import bcrypt from 'bcryptjs'
import { validate } from '../../models/register-username.mjs'
import { upsertAccount } from '../../models/accounts.mjs'

export async function get(req) {
  let { problems, register, ...newSession } = req.session

  if (problems) {
    return {
      session: newSession,
      json: { problems, register }
    }
  }
}

export async function post(req) {
  const session = req?.session
  // eslint-disable-next-line no-unused-vars
  let { authorized: removedAuthorize,
    problems: removedProblems,
    register: removedRegister,
    unverified: removedUnverified,
    ...newSession } = session

  let { problems, register } = await validate.create(req)

  if (problems) {
    // eslint-disable-next-line no-unused-vars
    let { password:removedPassword, confirmPassword:removedConfirm, ...sanitizedRegister } = register
    return {
      session: { ...newSession, problems, register: sanitizedRegister },
      location: '/register/username'
    }
  }

  try {
    delete register.confirmPassword
    register.password = bcrypt.hashSync(register.password, 10)
    register.authConfig = { loginWith: {password:true, email:true, phone:true} }
    register.scopes = ['member']
    // eslint-disable-next-line no-unused-vars
    const { password: removePassword, ...newAccount } = await upsertAccount(register)

    return {
      session: { unverified: newAccount },
      location: '/verify'
    }
  }
  catch (err) {
    console.log(err)
    return {
      session: { ...newSession, error: err.message },
      location: '/register/username'
    }
  }

}

Login

The login handler is shown below (or in the source repo on GitHub). A password and username submitted as a POST request. They are then checked against the database to find a matching verified account. If one is found an authorized object is added to the session to identify the account.

// /app/api/login/username.mjs
import bcrypt from 'bcryptjs'
import xss from 'xss'
import { getAccounts } from '../../models/accounts.mjs'
// Hardcoded admin account to bootstrap accounts.
// The password is defined in environment variables
const hardcodedAdmin = {
  username: 'hardcoded',
  scopes: ['admin'],
}

export async function get(req) {

  const { problems, login, ...newSession } = req.session
  if (problems) {
    return {
      session: newSession,
      json: { problems, login }
    }
  }
}

export async function post(req) {
  const session = req?.session
  const { password, username } = req.body
  const { redirectAfterAuth = '/' } = session

  if (!password || !username) {
    return {
      session: {
        problems: { form: 'Missing Username or Password' },
        login: {username:xss(username)}, redirectAfterAuth
      },
      location: '/login/username'
    }
  }

  // Hardcoded admin account to bootstrap accounts
  // To disable remove the HARDCODED_ADMIN_PASSWORD from environment variables
  if (process.env.HARDCODED_ADMIN_PASSWORD && username === hardcodedAdmin.username
    && password === process.env.HARDCODED_ADMIN_PASSWORD) {
      return {
        session: { authorized: hardcodedAdmin },
        location: redirectAfterAuth ? redirectAfterAuth : '/'
      }
    }

  let accounts = await getAccounts()
  const account = accounts.find(a => a.username === username &&
    a.authConfig?.loginWith?.username)
  const match = account ? bcrypt.compareSync(password, account?.password) : false
  const accountVerified = match ? !!(account.verified?.email || account.verified?.phone) : false
  const { password: removePassword, ...sanitizedAccount } = account || {}
  if (accountVerified) {
    return {
      session: { authorized: sanitizedAccount },
      location: redirectAfterAuth ? redirectAfterAuth : '/'
    }
  }
  if (match && !accountVerified) {
    return {
      session: { unverified: sanitizedAccount, redirectAfterAuth },
      location: '/verify'
    }
  }
  if (!match) {
    return {
      session: {
        problems: { form: 'Incorrect Username or Password' },
        login: {username:xss(username)}
      },
      location: '/login/username'
    }
  }
}

Hard coded accounts

One problem setting up an app is creating the first account with administrative permissions. The new account registration form should only create regular accounts without admin permissions. There are many ways to do this, but one option is to hard code a user into the app to start. This is an alternative password authentication like that shown in Bulletproof Sessions. In this demo the username is hardcoded and the password is set with the environment variable HARDCODED_ADMIN_PASSWORD. This account can be used in the admin dashboard to modify permissions of other accounts as needed.

Next Step

After registering a new account the email or phone number must be verified. This will be covered in the next blog post. To see it in action here is an example deployed on Begin. You can register an account to try it out. Note that in this deployed example accounts are automatically deleted from the database after a day. The code for the full example can be found on GitHub at https://github.com/enhance-dev/enhance-auth.

My Experience Learning to Code

Ryan Bethel — Tue, 14 Mar 2023 20:33:47 +0000

Photo by Tobias Rademacher on Unsplash

About 10 years ago, I built my first site using the web. A few years after that, I made it a goal to do it professionally. Since that time, the number of developers worldwide has grown by roughly 5-10X¹. Which means most of us are new here.

Early on, I read anything I could find sharing how others learned to code. Some were helpful. Some were not. I hope this is helpful. But to be clear, this is not meant to be prescriptive or motivational. Everyone's experience is very different. There are a lot of thought pieces lately by long time industry experts about the rise and fall of frameworks etc. This is not that. As I said, I am relatively new here. These are just observations from my own experience.

TL&DR

One of the biggest challenges in learning to code on my own was choosing what to learn and how to invest my limited time and resources. Learning to code, as I experienced it, was a very non-linear process. I felt utterly stuck for long periods, followed by sudden breakthroughs where all the pieces fell into place. I chose early to invest heavily in learning React. Looking back, I would do that differently. One especially true paradox in coding is that it takes much longer to master, but less time to be productive than you think. I got stuck for a long time on the content consumption treadmill taking course after course. A better use of time is creating more real things that people use, even if they are just for yourself.

Background

I started my career as an engineer doing integrated circuit design. I did no web development, but I did use scripting languages like PERL, source control (before there was Git), and vim was my daily driver. Later that technical foundation helped accelerate my learning. But after just a few years, I burned out in that job. I made a career change away from tech completely for the next 12 years.

For most of those years, I worked on a small specialized team in a much larger organization. I was a guide leading outdoor trips like whitewater rafting. As I slowly recovered from burnout, I realized I had missed working with technology.

As our team grew, we needed some tools to manage our trips, but we had no budget and no resources to get them. So we decided to build them ourselves. Building and maintaining those internal tools became a part time job for the next couple of years. Initially it was all about making things work any way I could. This was low code, no code, just get it done. I built many small websites and single purpose web apps to serve our teams. It was fun. In my mind, I could never build a "real app", but I could make a thing people could get to with a URL to do what they needed.

I would not have called myself a developer at this point. But when I decided to make this my career, my pragmatic technology choices were suddenly clouded by asking myself how a real developer would do this. That might have been a good question if I knew the answer, but since I did not, it led me to overcomplicate things with tools I saw people talking about on Twitter.

My Goal

I had two goals in learning to code. First, to become proficient in solving problems using the web and second, to get a job as a developer. There is a big difference between those two goals. They are often completely at odds.

Hiring in this industry is broken.

Interviews are broken.

If you are here to find my advice on how to get a job, I am sorry. I wish I had answers, but I don't. And I am not the right person to write that post.

If, on the other hand, your goal is to build useful things, I can point out some mistakes I made that will hopefully save you time. The biggest mistake was not resisting the urge to follow what is popular. This is hard to do because it feels like a shortcut to credibility, but these fads die as fast as they are born.

A podcast host (who I greatly respect still and will not name) recommended people learn Gatsby and GraphQL to advance their careers. I followed that advice. Recently, he was dumping on both of those tools on that same podcast. Few things feel as bad as spending months or years to learn Gatsby only to watch it become a punchline. That is the danger of chasing the hype train. To be fair, this is complicated. I do think that some of my early job opportunities came from having React, GraphQL, and Gatsby on my resume.

Finding time

So I decided to get serious about becoming a developer. But I had a full-time job and a family. Bootcamp was not realistic, so I had to find a different way. My solution was to wake up early. Before my family was up and before work started, for about two years (give or take), I woke up around 4 am to spend about two hours learning before my normal day started. I do not endorse this. I was sleep deprived and unhealthy because of it. If you are trying to make space and time for a career change, don't neglect your mental health, physical health, or family. If you do you might reach your goal and find it is not all you hoped it would be.

Another way I supplemented learning was by consuming all the web dev content I could find. I didn't care what was popular, but I did take what was popular as a proxy for what must be good. I had no other sources. I didn't even know anybody in the industry personally. Twitter, podcasts, and blogs were what I had. So I immersed myself in the web dev communities there. Watching, listening, occasional DM's, but rarely making any noise myself. I listened to a steady diet of content while driving, washing dishes, mowing the lawn, etc. Like learning a foreign language by watching films without subtitles. It is not the fastest way, but it is one way to make use of unfocused time. If I heard something like "DOM" come up repeatedly I would make a note to study that up when I had time.

What to learn

Choosing what to learn on your own is like writing the curriculum for Calculus without any perspective of what is most important and what order to put it in. You don't know enough to do it well, but you have a sense that many people suggesting an answer have ulterior motives. Prior to this, my first real app was four HTML pages with forms served with JavaScript from Google Apps Scripts (GAS). The database/backend was a Google Sheet. I cobbled it together knowing almost nothing about web dev. I copy/pasted most of my code. I used Bootstrap to make it look decent. But now I wanted to be a professional, so this approach wouldn’t do. Everything I heard from my steady content diet said I had to choose Angular or React to be a pro. This was around 2016, so Angular was the industry leader and React was the new kid. No one (that I heard) was even suggesting you could build for the web without a framework. This was right during the Angular to Angular 2 transition that created a lot of negative sentiment toward Angular. That was enough to tip the scales for me toward React. So I went all in. From then on I narrowed my content consumption and learning heavily to React. React only grew from there. Which confirmed to me I had made the right choice.

What I didn’t realize at the time was that many professionals just mind their business and do their job quietly. Their opinions are underrepresented on Twitter. Some of the most vocal people online are trying to look like professionals. Frameworks are still a huge industry, but a growing community is pointing back to using the platform with fewer layers of abstraction. I did not find this community until I was already working with an agency that placed React developers as contractors. I was starting to see the problems with jamming every website into a React shaped box. Even simple tasks required dozens of dependencies (meaning hundreds or thousands of sub-dependencies) that would break frequently. The simplest “Hello World” required complicated and fragile tooling setups. And the list goes on.

The lesson I learned from my long framework detour is I wish that I had invested more time in learning the underlying fundamentals of the platform first. Whatever the hot framework is next year will be built on those same fundamentals. If you feel like you need to learn the blazingly fast, awesome DX, Kerplunk framework to get a job, then you should do that. But you can learn it faster and better knowing the bricks that it is built out of.

How to learn

I was not in tech at the time I was learning.
We did not have a family budget that allowed for throwing money at this problem. Then, like now, there was no shortage of paid content (courses, books, etc.) promising shortcuts. I was not opposed to spending some money, but I started with all the free resources I could find. Here is a sample of the material I used to learn. It’s only a partial list, but these are some of the better sources I can recommend. I will not list the junk content I encountered but suffice it to say there is a lot out there.

Learn Enough -I started with these short tutorials about web development topics to fill in some gaps. They are paid courses now. At the time, they were being developed, and the drafts were free. It's been years, and I think they were even sold to another company. I used these kinds of tutorials to fill in the gaps of HTML, CSS, JavaScript, Git, etc.
freecodecamp.org - Free education content. Can't speak highly enough of this organization. I did not follow through their curriculum as recommended but I read and used a lot of the pieces.
Frontend Masters - Brian Holt React Course - Frontend masters is a fantastic educational resource for professional development. It is paid and pretty expensive. It is targeted at developers with existing experience trying to learn a new thing. They offered a free weekend preview or another promotion that I took advantage of. I binged the course because it was only free for the weekend. It was like drinking from a firehose. I have always liked to learn by jumping in the deep end, especially where the risk of failing publicly is low. Frontend Masters is great. I highly recommend them. Their courses are excellent. But as I mentioned they are not really targeted at or priced for beginners, so beware.
Wes Bos - The first web development course I paid for was Wes Bos's Advanced React course. Should I have started with his beginner React course, or better yet, his Beginner JavaScript course? Probably yes, but as I said I like to jump in the deep end. Wes is great. He is a good teacher and his courses are worth the time and money.

A winding path

Like many complicated things, learning to code is not a linear process. Looking back, I can see the path I should have taken that was unclear when I first started my journey. My experience is no different from the many people who are learning these skills independently. Few industries approach similarly hard things without a university/college program or a skilled mentor (i.e. new electricians work under a master electrician, or a new wilderness guide works under a seasoned veteran guide) to teach them.

When are you a developer? I cannot say. Some gatekeepers guard the term jealously. But if you build things for the web and that is what you want to be, own it, and then make it true. There will always be more to learn. I distinctly remember a moment when I suddenly had this insight: "Oh, so a Promise is just like a regular Object, and the value just drops in sometime in the future". So many things clicked into place with that one thought. Was I a developer before then with such an incomplete mental model of asynchronous programming? Am I one now with all the things I have yet to learn? The answer to both questions is "yes". But the most important thing is that I have learned how to learn what I need when I need it, so yes, I call myself a developer.

Build real things

At one point, I wrote a small web app to manage some data for my team. It was a basic HTML page with forms served from Apps Script. When I was later learning React, I figured out a way to rewrite it with React and serve it from GAS. Which was not easy to do, and I may be the only one who ever figured out how to do it (probably not, but I felt like a conqueror). I had accomplished something worthwhile. Now we could do so much more with this tool since it was written in React like a professional web app. In reality, this was wasted effort. In fact, this made the app more fragile and harder to maintain. But I would not have seen that unless it was a real app used by real people.

Building real things is the most critical part of learning to code. Building sample apps is helpful for learning concepts, but problems only show up with real use. If you are on your own and don't have anyone to build things for, it is even useful to build things for yourself. Simple tools, scripts, web sites, can turn into powerful feedback loops for learning if you use them to do a job. Look for those opportunities wherever you can.

Conclusion

Getting clear on your goals is essential. Becoming a professional web developer on your own is a slog. You should know that. And anyone who says otherwise is lying. If your goal is to solve problems using the web, it is much more achievable. Don’t get lost learning the next big thing. I wish I had focused on learning the fundamentals of the web platform earlier on.

I ultimately think that the modern web development hype cycle is driven by exploiting the insecurity of developers. Looking back on my journey, I realized that when I was in that cycle, I felt the urgency to learn the next thing: React, Redux, GraphQL, Gatsby... because I felt like I could get the basics and put those words on my resume I stood a better chance of finding a job. It was because I felt genuinely learning the fundamentals was out of reach, and I needed to keep up with the latest thing to stand any chance. Now I don't feel that urgency, not because I feel smarter, but because I know enough, and I can learn anything I need to when I need to.

If you want to learn some web fundamentals check out these resources:

MDN:Learn Web Development - This set of articles aims to guide complete beginners to web development with all that they need to start coding websites.
W3Cx - Master the foundational programming languages for Web development, HTML5, CSS and JavaScript.
enhance.dev - A tool that makes it easy to build functional web apps easier than anything I know of.

That is taking GitHub user growth as a proxy for developers in general (wikipedia) ↩

Uploading files with HTML forms: Part 2

Ryan Bethel — Fri, 17 Feb 2023 18:49:38 +0000

Photo By Dominik Dancs

In Part 1 of this post we built a form and backend code to accept files and images uploaded from a browser. If you missed that post you should read it first and then come back here. We built it with an HTML first approach so that it will work with no client side JavaScript. Here in Part 2 we will add some improvement using a little JavaScript.

Preprocessing image on the client using `<canvas>`

In Part 1 images were reduced on the server so that large images did not waste space, but more importantly so that we did not have to serve those large images back as part of displaying the user profile.

The problem with the server side processing for scaling the image is that you have already sent a larger than necessary image to the server to throw away most of those bits. It would be better to only send what you need. Another problem is that the AWS lambda request has a payload limit of 6 MB maximum. Large images can easily exceed this, and the request will fail before your server sees it.
It also adds a significant delay to the response when submitting to wait for the server to process it.

A better solution is resizing it in the browser before sending it over the network. So with our non-JavaScript solution in place, we will add our first client side JavaScript enhancement.

There are many ways to scale an image in a browser. One good solution might be to send the wasm-vips library to the client and use it inside a worker to run the same scaling in the browser. The pros are the symmetry between the client side result or the server side result.
One downside is the extra dependency and having to ship that code for the library to the client in the first place.

In this example, we will use the browser itself, specifically the <canvas> element to do the image scaling. With no added dependencies! You can read more about using the canvas element for this purpose here (https://imagekit.io/blog/how-to-resize-image-in-javascript/).

The code here is encapsulated in a custom element using an Enhance single file component. First, we add a hidden image tag at the end of the form that will eventually be a preview of the scaled image. The code at the bottom has a resize method that uses the FileReader API to read the image the user selects from disk and assign it as the src of an image we created.
When that image is loaded, we define the scaling parameters and then redraw the image in a canvas element with the new size. We then create a data URL with that scaled image. And finally, we assign that data URL to our preview image and change it from hidden to visible.

// /app/pages/profiles/new.mjs

export default function Html ({ html, state }) {
  return html`
    <main class="container">
      <article>
      <h1>New Profile</h1>
      <form action="/profiles/new" method="POST" enctype=multipart/form-data>
        <label>First Name
          <input type="text" name="firstname" />
        </label>
        <label>First Name
          <input type="text" name="lastname" />
        </label>
        <label>
          Profile Picture
          <input type="file" autocomplete="off" name="picture" />
        </label>
        <img class="hidden" id="profile-preview" alt="profile picture preview"/>
        <button type=submit >Save</button>
    </form>
      </article>
  </main>

  <script type=module >
    class PageProfilesNew extends HTMLElement {
      constructor() {
        super()
        this.form = this.querySelector('form')
        this.imageInput = this.querySelector('input[name=picture]')
        this.imagePreview = this.querySelector('#profile-preview')

        this.resize = this.resize.bind(this)
        this.dataURLtoBlob = this.dataURLtoBlob.bind(this)

        this.imageInput.addEventListener('change', (e)=>this.resize(e))
      }

      resize(e) {
        if (!e.target.files.length) return; // for input file null
        if (e.target.files) {
          let imageFile = e.target.files[0]
          const reader = new FileReader()
          reader.onload =  (e)=>{
            let image = document.createElement("img")
            image.onload = (event) => {

              const size = 350
              const width = image.width
              const height = image.height
              const widthScale = size/width
              const heightScale = size/height
              const outputScale = Math.min(widthScale,heightScale)
              const outWidth = outputScale*width
              const outHeight= outputScale*height

              const canvas = document.createElement("canvas")
              canvas.width = outWidth
              canvas.height = outHeight
              const context = canvas.getContext("2d")
              context.drawImage(image, 0, 0, outWidth, outHeight)

              const dataurl = canvas.toDataURL(imageFile.type)

              this.imagePreview.src = dataurl
              this.imagePreview.classList.remove("hidden")
            }
            image.src = e.target.result
          }
          reader.readAsDataURL(imageFile)
        }
      }
    }

    customElements.define("page-profiles-new", PageProfilesNew)
  </script>

  `
}

We now have a scaled image created on the client that we can send to the server instead of the larger raw version of that image.

To Fetch or Not to Fetch

The question is, now that we have a transformed image in the canvas, how should we send that image to the server. There are two options, each with their own trade-offs. We can stay in JavaScript and send the new file with a fetch request, or we can push the new image back into the <form> to be sent with the built-in form submission.

Fetch is probably the most common choice in this situation, so let's briefly cover how to do that. The advantage is that we don’t have to push the new file back into the form's original form state (which is a little complicated). The disadvantage is we have to pull the rest of the form data over into JavaScript and then handle the response, including error handling and redirect after success, with JavaScript. The basic code for fetch submit is:

const form = document.querySelector('form')
const formData = new FormData(form)

formData.set('picture', processedFile)

fetch('/profiles/new', {
  method: 'POST',
  body: formData,
})

Any error handling and redirect on success would then need to be added.

One important caveat is for this to work with multipart encoding, you cannot set Content-Type headers for this fetch request ( https://muffinman.io/blog/uploading-files-using-fetch-multipart-form-data/). Letting the browser set them automatically will work, but if you set them here, it will fail.

Our previous HTML and server solution allowed for handling errors on the server, setting session, and redirecting following the POST request. With fetch, we have to replicate all that behavior in JavaScript. For this reason, and because it is the less common approach that might be useful to demonstrate here, we will submit the form with HTML instead.

Using JavaScript to populate a form `<input type=file/>`

You cannot programmatically add a file to an <input> from the user's disk because of the browser's security model. But you can create a file in memory and use the browser's DataTransfer API to add that to the input. This code is added below after the data URL (a data URL is a base 64 encoded string of binary data) is added to the preview image.

First, the data URL is converted back into a blob (a Binary Large Object) which is passed to the File constructor. This file is then wrapped with the DataTransfer constructor and added to a new <input type=file name=processed-picture/>. Now we need to set the original file input to null so we don’t send the original image when we submit.

// /app/pages/profiles/new.mjs

export default function Html ({ html, state }) {
  return html`
    <main class="container">
      <article>
      <form action="/profiles/new" method="POST"
        enctype=multipart/form-data>
        <label>First Name
          <input type="text" name="firstname"/>
        </label>
        <label>First Name
          <input type="text" name="lastname"/>
        </label>
        <label>
          Profile Picture
          <input type="file" autocomplete="off" name="picture" />
        </label>
        <img class="hidden" id="profile-preview" alt="profile picture preview"/>
        <input type=file name="processed-picture" class="hidden"/>
        <button type="submit" >Save</button>
    </form>
      </article>
  </main>

  <script type="module">
    class PageProfilesNew extends HTMLElement {
      constructor() {
        super()
        this.form = this.querySelector('form')
        this.imageInput = this.querySelector('input[name=picture]')
        this.imagePreview = this.querySelector('#profile-preview')
        this.scaledImageInput = this.querySelector('input[name=processed-picture]')

        this.resize = this.resize.bind(this)
        this.dataURLtoBlob = this.dataURLtoBlob.bind(this)

        this.imageInput.addEventListener('change', (e)=>this.resize(e))
      }

      resize(e) {
        if (!e.target.files.length) return; // for when input type=file is set to null
        if (e.target.files) {
          let imageFile = e.target.files[0]
          const reader = new FileReader()
          reader.onload =  (e)=>{
            let image = document.createElement("img")
            image.onload = (event) => {

              const size = 350
              const width = image.width
              const height = image.height
              const widthScale = size/width
              const heightScale = size/height
              const outputScale = Math.min(widthScale,heightScale)
              const outWidth = outputScale*width
              const outHeight= outputScale*height

              const canvas = document.createElement("canvas")
              canvas.width = outWidth
              canvas.height = outHeight
              const context = canvas.getContext("2d")
              context.drawImage(image, 0, 0, outWidth, outHeight)

              const dataurl = canvas.toDataURL(imageFile.type)

              this.imagePreview.src = dataurl
              this.imagePreview.classList.remove("hidden")

              const fileName = 'scaled-image'
              const blob = this.dataURLtoBlob(dataurl)
              const file = new File([blob], fileName,
                {type:blob.type, lastModified:new Date().getTime()}, 'utf-8')
              const container = new DataTransfer()
              container.items.add(file)
              this.scaledImageInput.files = container.files
              this.imageInput.value = null
            }
            image.src = e.target.result
          }
          reader.readAsDataURL(imageFile)
        }
      }

      dataURLtoBlob(dataurl) {
        let dataUrlParts = dataurl.split(',')
        const mime = dataUrlParts[0].match(/:(.*?);/)[1]
        const binary = atob(dataUrlParts[1])
        let n = binary.length
        const binaryArray = new Uint8Array(n);
        while(n--){
          binaryArray[n] = binary.charCodeAt(n);
        }
        return new Blob([binaryArray], {type:mime});
      }
    }

    customElements.define("page-profiles-new", PageProfilesNew)
  </script>

  `
}

Now we add logic to the server side handler so that if we send a processed image it doesn't get resized again on the server.

// /app/api/profiles/new.mjs
...

const preprocessed = parsedForm.files?.find(file=>
  file.fieldname==='processed-picture')
const unprocessed = parsedForm.files?.find(file=>
  file.fieldname==='picture')

const picture = preprocessed || unprocessed
const pictureBuffer = picture.content
const profilePicture = preprocessed ? pictureBuffer : await resize(pictureBuffer, 350)

  ...

Avoiding a Double Submit

If a user accidentally clicks submit multiple times the browser will happily send multiple POSTs. This is the double submit problem we discussed in depth in part 1.

We built a working solution without the need for client side JavaScript. With just a sprinkle of JavaScript we can make it even better by making sure the extra POST is not sent rather than handling it after the fact.

To prevent a double submit we disable the submit button after it’s clicked. We could also display a “submitting” message on the button or another visual indication for the user.

// /app/pages/profiles/new.mjs
...
  <script type=module >
    class PageProfilesNew extends HTMLElement {
      constructor() {
        super()
        this.form = this.querySelector('form')
        this.submitButton = this.querySelector('button[type=submit]')
        this.tempDisableSubmit = this.tempDisableSubmit.bind(this)
        this.form.addEventListener('submit', this.tempDisableSubmit)
      }

      tempDisableSubmit() {
        this.submitButton.disabled = true
        setTimeout(() => {this.submitButton.disabled = false }, 5000)
      }

    }

    customElements.define("page-profiles-new", PageProfilesNew)
  </script>
...

We add an event listener to the submit and call a tempSubmitDisable() method. This method disables the submit button for 5 seconds. The reason for the timer is in case the request does fail for some reason the user can try to submit again after a short delay.

This is a clean concise solution to the double submit problem, but it does require JavaScript. My goal is for this example is an acceptable HTML first solution. Avoiding a double submit is really a must have feature for a functional site so we will look for a reasonable solution without JavaScript.

Summary

In Part 1 of this series we covered the HTML first solution to uploading files. In this we have made some small improvements with JavaScript. The full example repository with the code in this post can be found here: https://github.com/ryanbethel/thumbnail-upload-example. Try out enhance.dev for your next project. It has pretty much everything you need to build a functional web app.

Uploading files with HTML forms: Part 1

Ryan Bethel — Wed, 08 Feb 2023 15:35:00 +0000

Photo by Kenny Eliason on Unsplash

Most websites send images, data and files from the server to the user's browser. Sending files and images from the browser to the server is less common but still critical for many useful applications.

In this post, we will cover what you need to know to upload images to the server from the browser using an HTML first approach. HTML first to me means: Make it work with HTML (and CSS) and then make it better with JavaScript. This is similar to what people used to call Progressive Enhancement although most people confuse that for Graceful Degradation which is completely different.So we will start with the simplest approach that could possibly work and then tackle some of the most common problems to make it more robust. This post will be in two parts. The first covers how to make it work without client side JavaScript. The second part will add improvements with some client side code.

Most of this will work for any website hosted anywhere.Some of the details, specifically where and how to store and serve the images, is specific to hosting an Enhance project on Begin.com.

Start with a real `<form>`

Building sites today many people skip right over platform features like HTML <form>s. That is a mistake. Forms are battle-tested, and if you are looking for the simplest thing that could possibly work you should not rebuild form handling from scratch with JavaScript and fetch. The <input type="file"/> was added to HTML in the late 90s. Adding this one tag to an HTML form will allow someone to pick a file from their computer or mobile device to upload.

To send that binary data for the file along with other form fields the encoding type must be set correctly with: <form ... enctype="multipart/form-data">.

The example here demonstrates creating a user profile that includes first name, last name, and an uploaded profile picture. The basic form is shown below.

<h1>New Profile</h1>
<form action="/profiles/new" method="POST" enctype="multipart/form-data">
  <label>First Name
    <input type="text" name="firstname"/>
  </label>
  <label>Last Name
    <input type="text" name="lastname"/>
  </label>
  <label>Profile Picture
    <input type="file" name="picture"/>
  </label>
  <button type="submit">Save</button>
</form>

With this we have a working solution. Now we need a place for those files to go.

Decoding the multipart form-data

The form data and file will arrive at the server encoded in a combined payload. We now need to decode that payload. There are many parsers that will handle multipart data for node. We will use the lambda-multipart-parser that is specifically for use in AWS Lambda.

npm install lambda-multipart-parser

Architect does some preprocessing of the request body, but in this case, the multipart parser expects to receive the raw body. When you pass the request to the parser, you can substitute rawBody for body as shown below in line 5.

import multipart from 'lambda-multipart-parser'

export async function post (req) {
  // the body property needs to be swapped out for rawBody
  const form = await multipart.parse({...req, body:req.rawBody})
  const { firstName, lastName } = form
  const buffer = form.files[0].content

  // Do something with it
  return { /* redirect */  }
}

After decoding the content property holds the file itself as a buffer (an in memory binary chunk of data that represents a file).

Storing Files on the Server

This section is specific to hosting an Enhance project on Begin.com. This will also work for Architect to deploy directly to AWS. Architect and Begin.com provision an S3 bucket for each project deployed. This is where static assets like CSS files and images placed in the /public folder of the project end up. These assets are then served with a special URL directly from this bucket. We will store the uploaded images directly in this bucket when received without using the public folder locally.

The code below identifies the S3 bucket with process.env.ARC_STATIC_BUCKET. It writes the image using the S3 client in the AWS SDK. For local development, it writes the file to a temporary folder instead. The code for local development uses dynamic imports so that importing these dependencies doesn’t slow down the project once deployed.

Begin-data is a thin wrapper for DynamoDB. We will use it (@begin/data) as a database to store the other form data. We generate a UUID as a random filename for the image and then put the reference to that filename in the database as well.

import multipart from 'lambda-multipart-parser'
import crypto from 'crypto'
import data from '@begin/data'
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'

const env = process.env.ARC_ENV
const isLocal = env === 'testing'
const staticDir = process.env.ARC_STATIC_BUCKET
const imageFolder = '.uploaded-images'
const REGION = process.env.AWS_REGION

export async function post (req) {
  // the body property needs to be swapped out for rawBody
  const parsedForm = await multipart.parse({...req, body:req.rawBody})
  const { firstName, lastName } = parsedForm
  const profilePicture = parsedForm.files[0].content

  // Save the image to S3 bucket (or temp folder for local dev)
  const filename = crypto.randomUUID()
  if (isLocal) {
    const {writeFileSync,mkdirSync} = await import('fs')
    const {join} = await import('path')
    const {default:url} = await import('url')
    const __dirname = url.fileURLToPath(new URL('.', import.meta.url));
    const imageDir = join(__dirname,'..','..','..',imageFolder)
    try { mkdirSync(imageDir) } catch(e){ }
    writeFileSync(join(imageDir,filename),profilePicture)
  }
  else {
    const client = new S3Client({ region: REGION });
    const command = new PutObjectCommand({ Bucket:staticDir,
      Key:`${imageFolder}/${filename}`, Body: profilePicture})
    await client.send(command)
  }
  await data.set({table:'profile', firstName, lastName, filename)
  return {
    location: '/profiles'
  }
}

To avoid the possibility of uploaded files being accidentally pruned the following line should be added to the .arc file.

@static
ignore .uploaded-images

Again this approach is specific to Architect and Begin.com projects. It works even with free accounts since the S3 bucket for static asserts is available to any project.

Caveats

Using this S3 bucket for uploaded images comes with a few caveats. It is configured as a public bucket since its primary purpose is to serve public assets. This makes it a good choice for something like profile pictures that are shown as the public part of a user's account. Using a random filename (UUID, for instance) gives only minimal security through obscurity for these files. But it is safest to assume that anything stored here could be accessed. Again good for public profile pictures and bad for tax documents.

There are many other ways to store files on the backend. With Architect by itself there is a plugin available to provision a private bucket that can put these files behind whatever authentication you use for the rest of your app. The private bucket plugin is not currently supported on begin.com but stay tuned.

Serving Uploaded Images

Architect has built-in handlers for serving assets in the public S3 bucket, but since we are using this bucket indirectly, we need to build a handler to serve them correctly. First, we will host this API handler in /app/api/image/$image.mjs. With Enhance's file-based routing, any request to /image/my-picture.jpeg will access these images with the filename as the last part of the path. The $image.mjs is a catchall for that route.

This handler uses the @architect/asap package to serve the images from the S3 bucket. It modifies the request path to point to the correct path inside the bucket. This package helps set headers properly based on content type etc. For local development, it serves files from the same temporary folder we used in the other handler. Again the local development dependencies are only imported when it runs locally. One crucial detail is that here the cache-control header is set to the max length, which effectively tells the browser or CDN these files can be cached for a long time. This is appropriate because we stored them with a UUID, so every one will be unique even if the same image is modified by the user and uploaded again. If this is not true for your implementation (i.e. if users can edit the images and store them again with the same name) this cache setting can be changed.

import asap from '@architect/asap'
const env = process.env.ARC_ENV
const isLocal = env === 'testing'

function escapeRegex (string) { return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') }

const uploadFolderName = '.uploaded-images'
const pathPrefix = '/image' // partial path to remove

export async function get (req) {
  try {
    if (isLocal){
      const {join} = await import('path')
      const {readFileSync} = await import('fs')
      const {default:url} = await import('url')
      const {fileTypeFromBuffer} = await import('file-type')
      const __dirname = url.fileURLToPath(new URL('.', import.meta.url))
      const rootDir = join(__dirname,'..','..','..')
      const imageDir = join(rootDir,uploadFolderName)
      const imageFilename = req.params.image

      const buffer = readFileSync(join(imageDir,imageFilename))
      const mime = fileTypeFromBuffer(buffer)
      return { statusCode: 200,
        headers: {
         'cache-control': 'max-age=31536000',
         'content-type': `${mime}; charset=utf8`,
        },
        isBase64Encoded: true,
        body: buffer.toString('base64')
      }

    }
    else {
      const config = {
        assets: {},
        cacheControl: 'max-age=31536000',
      }
      req.rawPath = req.rawPath.replace(escapeRegex(pathPrefix), uploadFolderName)
      return asap(config)(req)
    }
  }
  catch(e) {
    return {
      statusCode:404
    }
  }

}

Back button confusion

The browser's security model has safeguards around accessing files in the user's disk. This can result in an unexpected behavior if you are used to dealing with forms in other contexts. If you fill out a form, navigate away from the page, and then use the back button to return to that page, the form state that you left is usually restored. But with files selected from the file system, this is not true. The filename will sometimes still be populated, giving the impression that a file is selected, but if you submit the form, the actual file is null. If you set autocomplete="off" on that input the filename field will not be populated when you hit back, which makes it more apparent that no file is selected. Note that the bfcache (back/forward cache) can interfere with this autocomplete setting. This fix works for Enhance/Begin/Architect projects because they set the Cache-Control:no-store header for HTML content which blocks the bfcache.

<h1>New Profile</h1>
<form action="/profiles/new" method="POST" enctype="multipart/form-data">
  <label>First Name
    <input type="text" name="firstname"/>
  </label>
  <label>Last Name
    <input type="text" name="lastname"/>
  </label>
  <label>Profile Picture
    <input type="file" autocomplete="off" name="picture" />
  </label>
  <button type="submit">Save</button>
</form>

Preprocessing Images

Processing images before storing them is not strictly necessary, but almost. You can (and probably should) avoid that complexity initially for a small app or website until you need it. But you don’t have to become Facebook sized before it becomes a problem worth solving. The real problem is not storing the images because disk space is cheap. But if every user who loads that profile thumbnail has to download the 5 MB image, that is a much bigger problem. With smartphone cameras increasing in resolution every year, raw images are enormous relative to what is needed in most web applications.

The easiest way to solve this problem is to process images on the server to scale them down before storing them. This is also the only solution that does not require JavaScript on the client. In Part 2 of this post we will add a client side solution that does not add any dependencies using a <canvas> element.

For this example, we will scale the received image down to 350 pixels max in height or width. And we will use the wasm-vips library to do it. The library is web assembly bindings for the Vips image library. Because it is web assembly it can be used in the AWS lambda node runtime without needing a custom native binary like some other image tools (i.e. ImageMagick).

The code below adds the wasm-vips library and a resize function that scales the image down before storing it.

import multipart from 'lambda-multipart-parser'
import crypto from 'crypto'
import data from '@begin/data'
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'
import Vips from 'wasm-vips'

const env = process.env.ARC_ENV
const isLocal = env === 'testing'
const staticDir = process.env.ARC_STATIC_BUCKET
const imageFolder = '.uploaded-images'
const REGION = process.env.AWS_REGION
async function resize(buffer, size){
  const vips = await Vips()
  const image = vips.Image.newFromBuffer(buffer)
  const heightIn = image.height
  const widthIn = image.width
  const output = image.resize(Math.min(size/heightIn,size/widthIn))
  return output.writeToBuffer('.jpeg')
}

export async function post (req) {
  // the body property needs to be swapped out for rawBody
  const form = await multipart.parse({...req, body:req.rawBody})
  const { firstName, lastName } = form
  const buffer = form.files[0].content

  // Save the image to S3 bucket (or temp folder for local dev)
  const filename = crypto.randomUUID()
  if (isLocal) {
    const {writeFileSync,mkdirSync} = await import('fs')
    const {join} = await import('path')
    const {default:url} = await import('url')
    const __dirname = url.fileURLToPath(new URL('.', import.meta.url));
    const imageDir = join(__dirname,'..','..','..',imageFolder)
    try {
      mkdirSync(imageDir)
    } catch(e){ }
    writeFileSync(join(imageDir,filename),profilePicture)
  }
  else {
    const client = new S3Client({ region: REGION });
    const command = new PutObjectCommand({ Bucket:staticDir,
      Key:`${imageFolder}/${filename}`,
      Body: await resize(pictureBuffer, 350)})
    await client.send(command)
  }
  await data.set({table:'profile', firstName, lastName, filename)
  return {
    location: '/profiles'
  }
}

As mentioned this is the simplest solution and it requires no changes to the HTML.

Avoiding a Double Submit

HTML forms typically send data to the server as a POST request. The browser can send multiple POST requests before a response is received. In this example, that could mean two identical profiles are created in the database, and two copies of the same image are stored in the S3 bucket. If your server responds fast enough, it minimizes the time for these multiple erroneous submissions. The double-click double submit is the hardest to catch because it can be very fast. Users often double-click the submit because they are conditioned to double-click in other contexts.

This problem is more pronounced because the time to upload a large file and process it on the server can be multiple seconds. During this time, the user might have no indication of what is happening and a very natural response is to assume the message was not sent and to click again until something happens.

If users edit a record, a double submit could be detected by comparing the IDs. When creating a new record, you don’t have an ID to compare.

The easiest solution to the double submit is to disable the submit button after it’s clicked once. But we need JavaScript to do that so we will wait for Part 2 to do that.

Using a Submit Token to avoid Double Submit

The only real way to avoid the double submit without JavaScript is to have the server handle it. The key is to detect if one client tries to make multiple POSTs in rapid succession. Enhance, and Architect have built-in sessions that make this easier. There is no real magic behind sessions. It's done by setting an HttpOnly cookie when a response is sent and detecting that cookie when a request is received. For details, check out this Brian Leroux post.

We will create a unique submit token for every GET request to the /profiles/new route. Then we check that token for every POST request received. If multiple posts are made with the same token, they can be assumed to be duplicates. This is a simple concept, but the bookkeeping to make it happen gets a little tricky.

The double click is the worst-case scenario to catch since the two requests could be received with as little as 100 ms delay. This is fast enough that the first request will probably not be completed before a second is received by another lambda. The process is:

Set unique submit token in the session on GET requests to /profiles/new
For each POST to /profiles/new check token table to see if the submit token has been used
If the token has been used:
- Redirect (while keeping that same submit token on the session) to the /profiles list page.
- At the /profiles page check the profiles to see if the profile with that submit token has been entered.
- If it has been received, respond with the list of profiles OR if not wait for a short delay (~2 sec), check again and return the list of profiles.
If the token has not been used:
- Record that token in the tokens table so that it can’t be used again.
- Record the new profile in the database and save the file to S3.
- Clear the submit token and forward the response to /profiles
Also note that when users hit the back button they make a new GET request and new submit token

Below the API handler for the GET and POST requests is added with the submit token handling.

import multipart from 'lambda-multipart-parser'
import crypto from 'crypto'
import Vips from 'wasm-vips'
import data from '@begin/data'
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'
const env = process.env.ARC_ENV
const isLocal = env === 'testing'
const staticDir = process.env.ARC_STATIC_BUCKET
const imageFolder = '.uploaded-images'
const REGION = process.env.AWS_REGION


export async function get (req) {
  const newToken = crypto.randomUUID()
  return {
    session:{...req.session, profileSubmitToken:newToken},
  }
}

export async function post (req) {
  const {profileSubmitToken, ...newSession} = req.session
  const submitting = await data.get({table:'token', key:profileSubmitToken})
  if (submitting) {
    return {
      // session will continue unless reset. If reset the submit token needs to be added to pass on
      // session:{...req.session, profileSubmitToken},
      location: '/profiles'
    }
  }

  const ttl = (Date.now() / 1000) + (60 * 60 * 24 * 1) // One day
  await data.set({table:'token', key:profileSubmitToken, ttl})

  const parsedForm = await multipart.parse({...req, body:req.rawBody})

  const firstname = parsedForm?.firstname
  const lastname = parsedForm?.lastname

  // Get uploaded image
  const unprocessed = parsedForm.files?.find(file=>file.fieldname==='picture')
  const profilePicture = unprocessed.content

  // Save the image to S3 bucket (or temp folder for local dev)
  const filename = crypto.randomUUID()
  if (isLocal) {
    const {writeFileSync,mkdirSync} = await import('fs')
    const {join} = await import('path')
    const {default:url} = await import('url')
    const __dirname = url.fileURLToPath(new URL('.', import.meta.url));
    const imageDir = join(__dirname,'..','..','..',imageFolder)
    try {
      mkdirSync(imageDir)
    } catch(e){ }
    writeFileSync(join(imageDir,filename),profilePicture)
  }
  else {
    const client = new S3Client({ region: REGION });
    const command = new PutObjectCommand({ Bucket:staticDir, Key:`${imageFolder}/${filename}`, Body: profilePicture})
    await client.send(command)
  }

  // Store profile and redirect to profiles list view
  await data.set({table:'profile', firstname, lastname, filename, submitToken:profileSubmitToken})
  return {
    session: {...newSession}, // submit token is removed before continuing
    location: '/profiles'
  }
}

async function resize(buffer, size){
  const vips = await Vips()
  const image = vips.Image.newFromBuffer(buffer)
  const heightIn = image.height
  const widthIn = image.width
  const output = image.resize(Math.min(size/heightIn,size/widthIn))
  return output.writeToBuffer('.jpeg')
}

Below is the API for the /profiles route checking for the submit token.

import data from '@begin/data'

export async function get (req) {
  const {profileSubmitToken, ...newSession} = req.session
  let profiles = await data.get({table:'profile'})
  if (profileSubmitToken) {
    const newProfile = profiles.find(p=>p.submitToken===profileSubmitToken)
    if (!newProfile) {
      await new Promise(resolve => setTimeout(resolve, 2000)) // wait 2sec
      profiles = await data.get({table:'profile'})
    }
  }
  return {
    session:newSession,
    json:{profiles}
  }
}

Summary

Uploading files from a website to a server can be done simply with an input of type="file". If you are building an MVP just start there and don’t over complicate it. You can incrementally solve problems as they arise including scaling images, and avoiding double submits, etc. Validation is a critical feature of well built forms. But it is not unique to upload forms so we did not cover it here. Check out Simon MacDonald’s excellent post on form submission problems for more on that. The full example repository with the code for the example in this post can be found on GitHub. Try out enhance.dev for your next project. It has pretty much everything you need to build a functional web app.

Part 2 of this post will cover improvements that can be made with a little client side JavaScript.

Maintain scroll position across page loads without an SPA

Ryan Bethel — Thu, 12 Jan 2023 16:12:17 +0000

Photo by Taylor Flowe on Unsplash

Get the SPA features you want with the ease of a server-rendered MPA

This post shows how to fix the problem of preserving scroll position across page reloads without the need to buy into a messy Single Page App (SPA) architecture. If you want the TL;DR, skip to the solution at the end of this post.

Server-rendered apps, multi-page apps (MPA), and Server Side Rendering (SSR) have seen renewed interest lately. All these jargon terms used to be just how the web worked. But as the web evolved, developers wanted their websites to feel more like native apps. It just wasn’t cool anymore to have your whole page reload just because you clicked on an item in the sidebar.

For these reasons, the SPA was born. If we treat the entire site as one page and load a ton of JavaScript to handle all interactions right inside that page, everything would surely be awesome.

Narrator: “It was not awesome”.

The problem is we had to throw out lots of good things and start from scratch. Out with <a>s and URLs in favor of client-side routers and a URL that does not reflect the actual state. Out with battle-tested HTML <form>s in favor of DIY forms backed by complicated state management libraries. These SPA architectures caused more problems than they solved with their increased complexity, required JavaScript, and resulting fragility.

I want to show that we can have some of those nice things that were the goal of SPAs, without having all the downsides. One of those nice things is maintaining the scroll position.

The Jumping Scroll Problem

Picture a content site with a sidebar to navigate the content. As you scroll down the sidebar and click a link, it is nice to have the content show up but have the location in the sidebar not jump back to the top of the menu.
With an SPA you get this because you are likely only changing the center content. But with a server-rendered app, the whole page is loading so by default your sidebar will jump up to the top again. Bummer right?

This is a perfect opportunity for progressive enhancement.
Up to now, we could have built this content site with no JavaScript, making it fast, simple to build, and resilient. And even with the jumping scroll problem, annoying as it is, the site works. But with just a little bit of JavaScript, we can solve this problem.

The Solution

To solve the scroll jumping we can monitor the scrollTop setting for the sidebar and restore that location if we reload the page. Enhance.dev uses custom elements to build reusable components. The <docs-layout> is one of those components. One of the benefits of custom elements is that we can easily attach JavaScript progressive enhancement behavior in a script tag defining that element.

<script type="module">
      class Layout extends HTMLElement {
        constructor() {
          super()
          let sidebar = this.querySelector('#sidebar')
          let top = sessionStorage.getItem('docs-sidebar-scroll')
          if (top !== null) {
            sidebar.scrollTop = parseInt(top, 10)
          }
          window.addEventListener('beforeunload', () => {
            sessionStorage.setItem('docs-sidebar-scroll', sidebar.scrollTop)
          })
        }
      }
      customElements.define('docs-layout', Layout)
    </script>

When the JavaScript initializes it checks to see if a location was previously stored for the scroll. If so it restores that location. An event listener is set on the window using the beforeunload event trigger. This event will fire just prior to navigating away from the page. It stores the current location of the scroll using the sessionStorage. Session storage works similarly to local storage except that it is only attached to that session (or tab in most browsers). This means if you have multiple tabs open your scroll will be maintained independently in each tab.

With only 15 lines of code, we have fixed this scroll problem without buying into the whole SPA architecture with its many downsides. Granted, this is only one of the reasons people choose SPAs. But many of those reasons have similarly easy solutions, which we will explore in future blog posts.

If this approach appeals to you give enhance.dev a try. It is the easiest and best way to build functional web apps.

Fixing mobile nav state with 'autocomplete=off': Obscure HTML attributes FTW

Ryan Bethel — Tue, 20 Dec 2022 20:29:15 +0000

Photo by Tekton on Unsplash

Updated: February 08, 2023 to include the affects of bfcache (back/forward cache) on this solution.

I am a big believer in the power of HTML first development.
It is simpler, faster to build, and easier to maintain. JavaScript has its place (mainly as a progressive enhancement), but it is not the first tool you should reach for. We built the Enhance framework to make this way of building apps easier.

Building the enhance.dev site, we used a checkbox for the mobile nav control. This technique is not new and has been written about in lots of other places (i.e. here). But one edge case caused an annoying navigation problem. If you open the mobile nav and follow a link, and then use the back button to return to the previous page, the mobile menu will be open. This is because of the way form elements work in browsers.

The input check box was checked when you navigated away, so the browser tries to help you when you return by restoring the state of that input. This would usually be the desired behavior if this were a normal form. It would allow the user to start where they left off. But as a control for a mobile nav menu, this is very unexpected behavior.

My initial instinct was to fix this with JavaScript. It is a simple problem that a few lines of code could fix. But because I learned to think HTML first, I did some investigation to find a non-JavaScript solution. There is a way! HTML inputs have an autocomplete attribute that has many useful settings. We can set the value to autocomplete=”off” to solve this problem. This tells the browser not to try to restore the checkbox state even when the back button is pressed. Problem solved!

Note: The bfcache (back/forward cache) in some browsers can interfere with this autocomplete setting. This fix works for Enhance/Begin/Architect projects because they set the Cache-Control:no-store header for HTML content which blocks the bfcache.

<input
  id="burger-control"
  class="absolute opacity-0 z-1"
  type="checkbox"
  name="open-burger"
  autocomplete="off"
  aria-label="Open navigation" />

Ok, so the autocomplete attribute is not that obscure but using it to fix this edge case was not easy to google for, so I am putting it out here for anyone trying to do the same (also for future me). The other lesson here that was reinforced for me is never to count HTML and CSS out. MDN Docs are your friend. Whenever I think I need to reach for JavaScript to solve a problem, I question that assumption and spend just a few minutes looking for another solution. It doesn’t always work, but when it does, your app will be better for it. Better how you ask:

Faster - No JavaScript means no waiting until JavaScript loads.
More robust - No fragile code connecting by ids or querySelector.
Simpler - An attribute directly on the HTML is collocated and with the thing it controls.

An Accessible Modal without JavaScript: The Final Boss

Ryan Bethel — Tue, 18 Oct 2022 20:25:00 +0000

A modal is a pop-up dialog message that requires immediate attention. Since modal dialog's are intentionally disruptive, they should be used sparingly. But when you need one, it’s usually for something critical. Building an accessible modal is difficult. Doing it without JavaScript has been almost impossible. I am a fan of progressive enhancement. Build it to work with only HTML (and CSS). Then improve it with just a little JavaScript. It is better for people, and it’s usually easier to build. Modals are an exception to the easy part. They need to hijack other built-in HTML interactions. Without JavaScript, that is hard to do. It’s been called the final boss of accessible components for a long time.

TL&DR

The goal is to build the best modal possible without JavaScript. Then add what’s missing with as little JavaScript as possible. <dialog> seems like the obvious answer, and it comes frustratingly close but for one fatal flaw. The real solution is a checkbox and a form reset button, combined with the new CSS :has selector. :has() is the final piece to unlock this longstanding problem component. It is available in two of the three major browsers and has reached the final stages in the third.

Background

The web is a great platform to build on. It steadily improves with new capabilities, but most critically, it remains backward compatible. Things that used to work will continue to work if they rely on web APIs. My old solution for this modal problem was quite complex, requiring hidden radio buttons and some complicated CSS using :checked selectors. But as time moves on, the web platform has upgraded itself, and the new solution using :has() is much simpler. That Rube Goldburg modal can fade into history (except for those sites where I used it, which I am confident will continue to work until I want to change them).

People coming into web development from this point forward will never know how many hacks it took to make some things like this work. That is a good thing. They can focus on new challenges.

Why not `<dialog>`

Before showing the working solution, you might ask “why not use the <dialog> element.” It is the most semantically correct element for a modal. Most of the difficult requirements, like trapping focus and escape to close, are built in. But <dialog> has one fatal flaw. You cannot open it without JavaScript. It can start, on the initial render, in the open state, and then a form submit button can close it without JavaScript. But rarely do you want to render a page with a modal open already. If you give up on an accessible working no JavaScript solution, then I recommend <dialog>. It is the cleanest and most straightforward solution with JavaScript. Web.dev has a good post with examples. But I am not giving up, so <dialog> is out.

Requirements

Some of the critical features for a modal are:

Opens above main content (i.e. not covered by sticky headers)
Fully keyboard accessible
Traps focus in the modal when open
De-emphasizes main page content to focus attention on the modal
Triggered from anywhere
Multiple open and close targets
Click outside to close
Escape to close

The markup below shows the ideal author experience for this modal. The dialog itself is after the other page content in the document flow. That ensures that no matter what sticky header z-index shenanigans happen in the main content, you can ensure the dialog is not trapped under some other content on the page.

<div class="main-content">
  main content
  <trigger-modal name="my-modal">open modal</trigger-modal>
</div>

<dialog-modal name="my-modal" main-content-class="main-content">
  Dialog Message
  <close-modal name="my-modal">close modal</close-modal>
</dialog-modal>

This example uses custom elements (i.e. <dialog-modal>) rendered with Enhance, which expands them with the internal elements necessary to make it work without JavaScript. The custom tag is just a container to author with and later a place to attach the progressively enhanced behavior. This same solution would work without Enhance. You just need to render the necessary elements into your page somehow.

Here is a CodePen showing a full example.

The key to this modal is to combine a checkbox, <form>, reset button, and :has(). The so-called checkbox hack has been used to control UI elements for many years. But you had to have the checkbox positioned above and outside of the thing you wanted to control. That has all changed with :has(). Now that checkbox can be anywhere in the document, and it can affect the styles for elements anywhere on the page. We use a checkbox to trigger the modal to open, but how do we close it? If that checkbox trigger is in the main content, and the modal blocks that content once open, we can’t directly interact with it to close it. The solution is to add that checkbox to a <form> and use a type=reset button to close it. When that form is “reset” it will toggle the checkbox, no matter where it is in the document, back to its original state, which will close the modal.

The markup below shows the modal in its most basic form. This is all you need to make this work. It takes a little more to make it nice (like styles and labels), but this is the essence of how it works. Note you can have any number of input checkboxes to open the modal and any number of reset buttons to close it if needed.

<style>
  /* Main content visibility hidden to trap focus in modal */
  .main-content:has(input[form="modal-form-my-modal"]:checked) {
    visibility: hidden;
  }
  /* Show modal when open */
  body:has(input[form="modal-form-my-modal"]:checked) .modal-body {
    display: block;
  }
</style>

<body>
  <div class="main-content">
    <input type="checkbox" role="button" form="modal-form-my-modal">
  </div>

  <div class="modal-body" role="dialog">
    <button type="reset" form="modal-form-my-modal">
       Close Modal
    </button>
    <form id="modal-form-my-modal"></form>
  </div>
</body>

Trap Focus without JavaScript

One of the most challenging requirements, without JavaScript, is to trap focus inside the modal when it’s open. Even solving this with JavaScript used to be difficult. You had to capture keyboard events and call focus() back inside the modal when it tried to escape. This is far easier now with the inert attribute. But without JavaScript, we can’t add an attribute. The solution is to set visibility:hidden on the page content instead. The downside is that this hides the main content completely; ideally, we only want to add an opaque layer to obscure the background. But setting visibility to hidden works for accessibility and is a reasonable compromise.

The last two items in the list of requirements are: click outside to close and escape to close. An easy way to add click outside to close is to make the entire backdrop into another reset button when the modal is open. We set tabindex=-1 so that this button is only a click target, and click to close is done.

Progressive Enhancements

That only leaves escape to close. As far as I know, there is still no way to do this without JavaScript. I consider it a nice to have, and I feel comfortable handling this as a progressive enhancement instead. The other compromise we made building the modal was using visibility:hidden instead of inert. Ideally, we wanted to keep the main content visible but set the inert attribute. And, finally, for keyboard control a button should respond to a space key or enter key. Without JavaScript a checkbox will only toggle with space key, so we should add enter key control. Let’s tackle these three items as progressive enhancements now. As a bonus, we will add them to the modal as a JavaScript upgrade in the form of a custom element definition.

class DialogModal extends HTMLElement {
  constructor() {
    super();
    this.form = this.querySelector("form");
    this.inertMain = this.inertMain.bind(this);
    this.unInertMain = this.unInertMain.bind(this);
    this.escClose = this.escClose.bind(this);
    this.enterOpen = this.enterOpen.bind(this);
    this.form.addEventListener("reset", this.unInertMain);
    window.addEventListener("keyup", this.escClose);
  }
  connectedCallback() {
    this.mainContentClass = this.getAttribute("main-content-class");
    this.mainContent = document.querySelector("." + this.mainContentClass);
    this.name = this.getAttribute("name");
    this.allTriggers = document.querySelectorAll(
      "input[type=checkbox][form=modal-form-" + this.name + "]"
    );
    this.allTriggers.forEach((trigger) => {
      trigger.addEventListener("change", this.inertMain);
      trigger.addEventListener("keydown", this.enterOpen);
    });
  }
  disconnectedCallback() {
    this.allTriggers.forEach((trigger) => {
      trigger.removeEventListener("change", this.inertMain);
      trigger.removeEventListener("keydown", this.enterOpen);
    });
    this.form.removeEventListener("reset", this.unInertMain);
    window.removeEventListener("keyup", this.escClose);
  }
  inertMain() {
    this.mainContent.setAttribute("inert", "");
  }
  unInertMain() {
    this.mainContent.removeAttribute("inert");
  }
  enterOpen(e) {
    if (e.code === "Enter") {
      e.target.checked = true;
      this.inertMain()
    }
  }
  escClose(e) {
    if (e.code === "Escape") {
      this.form.reset();
    }
  }
}
customElements.define("modal-dialog", DialogModal);

Some extra boilerplate comes with the custom element solution, but I love knowing that this solution will always work and is not tied to the shrinking lifespan of most frameworks.

All this code does is add an event listener to any checkbox triggers that will add the inert attribute to the main content. And an event listener to the form reset that will remove that inert when it resets. We also set a keyup event listener to reset the form if escape is pressed.

Hidden benefits of a progressive enhancement mindset

One problem I encountered writing this JavaScript is how to undo the visibility:hidden that we set up for the initial solution. I could have done it all with JavaScript, but progressive enhancement has taught me to stop and think “would this be easier in HTML or CSS?” I realized that if we are using inert, then we don’t want to use visibility so I can update the CSS rule by adding a :not([inert]) pseudo selector. I used to use JavaScript as my primary tool for everything. But when you first look to HTML and CSS to do what they do best, a lot of problems that take a lot of JavaScript are much easier to solve in the right domain.

.main-content:not([inert]):has(input[form="modal-form-my-modal"]:checked) {
    visibility: hidden;
}

Finishing accessibility

The focus of this post has been making the modal fully functional so that it can be made accessible. There is a little more work to do for accessibility. The first rule of ARIA is don’t use ARIA if you are using semantic markup. But in this case we are using a checkbox for a button so we need some ARIA attributes. The checkbox needs to be set to role=button. Other attributes will be specific to the use case but you will probably need a few.

Is progressive enhancement really worth it

I think so. I am sure on Twitter at this very moment someone is saying, “who disables JavaScript these days.” While they are competing for likes with the other guy yelling, “HTML isn’t a real programming language” there are many more people quietly trying to build things in the best way they can. Progressive enhancement, using HTML and CSS first to do what they does best and adding small enhancements with JavaScript, is the best way. If that describes you, try using Enhance. It is the best way to build progressively enhanced HTML-first web apps.

If you are curious to see how this modal looks in Enhance here is a CodeSandbox with the Enhance components:
https://codesandbox.io/p/sandbox/enhance-modal-jyzhbf

How-to: resize and transform images automatically without relying on 3rd party APIs

Ryan Bethel — Fri, 07 Jan 2022 16:02:37 +0000

TL;DR

I got fed up with too many "A"s in my JAMstack and built an image processing plugin to eliminate one of them. If you are only interested in the Architect plugin for transforming images and want to skip my hot takes feel free to jump to Part 2.

Part 1: What’s my beef with 3rd party APIs?

About a year ago (before I worked for Begin), I was building a serious side project (meaning it made no money, but I had big dreams). It required managing lots of images. I used static hosting with a SPA frontend framework. I'm not naming names because I am not trying to throw shade on anyone in particular. All of these tools and techniques have a place and time when they work well. And a place where they break down, but I'm getting ahead of myself.

So stop me if you've heard this one before: dealing with images on the web is a pain in the app. To be performant requires many sizes and types for each image. And it would be best if you had some naming convention or system to organize them so you can use them correctly. This used to mean a bespoke manual process or custom scripts. But then there was JAMstack.

The JAMstack way

At its core, the JAMstack architecture has three tools for handling any problem:

Client-Side JavaScript
Build processes that produce static assets
3rd Party APIs

When you only have a hammer, every problem looks like a nail. For dealing with images, any client-side solution means you already sent the wrong image over the network, leaving only the last two hammers.

Most frontend meta-frameworks now include some image component to generate these assets at build time. They are super popular and a big part of the rapid adoption of these frameworks. They are also why your "Hello World" may be hella fast, but your real site takes 30 minutes to build.

Here come the APIs

There are 3rd party APIs to do almost anything you can imagine, including image processing. Just upload your image to one of these services and request the converted version from a special url. This is an awesome solution to the problem.

But APIs are like npm packages. You probably need a few to get anything done, but they multiply fast until the fate of your app is no longer in your hands.

The whole JAMstack movement was built on the premise that you can outsource almost any part of your app to other services through an API. It is the release valve for anything the architecture itself can't do.

So what is wrong with offloading complexity to other services?

You can’t remove complexity without paying for it eventually. Either literally with money or through the increasing complexity of coordinating all these services (see the Law of Conservation of Complexity).

No, really, what is wrong with a Frankenstein app approach?

Some specific downside of depending on lots of 3rd party solutions bolted together are:

Cost: Each service has different limits and pricing structures that combine to make it hard to figure out what you will pay as your app scales.
Coordination: Depending on one service can be tricky, but linking many of them together becomes exponentially difficult. Just the lack of local development and testing support for these services means large parts of your app may be untestable.
Changes: These services are constantly changing. As much as you hope for proper API versioning, you are not guaranteed to get it. Even additive changes result in unplanned work and unscheduled downtime.

Climbs off soapbox

There are three kinds of apps: a toy, a side project, and a business. For the first type, it matters less. Use whatever you can to learn what you want to know. For the last one, you probably have a team of intelligent people with spreadsheets to weigh the trade-offs, so once again, do what you will. But for the serious side project that you hope will eventually become a business or that you want to run long term for other reasons, beware of depending on too many 3rd party services. At least choose a way of building that allows you to build some of this into your project. Architect and Begin are both excellent choices, in my opinion.

Part 2: An Architect plugin for transforming images

If you wanted to skip the hot takes and see the tech, you are in the right place. Let me show you the image plugin I built. And to be clear the real hero of this story that I am trying to promote is the way of building apps that makes this kind of bite size dynamic functionality easy to build in.

What it does

What I needed is to drop in large images and be able to use any size version of those images by just requesting them. The plugin itself is very basic. It does not do anywhere near as much as the 3rd party APIs it replaces. But for me, it does all that I need it to do.

How to Install it

You can install the plugin for any Architect app with npm install @ryanbethel/arc-image-plugin. You can then add it to your app manifest like this:

#app.arc
@app
image-app

@http
get /
get /transform/*  #transform route for arc-image-plugin

@plugins
ryanbethel/arc-image-plugin

Currently, you need to add the get /transform/* in line 7 to register a route for the handler. This is because plugins cannot currently add new routes easily, but that is coming soon. You also need to add the handler itself and the config file below for the same reason. Again will we will add this into the plugin in a future version.

// src/http/get-transform-catchall/index.js
let arc = require('@architect/functions')
let { imageHandler } = require('@ryanbethel/arc-image-plugin')

exports.handler = arc.http.async(imageHandler)

#src/http/get-transform-catchall/config.arc
@aws
memory 1152
timeout 30

How to use it

The Architect framework serves static assets from a local folder that becomes an S3 bucket when deployed to AWS. You drop your giant.jpeg image in the public folder, and then once deployed, you can access it from anywhere in your app at http://example.com/_static/giant.jpeg or with a root relative path at /_static/giant.jpeg.

Architect includes built in fingerprinting, but lets ignore that for the moment for simplicity. With the image plugin, you can request the same image by swapping the “_static” for “transform” and include query parameters to get a different size (/transform/giant.jpeg?width=100&height=100). This will scale the image to fit in those dimensions while maintaining the aspect ratio.

The first time you make a request, it is transformed in a lambda and that new version is saved to a cache S3 bucket. The next request for that size is served from the cache. The uncached request can take 1-3 seconds, but the cached response takes only around 100ms. Scale to fit, cover, and contain transforms are supported as well as grayscale.

Fingerprinting and Updates

Best practice for modern web development uses fingerprinting of static assets. This is so that if you change giant.jpeg, but use the same filename, all the many layers of cache between a user and your site will not block them from seeing your updates. If you use fingerprinting through Architect (see fingerprinting in the docs), a request to giant.jpeg is turned into a request to giant-123abc.jpeg. And transform requests become /transform/giant-123abc.jpeg?width=100. Then when you update the file the fingerprint changes to giant-456xyz.jpeg, and the plugin will generate a new version because you are now requesting /transform/giant-456xyz.jpeg?width=100.

If you are interested, you can see the code for the plugin at https://github.com/ryanbethel/arc-image-plugin.

Local Development

One of the most valuable features of using Architect is local development support for almost everything. I built this plugin to have the same. It uses a local temp directory as a cache for transformed images, and it works just as it would when deployed.

What if I don’t want to write my own plugin

You are welcome to install and use this plugin for your own project. It is brand new, and you should consider it under development. Pull requests are welcome or if you want to fork it and make it your own that's great too.

If you are new to Architect or if you are using Begin.com (which does not yet have full support for plugins), you can replicate most of what it does in a single route/lambda in your app. That is what I did before I rolled it into a plugin to be more portable. If you aren’t using Begin or Architect, you should try them out.

'Avoid Surprise Bills from AWS'

Ryan Bethel — Tue, 16 Nov 2021 18:43:54 +0000

Architect users often ask for some way to avoid large surprise bills from AWS. The budget-watch plugin sets a cost limit for your app and temporarily shuts it down when the limit is reached. It is the first step to solving this problem for our community.

Scaling is a built-in feature of dynamic apps using cloud functions. But if your app can scale to infinity, so can your bill. Many users have this concern, even though it rarely happens. Amazon has many services to help monitor billing. Setting an account-wide budget alert is a relatively easy first line of defense. But configuring more fine-grained monitoring for a single app is not so easy. It's a little like trying to fix your car by breaking into your mechanic's garage. The billing services are too complicated for a small team to devote their limited resources to.

The unit of deployment in Architect is an individual app. None of the AWS budget solutions easily scope to a single app. You might have one app you expect to cost $100 per month and a dozen other experimental apps that you expect to be free. You should be able to easily set these limits and have some assurance there will be no surprises.

What budget-watch does

There are many ways an app could run up your bill. Maybe you hit the top of hacker news, or you might have an infinite loop. The simplest way to stop a runaway app is to shut down the compute resources. By setting reserved concurrency (how many simultaneous executions can run) on all lambdas to zero, you can effectively stop most things that cost money in your app. You install the plugin with npm install @architect/plugin-budget-watch, and then add the four lines shown below to your app manifest.

@plugins
budget-watch

@budget
limit $40

If the app uses multiple plugins, budget-watch should be listed last so that it is the last one applied. When deployed, it attaches a budget alert scoped to just the resources of the app. If the cost of those resources exceeds the limit set, a shutdown is triggered. To restart the app, the limit can be increased or removed and the app redeployed. This resets the lambda concurrency, and the app will resume operation. You can see the code on github.com/architect/plugin-budget-watch for more details.

Implementation details (the messy parts)

We would love to see Amazon build this feature into the platform. Solving this problem from the outside exposes many rough edges in the billing and CloudFormation APIs. The three biggest challenges are:

Enabling tags
Slow billing updates
Managing configuration drift

Enabling tags

An ideal solution would be deployed entirely through the app manifest using CloudFormation (AWS's infrastructure as code that Architect uses underneath). All that is needed to scope to an app is a single auto-generated tag (aws:cloudformation:stack-name). But before this tag can be used, someone needs to navigate deep into the AWS console to activate it. This only needs to be done once, but it breaks the ideal user experience of avoiding the console altogether. Some users may not be allowed to activate tags because of account limits set by their organization.

Billing updates

Amazon bills you by the millisecond but only lets you check three times a day. It's like touching a stove and finding out eight hours later that you got burned. You can set all kinds of alerts on all sorts of dimensions, but you can't do any better than 8-12 hours of granularity. Even if you set an alert for a limit you have already passed, the notification may still be delayed for half a day until the next billing update.

Drift detection

Deterministic deploys are a core feature of Architect. You get the same infrastructure every time you deploy the same app manifest. This plugin should not break that contract. Setting a budget limit and resetting it after it has triggered are all done from the app manifest. To shut down the app the plugin changes all concurrency by directly calling the lambda API. This causes a drift (out of band configuration changes to infrastructure) between your app and manifest.

This drift needs to be reversed if the app is restarted. If you deploy the same CloudFormation template, it will not make any update because the template has not changed. AWS has "Drift Detection" that you can enable to monitor the out-of-band changes, but it requires clicking around the console to enable it. There is no way to turn drift detection on with CloudFormation, and there is no way to automatically reconcile that drift by having your template overwrite those configuration changes. Not only that, drift detection does not even monitor the relevant lambda configurations.

The workaround for this drift reset is to use a CloudFormation Custom Resource as a reset mechanism. Custom resources are intended for provisioning infrastructure outside of AWS and connect them to your stack through CloudFormation. They have lifecycle hooks for create, update, and delete that run custom logic. After the budget-watch limit is triggered, it can be reset by increasing the limit in the manifest or by removing the limit. This triggers an update in the custom resource that resets the concurrency back to its original settings.

Other approaches considered

There are many ways to build a feature like this with AWS building blocks, but most suffer from the same limitations (i.e., 3x/day updates). Two promising approaches considered were cost anomaly detection w/SNS alerts and budget triggered actions. Cost anomaly detection was not used primarily because the alerts look for deviations in billing rather than absolute limits. What is the "expected" budget for the app I just deployed for the first time?

Budget triggered actions seemed like the most promising solution provided by AWS. You set a budget with an alert that has automated actions attached. The challenge is that the actions you can specify are permissions, policy, and instance-based. These actions end up tightly coupled to implementation details of the app and must be updated as the app changes.

Limitations

The actual causes of large surprise bills are as varied as the many services that AWS offers. This solution cannot possibly catch every one. It focuses on the biggest source and applies the broadest intervention. Architect is a very flexible framework. It is possible (especially with custom plugins) to include infrastructure that will not be shut down by budget-watch.

This plugin is still in Beta. We encourage people to try it out and give feedback. If you want to add the plugin to your Architect project you can follow the instruction in the GitHub repository (architect/plugin-budget-watch).

What about Begin users?

Begin.com has a generous free tier, so users do not have to worry about costs. But for those on the paid tier, we will soon make this feature available to all users of Begin. We hope that AWS will build this into the platform in a more usable way. Until then, we hope to relieve some of the fear of unexpected bills. If you want to build scalable web apps with Begin sign up for a free account today.

DEV Community: Ryan Bethel

OAuth Authentication

TL;DR

GitHub OAuth Keys

Local development without GitHub API keys

GitHub Login Button

OAuth log in

Registration

Other OAuth providers

Try it out

Next Step

Authentication with Magic links

TL;DR

Magic Link Login

Magic code/SMS

Email only streamlined registration

Try it out

Next Step

Verify email and phone for new accounts

TL;DR

Lego not Ikea

Twilio and SendGrid

Environment variables

Local development without API keys

Email verification

Phone verification

Forgot Password

Next steps

Authentication for a Username and Password flow

TL&DR

Secure Sessions

Register New Accounts

Validation

Storing Passwords

Login

Hard coded accounts

Next Step

My Experience Learning to Code

TL&DR

Background

My Goal

Finding time

What to learn

How to learn

A winding path

Build real things

Conclusion

Uploading files with HTML forms: Part 2

Preprocessing image on the client using <canvas>

To Fetch or Not to Fetch

Using JavaScript to populate a form <input type=file/>

Avoiding a Double Submit

Summary

Uploading files with HTML forms: Part 1

Start with a real <form>

Decoding the multipart form-data

Storing Files on the Server

Caveats

Serving Uploaded Images

Back button confusion

Preprocessing Images

Avoiding a Double Submit

Using a Submit Token to avoid Double Submit

Summary

Maintain scroll position across page loads without an SPA

Get the SPA features you want with the ease of a server-rendered MPA

The Jumping Scroll Problem

The Solution

Fixing mobile nav state with 'autocomplete=off': Obscure HTML attributes FTW

An Accessible Modal without JavaScript: The Final Boss

TL&DR

Background

Why not <dialog>

Requirements

Trap Focus without JavaScript

Progressive Enhancements

Hidden benefits of a progressive enhancement mindset

Finishing accessibility

Is progressive enhancement really worth it

How-to: resize and transform images automatically without relying on 3rd party APIs

Preprocessing image on the client using `<canvas>`

Using JavaScript to populate a form `<input type=file/>`

Start with a real `<form>`

Why not `<dialog>`