DEV Community: Alexey

WebGPU tutorial: compute, vertex, and fragment shaders on the web

Alexey — Thu, 16 Jan 2025 21:27:43 +0000

A gift to the world, WebGPU promises to bring cutting edge GPU compute capabilities to the web - to all consumer platforms everywhere with a shared codebase.

Its predecessor, WebGL, fantastic in its own right, sorely lacked compute shader functionality - limiting applications.

WGSL - the WebGPU shader/compute language - borrows from the best in its domain: Rust and GLSL.

This tutorial fills a gap in documentation I felt when I was learning to work with WebGPU: I wanted a simple starting point for using a compute shader to calculate data for the vertex and fragment shaders.

The single-file HTML with all the code explained in this tutorial can be found at https://github.com/alexey-dc/webgpu_html/blob/main/000_compute_vertex_fragment.html - read on for the breakdown.

Here's that HTML running on my domain for a single click demo: https://alexey-dc.com/wgsl_demo (requires WebGPU-enabled browser like Chrome or Edge https://caniuse.com/webgpu).

High level setup

This is a particle simulation - that is it occurs over time, in timesteps.

Time is tracked in JS/on the CPU, passed in as a (float) uniform to the GPU.

The particle data is managed entirely on the GPU - though there is still a handshake with the CPU that allows allocating the memory and setting initial values. It's also possible to read the data back to the CPU, but that's left out of this tutorial.

The magic of this setup is that every particle is updated in parallel with all other particles, enabling once mind-boggling compute and render speed capabilities in the browser (the parallelization maxes out at the number of cores on the GPU; we can divide the number of particles by the number of cores to get the true number of cycles per core per update step).

Bindings

The WebGPU mechanism for data exchange between the CPU to the GPU is bindings - a JS Array (like a Float32Array) can be "bound" to a memory location in WGSL with a WebGPU Buffer. The WGSL memory location is identified with two integers: a group number and a binding number.

In our case, both the compute shader and the vertex shader rely on two data bindings: time and particle positions.

Time - uniforms

The uniform definition exists in both the compute shader (https://github.com/alexey-dc/webgpu_html/blob/main/000_compute_vertex_fragment.html#L43) and the vertex shader (https://github.com/alexey-dc/webgpu_html/blob/main/000_compute_vertex_fragment.html#L69) - the compute shader updates position, and the vertex shader updates color based on time.

Let's take a look at the binding setup in JS and WGSL, starting with the compute shader.

const computeBindGroup = device.createBindGroup({
  /*
    see computePipeline definition at
    https://github.com/alexey-dc/webgpu_html/blob/main/000_compute_vertex_fragment.html#L102

    it allows linking a JS string with WGSL code to WebGPU
  */
  layout: computePipeline.getBindGroupLayout(0), // group number 0
  entries: [{
    // time bound at binding number 0
    binding: 0,
    resource: {
      /*
      for reference, buffer declared as:

      const timeBuffer = device.createBuffer({
        size: Float32Array.BYTES_PER_ELEMENT,
        usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST})
      })

      https://github.com/alexey-dc/webgpu_html/blob/main/000_compute_vertex_fragment.html#L129
      */
      buffer: timeBuffer 
    }
  },
  {
    // particle position data at binding number 1 (still in group 0)
    binding: 1,
    resource: {
      buffer: particleBuffer
    }
  }]
});

And the corresponding declarations in the compute shader

// From the compute shader - similar declaration in vertex shader
@group(0) @binding(0) var<uniform> t: f32;
@group(0) @binding(1) var<storage, read_write> particles : array<Particle>;

Importantly, we bind the JS-side timeBuffer to WGSL by matching the group and binding numbers in JS and WGSL.

This gives us the power to control the variable's value from JS:

/* Just need 1 element in the array since time is a single float value */
const timeJs = new Float32Array(1)
let t = 5.3
/* Plain JS, just set the value */
timeJs.set([t], 0)
/* Pass the data from CPU/JS to GPU/WGSL */
device.queue.writeBuffer(timeBuffer, 0, timeJs);

Particle positions - WGSL storage

We store and update the particle positions directly in GPU-accessible memory - allowing us to update them in parallel relying on the massive multi-core architecture of the GPU.

The parallelization is orchestrated with the help of a workgroup size, declared in the compute shader:

@compute @workgroup_size(64)
fn main(@builtin(global_invocation_id) global_id : vec3<u32>) {
  // ...
}

The @builtin(global_invocation_id) global_id : vec3<u32> value gives a thread identifier.

By definition, global_invocation_id = workgroup_id * workgroup_size + local_invocation_id - which means it can be used as a particle index.

For example, if we have 10k particles, and a workgroup_size of 64, we'll need to dispatch Math.ceil(10000/64) workgroups. We'll explicitly tell the GPU to do that amount of work each time we trigger a compute pass from JS:

computePass.dispatchWorkgroups(Math.ceil(PARTICLE_COUNT / WORKGROUP_SIZE));

If PARTICLE_COUNT == 10000 and WORKGROUP_SIZE == 64, we'll launch 157 workgroups (10000/64 = 156.25), and each will compute ranging local_invocation_id from 0 to 63 (while workgroup_id will range from 0 to 157). We'll end up doing slightly more calculations in one of the workgroups, since 157 * 64 = 1048. We deal with the overflow by discarding the extraneous invocations.

Here's what the compute shader ends up looking with those considerations:

@compute @workgroup_size(${WORKGROUP_SIZE})
fn main(@builtin(global_invocation_id) global_id : vec3<u32>) {
  let index = global_id.x;
  // Discard extra computations due to workgroup grid misalignment
  if (index >= arrayLength(&particles)) {  
    return;
  }
  /* Convert integer index to float so we can compute position updates based on index (and time)*/
  let fi = f32(index);    
  particles[index].position = vec2<f32>(
    /* No grand intent behind the formulas - just an example of using time+index */
    cos(fi * 0.11) * 0.8 + sin((t + fi)/100)/10,
    sin(fi * 0.11) * 0.8 + cos((t + fi)/100)/10
  );
}

These values will persist across compute passes, because particles are defined as a storage var.

Reading particles positions from compute shader in vertex shader

To read the particle positions in the vertex shader from the compute shader, we'll need a read-only view into the data, since only compute shaders are allowed to write to storage.

Here's the WGSL declaration for that:

@group(0) @binding(0) var<uniform> t: f32;
@group(0) @binding(1) var<storage> particles : array<vec2<f32>>;
/*
Or equivalent:

@group(0) @binding(1) var<storage, read> particles : array<vec2<f32>>;
*/

Attempting to re-use the same read_write style from the compute shader would just error out:

var with 'storage' address space and 'read_write' access mode cannot be used by vertex pipeline stage

Note that the binding numbers in the vertex shader don't have to match the compute shader binding numbers - they just need to match whatever the declaration for the bind group for the vertex shader are:

const renderBindGroup = device.createBindGroup({
  layout: pipeline.getBindGroupLayout(0),
  entries: [{
    binding: 0,
    resource: {
      buffer: timeBuffer
    }
  },
  {
    binding: 1,
    resource: {
      buffer: particleBuffer
    }
  }]
});

I opted for binding: 2 in the GitHub sample code https://github.com/alexey-dc/webgpu_html/blob/main/000_compute_vertex_fragment.html#L70 - just as a matter of exploring the boundaries of the constraints imposed by WebGPU

Running the simulation step by step

After all the setup is in place, the update-and-render loop is orchestrated in JS:

/* Start simulation at t = 0*/
let t = 0
function frame() {
  /*
    Use constant integer timesteps for simplicity - will render consistently regardless of framerate.
  */
  t += 1
  timeJs.set([t], 0)
  device.queue.writeBuffer(timeBuffer, 0, timeJs);

  // Compute pass to update particle positions
  const computePassEncoder = device.createCommandEncoder();
  const computePass = computePassEncoder.beginComputePass();
  computePass.setPipeline(computePipeline);
  computePass.setBindGroup(0, computeBindGroup);
  // Important to dispatch the right number of workgroups to process all particles
  computePass.dispatchWorkgroups(Math.ceil(PARTICLE_COUNT / WORKGROUP_SIZE));
  computePass.end();
  device.queue.submit([computePassEncoder.finish()]);

  // Render pass
  const commandEncoder = device.createCommandEncoder();
  const passEncoder = commandEncoder.beginRenderPass({
    colorAttachments: [{
      view: context.getCurrentTexture().createView(),
      clearValue: { r: 0.0, g: 0.0, b: 0.0, a: 1.0 },
      loadOp: 'clear',
      storeOp: 'store',
    }]
  });
  passEncoder.setPipeline(pipeline);
  passEncoder.setBindGroup(0, renderBindGroup);
  passEncoder.draw(PARTICLE_COUNT);
  passEncoder.end();
  device.queue.submit([commandEncoder.finish()]);

  requestAnimationFrame(frame);
}
frame();

Final words

WebGPU unlocks the power of massively parallel GPU computations in the browser.

It operates in passes - each pass having localized variables, enabled through pipelines with memory bindings (bridging CPU memory and GPU memory).

Compute passes allow orchestrating parallel workloads through workgroups.

While it does require some heavy set up, in my humble opinion the local binding/state style is a huge improvement over the global state model of WebGL - making it much easier to work with, while also finally bringing the power of GPU compute to the web.

Webapps need app-like monetization

Alexey — Sun, 19 Jun 2022 01:22:00 +0000

I wrote my first iOS app in 2010 while still in college - it was a zynga-style social RPG called Shovel Heroes.

The app store was a beautiful innovation: a new way to distribute software in the age of internet. One important part of the innovation was monetization built into the ecosystem - since Apple keeps the credit card tied to the device.

The app store unleashed an explosion of applications, because seamless distribution on a monetizable channel created the financial incentive, entirely new opportunities.

However, Apple has treated developers terribly with fees, gatekeeping, lack of support, poor documentation, and lackluster tooling.

Developing apps has always been expensive and laborious, and sadly native development has only gotten worse over time: XCode overheats my $4000 M1 Pro Max laptop after just 15 minutes, causing glitchy/laggy performance of the entire handheld supercomputer.

Recently, I built Spicebreaker - at first as a native app built with React Native, but eventually I re-wrote from scratch on the web.

The main punchline of this post is that payments, identity, and distribution are the major advantages of the app store over the web as a publishing platform, and that crypto wallets have the potential to solve for two of those problems on the web. If you wish, skip through the dev story and check out the UX crypto enables.

Pivoting from native mobile app to web

Much of what I say here primarily affects indie developers - large companies still suffer, but they have the budget to power through it.

On a limited budget, a cross-platform solution is that much more desirable.

I wrote a prototype in React Native on a weekend, and wrote a content management system on another weekend, which my wife used to lead the content creation. It took a few months to playtest it, iterate on the core concepts. After maybe 6 months it was ready to launch on mobile, perhaps needing a few finishing touches.

I thought it was one a finish line to launch, but ultimately I abandoned the React Native project and launched on web.

Issues with iOS

I've worked with many developer tools in my career: Sublime, Eclipse, Visual Studio, Vim, Jetbrains/IntelliJ, VSCode, Ruby Mine, even BorlandC++/BorlandPascal, Delphi, Notepad++... XCode is the absolute worst developer tool I have ever worked with. Sorry. Not sorry.

I've squandered hundreds of precious hours of my life dealing with arcane XCode issues throughout my career. There's no real understanding/pattern or depth behind dealing with them, and support is close to non-existent.

One example during Spicebreaker's development was the pain of updating the SQLite database with my questions. It's just a file - I expected to override it, and have the app pick up the new version, but it doesn't work that way.

I tried various solutions, including deleting the database before loading it - but not even that worked to reset the cache. After wasting more hours than it took to build the prototype, I settled on renaming the database file each time I update it.

The key point I want to drive home is not this specific minor issue, but that hundreds of little issues like this slow down development and drain the joy from making apps.

At a certain point, I bought a new M1 laptop (my old Mac Pro had developed debilitating overheating issues). XCode and iOS are famous for incompatibility issues, and sure enough my build broke with arcane inexplicable errors. I had to start a fresh app, and slowly carry over features from the old app, over 3 days on Christmas break - which had meant a 9 month pause on any development progress since the app wasn't building.

Apple's corporate attitude towards developers

I tried to register a special email for Spicebreaker's Apple developer account. Apple rejected every credit card my wife and I put into the registration. It took days to even reach a rejection, and support had nothing to offer us in help. We ended up using my wife's personal Apple account for the developer program.

Publishing a Beta on TestFlight requires a review/approval from Apple, and my first review failed with no note. How is that even possible? This is a Beta, not a release. Why do I need review?

Last straw

So in these 2 years the Epic vs Apple trial had unraveled. And one of the big arguments Apple made was that they are owed the 30% commission because they provide highly valuable tools to developers.

Given the above rants, I hope my passionately violent gut reaction can be understood through a rational lens.

I would pay lots and lots of money to not have to deal with their developer tools. I would pay a different app store 50% commission if only I got to not have to deal with XCode, TestFlight, the Apple Developer Program, and their review process. Apple fights vigorously against the possibility, though - obviously they are in no shape to compete.

I needed a website to promote the app. I started up a basic Express+NextJS server, and... re-built the entire game, designs included, in a day or two on a weekend. Oh, and by the way, I no longer have to deal with any of those SQLite issues - because I'm directly connected to my database by default. No build issues. No deploy issues. No issues publishing a Beta, no issues sharing the game with my friends. Christ almighty, what a breath of fresh air.

I just want to reiterate that these considerations become maximally important with a constrained budget, though arguably cheaper is always better as long as quality isn't compromised.

Switching to web

The pivot to web made me reflect on what pushed me to write an app in the first place.

Mobile apps have done very well in monetization. Mobile has nearly eliminated the friction of payments: on both the app store and inside apps, payments are just a few interactions away.

So what's the problem on the web? There's Stripe, PayPal - I can ~easily integrate payments. Why is the web an inferior payment platform for applications like Spicebreaker?

Payments and identity

The friction of payments on the web is identity.

Physical goods don't require a digital identity per se - but you'd need to put in your address each time you make a purchase without one.

Digital goods, on the other hand, absolutely require a digital identity, since their ownership is impossible otherwise. They represent most purchases on the app store: content subscriptions, in-game currency, unlockable features.

Spicebreaker sells packs of questions, i.e. digital content; to prove you've made a purchase and claim that digital content a year later on a different device, you need to be able to authenticate with a repeatable credential.

The need to set up those repeatable credentials creates the crux of the friction.

Identity on the web

The problem

Modern mobile devices provide identity by default: e.g. Apple ID or Google Account. That identity is shared by all apps, e.g. in-app purchases are tied to your Apple ID on iOS, and can be recovered when switching devices. The key is that the setup is done just once - when setting up the phone - and then consumed by all apps.

The web does not provide a cohesive identity platform: each account-based website is ignorant of your login information on other websites. This is by design, of course. OAuth is the strongest modern solution to the problem, but the experience of OAuth + Payment is not nearly as seamless as platform-based payment identity. OAuth also forces you to reveal your web activity to OAuth providers.

Today, browser-based web identity does not yet exist. Until recently, the only way to purchase digital goods on the web and retain ownership was to (ugh) Register an Account. This enormous friction is a major reason the web has not yet become a vibrant app publishing platform the way the app store is.

The solution

There already exists a technology that links identity and payments on the web that can act as a platform for applications: web3 wallet plugins. Wallet plugins need to be set up once per browser, and can unify identity across web sites via public keys (and private key signatures verifying ownership of those public keys).

If that sounds complicated, please check out the login flow on Spicebreaker. The UX itself is dead simple (that's the whole point):

Click a button to begin sign in process with Phantom
Click "Approve" inside of a Phantom Wallet popup

There are also more technical details below.

The purchase is done via USDC, so it's always $10. The purchase is a 3-step experience:

Connect Phantom Wallet
Click "Purchase"
Confirm payment in Phantom Wallet popup

Here's what it looks like:

The reason I chose Phantom wallet for this technical experiment is that the Solana blockchain facilitates extremely low fees (fractions of a cent). There are certainly alternatives out there (e.g. Polygon), and I look forward to trying them out in the future.

Low fees are extremely important for an application whose most expensive purchase is $10. Ethereum gas fees ($3-$100) are exorbitantly high for in-app purchase applications.

Technical details

How does the web3 identity/purchase solution work, exactly?

The gist of it is that a purchase is tied to a public key, and access is authenticated via a private key signature - the same way blockchains authorize transactions, except done on top of a traditional centralized database application.

A wallet is able to issue signatures and share the public key. So the mechanism is as follows:

The front end requests authentication for a public key via an API call
The API call returns some random string of data
The wallet issues a signature for that random string of data
The front end resolves authentication by providing the signature via an API call
If the signature is correct, the backend issues a session for the front end

A purchase is automatically tied to a public key, since every transaction sent to the blockchain contains information on which public key triggered it - so without any extra effort the purchase is tied to your identity, and you are able to prove your identity as long as you have the corresponding private key. Which you absolutely should always have if that's where your money is stored.

Distribution

Perhaps crypto wallets can solve identity and payments, but what about distribution - the third pillar of the App Store?

It's great to be promoted by the app store - winning a top award is a sure way to get millions of users.

The web doesn't have a single natural distribution platform. The distribution platforms of the web are SEO, social networks, bloggers, influencers etc.

There's no one way to play growth... but perhaps winning on the App Store is not easier than winning on a web distribution platform!

Harsh modern reality

Web3 and crypto are not mainstream.

I was extremely happy to imagine (and implement) a future with web payments as seamless as on the app store.

Today, the harsh reality is that a mass audience is not going to pay that way. So ultimately, Spicebreaker supports creating accounts via email, and paying with PayPal. I don't collect any personal info, and don't store passwords - the email sends a one-time code each login attempt, which I find much more usable than passwords. I'll eventually add various OAuth providers to also help with sign-up friction.

The UX isn't terrible, but it's nothing like the single-click seamless experience with wallets, or with in-app purchases.

There are many advances in the space. Brave browser is pioneering work on becoming a browser-based identity and payments platform, and they are getting traction with users. Even Chrome now supports identity in the browser - though it's to your Google Account, which these days is one of the least private types of accounts you can link to an application. Would you want Google to know what purchases you make on every web page?

In that world, as an indie developer, I would realistically integrate with the most used and least friction-prone payments platforms on the web; I'd grind my teeth but integrate with Google - so true privacy may be a luxury until crypto is adopted in the mainstream.

My hope is that web3 authentication and payments do become more mainstream, and wallets dominate web payments. That way, independent developers like myself can publish to an open, privacy-respecting, universally accessible platform, where we can use the best developer tools in existence: the web.

PM2 + Express + NextJS (with GitHub source): zero downtime deploys

Alexey — Sat, 11 Jun 2022 20:12:23 +0000

This article builds on a previous article of mine, which introduced a basic Express+NextJS setup that enabled hosting both a React-based front end and API on one service - reducing distributed system hassles.

This article moves that setup closer to production by introducing zero-downtime backend deploys via PM2. We'll also introduce logging for this setup via log4js, since it requires initializing PM2 in a log-compatible way.

I've deployed this project as a demo on an EC2 instance in AWS: https://nextjs-express.alexey-dc.com/

The source code

Like the previous template, I open sourced this under the MIT license - so you're free to use it for commercial and closed-source projects, and I would of course appreciate attribution.

https://github.com/alexey-dc/pm2_nextjs_express_template

The details for launching can be found in the README.md.

It inherits the same basic setup/pages, but has more sophisticated configuration on launch - that works with PM2. I'll dive into a few details here.

Zero downtime deploys

Two of the most common strategies for deploys without downtime are blue-green deployments and rolling deployments.

PM2 enables rolling deployments on a single machine.

This is possible because it allows running multiple threads running the same server code via cluster mode, which can be replaced one by one.

Here's an example of a sequence of commands that can achieve a rolling update with PM2:

# Launch 2 instances of a server defined under index.js (-i 2)
pm2 start index.js --name pm2_nextjs_express -i 2
# Perform rolling update with the latest code:
# First kill and replace the first instance, then the second
pm2 reload pm2_nextjs_express

Graceful PM2 setup

Here's how the template actually launches:

pm2 start index.js --name pm2_nextjs_express --wait-ready --kill-timeout 3000 -i 2

There are 2 additional flags: --wait-ready and --kill-timeout - they allow graceful booting and cleaning up.

Let's take a look at some key bits from index.js - which works with those flags. I've slightly modified the code here to focus on the points being made, but you can always read the real source code.

Graceful setup

We let PM2 know that we've completed setup by sending a process.send('ready') signal after all the configuration:

const begin = async () => {
//  ...
  const server = new Server(process.env.EXPRESS_PORT)
  await server.start()
  /*
    Let pm2 know the app is ready
    https://pm2.keymetrics.io/docs/usage/signals-clean-restart/
  */
  if (process.send) {
    process.send('ready')
  }
//  ...
}
begin()

Graceful teardown

During shutdown, PM2 sends a SIGINT signal, and expects us to process.exit(); it waits for --kill-timeout (3000ms in our case), and the sends a SIGKILL.

So to respect that lifecycle and perform cleanup, we listen for the SIGINT signal, perform cleanup, and exit:

  process.on('SIGINT', async () => {
    try {
      await server.stop()
      process.exit(0)
    } catch {
      process.exit(1)
    }
  })

Logging

Since PM2 runs on multiple threads, logging can be challenging. This is why I've included a sample integration of PM2+Log4js.

That does not work out of the box - but log4js explicitly supports a {pm2: true} flag in its configuration.

The log4js docs mention that pm2-intercom is necessary to support this. Using that as-is gives an error due to the process.send('ready') message we send, however:

  4|pm2-intercom  | Error: ID, DATA or TOPIC field is missing

Luckily, there is a fork of pm2-intercom that explicitly addresses this issue https://www.npmjs.com/package/pm2-graceful-intercom

I've also documented this in detail in the log configuration included with the project.

Debugging

I've included a setup for debugging as well.

# This will run on `pnpm debug`
pm2 start index.js --name pm2_nextjs_express_debug --wait-ready --kill-timeout 3000 --node-args='--inspect-brk'
# This will run on `pnpm stop_debug`
pm2 delete pm2_nextjs_express_debug

The --node-args='inspect-brk' flag enables debugging via a socket connection. It's a standard node flag. One great way to work with that debug mode is via Chrome's chrome://inspect. If you don't want to use chrome, just see the official Node.js docs for more options.

You'll notice I don't enable cluster mode for debugging - that's because it doesn't work well.

You'll also notice I launch it on a separate name, don't offer a reload, and the stop involves deleting the process from PM2, vs stopping it - like for the normal run mode. The main reason I did that is because the breakpoints can cause issues for restarts - PM2 will print errors and refuse to boot, and you'll end up having to manually delete the process anyway.

Async configuration

One other opinionated feature I've included in this template is a global namespace for re-usable code.

The reason I did that is two-fold:

There are very often globally configured resources, like database connections, that are shared all across the application - that require async setup when the application launches
There is also often utility code that is shared across the application - that is useful in other contexts, e.g. the debugger (or a repl console)

There are other ways of achieving this than making a global namespace - but I thought it may be more informative to show a specific style of async setup with PM2/Express.

So here is the thinking behind what's going on.

The global backend utility namespace

I expose a global.blib namespace - which is not global.lib, specifically because this setup combines NextJS with Express: with NextJS SSR, React code runs on the backend - thus, if lib is defined on the backend and front end, there will actually be a naming conflict leading to surprising results.

All re-usable/shared backend code lives under app/blib. The logic of pulling in the library is housed under app/blib/_blib.js, so the responsibility of keeping track of files can be encapsulated in the module. Another way of achieving this would be with a package.json file - but I opted for raw JS.

One reason the raw JS is handy is because the initialization logic works well in that same _blib.js file.

Other than pulling in libraries, it also exposes async init() and aynsc cleanup() functions.

Setting up and tearing down the library

The init and cleanup functions naturally plug into the PM2 lifecycle discussed above.

init runs before process.send('ready'):

const blib = require("./app/blib/_blib.js")
// ...
  /*
    If you don't like globals, you can always opt out of this.
    I find it easier to have consistent access across the application
    to often-invoked functionality.
  */
  global.blib = blib
  /*
    This is the only other global I like to expose - since logging is
    most common and most verbose.
  */
  global.log = blib.log
// ...
  /*
    Usually this will at least open database connections.
    In the sample code, a simple in-memory store is initialized instead.
  */
  await blib.init()
  const server = new Server(process.env.EXPRESS_PORT)
  await server.start()
  if (process.send) {
    process.send('ready')
  }
// ...

and cleanup is done in the SIGINT handler:

  process.on('SIGINT', async () => {
    try {
      await server.stop()
      await blib.cleanup()
      process.exit(0)
    } catch {
      log.app.error("Sonething went wrong during shutdown")
      process.exit(1)
    }
  })

Sessions: cookies vs localStorage - which one to use when

Alexey — Sat, 12 Feb 2022 18:58:05 +0000

Preamble

Have you ever debated whether to use cookies or localStorage for sessions?

I have spent many hours on many projects contemplating which one to use. For example, I've found this guideline from a stackoverflow post compelling:

If you create a Web application you need to deal with XSS; always, independently of where you store your tokens

If you don't use cookie-based authentication CSRF should not even pop up on your radar so it's one less thing to worry about

Generally, both options are considered viable. E.g. the Auth0 team recommendation is making tokens stored in localStorage shorter lived.

I was never fully satisfied with the distinction. After all my research it seemed the choice is somewhat arbitrary and the security argument isn't that strong; both can be made to work.

The dividing line

There is a stronger decision boundary than the security argument. Security is just something you'll need to deal with depending on what you choose, but there is a dividing line rooted in functionality.

Part 1: where localStorage wins: microservices, SPAs, pre-compiled HTML

Suppose we have several API services that serve very different purposes - e.g. a payment service, and a persistence API - which operate on the same user database.

Let's say the services require authentication with each API request.

This immediately sets up a dividing line: if the services live on different domains, cookies shouldn't be used (as they are not shared across domains).

As long as all services live on the same domain, cookies can work fine... However, they will still cause issues with local development if microservices are run on different ports - since each port is its own domain. This is solvable e.g. with a local nginx setup, but that makes the configuration for running locally more complex.

So, in a microservice context, localStorage/sessionStorage has these advantages over cookies for storing authentication credentials:

Simpler local development for locally run microservices: different ports off localhost don't share cookies
Cookies couple authentication logic and service architecture (by forcing services that share authentication onto a single domain), while localStorage does not

Part 2: where cookies win: server-side rendering

Now suppose you have a server that serves HTML from templates that it pre-populates with data.

Some of that data may depend on your session - e.g. maybe we're just putting the user name into the page.

In that case, cookie sessions are the way to go: the web page HTTP GET request that happens automatically when you load a URL in a browser does not leave room for injecting an authorization header [from localStorage], but comes with all the cookies for the domain.

Part 3: what if you have microservices AND want server-side rendering

This was my exact case - using NextJS's SSR capabilities, and a standalone cross-application API service.

I ended up backing out of server-side rendering (or rather pre-fetching log in data for the template), and instead exposing a GET /profile endpoint that would get the session parsed out from headers via the common session middleware, and returns a JSON with the logged in/out state data.

This impled a loading phase on the front end where after initial delivery, it sends out an HTTP request to fetch the profile, and only then finishes rendering.

Of course it's also possible to write both localStorage and cookies, but I decided against that as I prefer having a single source of truth when possible - but if the preload state is very undesirable, it could be worth it.

Or perhaps this consideration will make you think twice about extracting a microservice at all until it's really necessary!

Conclusion

I find the above considerations very illuminating when considering whether I am ready to split out a microservice for an authenticated application, and gives me a better understanding of my toolbox of monoliths/microservices/SSR/SPAs/preloaders: I feel I am better able to build an elegant architecture based on the complexity of the project, since certain decisions pull in more and more complexity, and different forms of modularity come with different costs - for the architecture, developers, and end user.

Little known vulnerability with ORMs and query builders

Alexey — Wed, 27 Oct 2021 05:45:16 +0000

This post is about an interesting problem that relates to crossing the boundary between a SQL database and code; spoiler alert: it has to do with a little known yet broadly relevant security pitfall.

The situation

Suppose you have a user database with extremely sensitive data, and also data that is less sensitive.

You wish to protect your most sensitive data, so you restrict access to it to some database users. Only privileged database credentials - available to a few micro-services and employees at the company - can grant read access to the data.

For example, in a typical SQL database like Postgres, this is possible through GRANT/REVOKE:

GRANT SELECT ON TABLE secret_user_data TO privileged_user;
REVOKE SELECT ON TABLE secret_user_data FROM basic_user;
GRANT ALL PRIVILEGES ON TABLE basic_user_data TO basic_user;
GRANT ALL PRIVILEGES ON TABLE users TO basic_user;

Now imagine how the primary keys work on those tables: users has an ID for sure; secret_user_data and basic_user_data are 1-to-1 with users via a user_id foreign key - so do they really need their own primary key?

I would argue no, it would never be useful, because the user_id column fully determines the row; you'd never need to query on a separate id field. So these side tables should inherit the primary key from users.

The problem

Now we're going to be writing this data from a server - be it Node, Sinatra, Flask, or any framework/language whatsoever.

The problem that exposes is that of mass assignment. For example, the Ruby ORM library Sequel goes to some lengths to deal with it - but most popular frameworks are in a similar boat.

Mass assignment can happen when user input is used in write queries. For example, if your query builder takes in maps, and your POST endpoint accepts JSON which is converted to a map, perhaps whatever you put into the map gets written into the database.

A particularly nasty accident can happen if the primary key field (in our case, user_id) is overriden (imagine something like

db.secret_user_data.update({user_id: currentUser.id}, data)

, where data = {user_id: -5}). Our side table entries would be orphaned from their users, and our data scrambled.

Solution

Sequel's solution is to "restrict primary keys": throw an exception if a primary key is included in a write. Perhaps your framework doesn't have this type of security measure in place - perhaps you even have the vulnerability somewhere in your codebase! But I'm sure all members of all your teams are responsible about writing only select fields and not the literal user input, even when pressed against deadlines.

In the case I outlined, it's not possible to restrict primary keys. We need to actually write the primary key when we create the entry - but we definitely want the protection against overriding them after they are created.

Luckily, this is easy to achieve at the database level. For example, in Postgres:

CREATE OR REPLACE FUNCTION immutable_id()
RETURNS TRIGGER AS
  $BODY$
    BEGIN
      -- Alternatively, we can just fail silently
      -- NEW.id = OLD.id;
      IF NEW."id" IS DISTINCT FROM OLD."id"
      THEN
        RAISE EXCEPTION 'Refusing to update id column';
      END IF;
      RETURN NEW;
    END;
  $BODY$
LANGUAGE PLPGSQL;

CREATE TRIGGER secret_user_data_immutable_user_id
BEFORE UPDATE OF "user_id"
  ON "secret_user_data"
FOR EACH ROW
EXECUTE PROCEDURE immutable_id();

No restricted primary keys, no mass-assignment ID override vulnerability - win-win! We can have our granular security, minimal id structure, and any syntax we desire w/o fear of scrambling our data.

The rest of the fields should probably still receive application-level care - but luckily that just means there's still a job for good engineers!

Express + NextJS - sample/tutorial integration

Alexey — Sun, 20 Jun 2021 21:41:00 +0000

Context

While NextJS is a wonderful tool in its own right, augmenting it with Express makes for a powerful combo.

One motivation may be simplicity - if you have a project you're trying to prototype and rapidly iterate on. These days, it's common to host the front end separately from the API, but then your project starts off as a distributed system - and you have to deal with extra complexity up front.

Some other use cases where it makes sense to do this type of combination:

Enabling an existing Express API server to serve some front end with React/SSR
Run some express middleware and fetch standard data for NextJS pages before they are served
Adding custom logic to NextJS routing
Adding WebSocket functionality (e.g. for a chat app)

This type of setup is documented in NextJS itself: https://nextjs.org/docs/advanced-features/custom-server

In the standard example, they use Node's http package; we'll use Express to take advantage of its middleware and routing capabilities.

Source code

I've provided an example barebones integration - as a github template - at https://github.com/alexey-dc/nextjs_express_template

I also wrote an article on how to make this type of setup production-ready with PM2: https://dev.to/alexeydc/pm2-express-nextjs-with-github-source-zero-downtime-deploys-n71

Using that setup, I've hosted the demo at https://nextjs-express.alexey-dc.com/ (it's just the template run on a public URL). The main difference with the code explained here is the PM2 configuration, which I use for zero downtime deploys.

The integration

Let's take a look at some highlights of this NextJS+Express setup.

The main entry point is index.js, which sets up the environment and delegates spinning up the server:

require("dotenv").config()
const Server = require("./app/server")
const begin = async () => {
  await new Server(process.env.EXPRESS_PORT).start()
  console.log(`Server running in --- ${process.env.NODE_ENV} --- on port ${process.env.EXPRESS_PORT}`)
}
begin()

Note that I'm relying on dotenv to load environment variables - e.g. EXPRESS_PORT, NODE_ENV, and a few others. You can see the full list of necessary environment variables in the README in the github repository.

In the server, both nextjs and express are initialzed, along with express midleware and a custom NextjsExpressRouter I built to take the routing over from NextJS into our own hands:

  this.express = express()
  this.next = next({ dev: process.env.NODE_ENV !== 'production' })
  this.middleware = new Middleware(this.express)
  this.router = new NextjsExpressRouter(this.express, this.next)

The middleware I included is quite barebones, but serves as an example of what you might have in a real application:

  this.express.use(bodyParser.json());
  this.express.use(bodyParser.urlencoded({ extended: false }));
  this.express.use(favicon(path.join(__dirname, '..', 'public', 'favicon.png')));

The NextjsExpressRouter is really the heart of the integration. Let's take a closer look.

NextjsExpressRouter

The idea is to allow GET routes for pages to co-exist with API HTTP routes:

class NextjsExpressRouter {
  constructor(express, next) {
    this.express = express
    this.next = next
  }

  async init() {
    this.initApi()
    this.initPages()
    this.initErrors()
  }

  initApi() {
    return (new (require("./routes/api.js"))(this.express)).init()
  }

  initPages() {
    return (new (require("./routes/pages.js"))(this.express, this.next)).init()
  }
// ...
/* Some standard error handling is also included in the repo code */
}

I split out the API from the page routes into separate files, and I find that as the codebase grows, it helps to impose some sort of grouping or hierarchy on endpoints. Pages and API calls seem like the most basic organization. Note I made the init() function async. In this case we don't need to run any I/O operations or other async initialization, but in the general case we might want to.

For my larger projects, the API typically itself has several sub-groups, and sometimes pages do as well. In this sample project that has very few routes, the API and pages are a flat list of routes:

const data = require("../data/integer_memory_store.js")

class Api {
  constructor(express) {
    this.express = express
  }

  init() {
    this.express.get("/api/get", (req, res) => {
      res.send({  i: data.value })
    })

    this.express.post("/api/increment", (req, res) => {
      data.incr()
      res.send({ i: data.value })
    })
  }
}

Obviously this is just a minimal sample API - all it does is allows you to read and increment an integer stored in memory on the server.

Here's how the NextJS page routes are defined:

const data = require("../data/integer_memory_store.js")

class Pages {
  constructor(express, next) {
    this.express = express
    this.next = next
  }

  init() {
    this.initCustomPages()
    this.initDefaultPages()
  }

  initCustomPages() {
    /* With a monolith api+frontend, it's possible to serve pages with preloaded data */
    this.express.get('/preload_data', (req, res) => {
      res.pageParams = {
        value: data.value
      }
      return this.next.render(req, res, `/preload_data`)
    })

    /* Special-purpose routing example */
    this.express.get('/large_or_small/:special_value', (req, res) => {
      const intValue = parseInt(req.params.special_value)
      if(isNaN(intValue)) {
        return this.next.render(req, res, `/invalid_value`, req.query)
      }
      if(intValue < 5) {
        return this.next.render(req, res, `/special_small`, req.query)
      } else {
        return this.next.render(req, res, `/special_large`, req.query)
      }
    })
  }

  initDefaultPages() {
    this.express.get('/', (req, res) => {
      return this.next.render(req, res, `/main`, req.query)
    })

    this.express.get('*', (req, res) => {
      return this.next.render(req, res, `${req.path}`, req.query)
    })
  }
}

module.exports = Pages

The page routes showcase setting up a root / path, and a fallback * path - if we're not able to match the GET request, we default to what NextJS's standard behavior is: rendering pages by filename from the /pages directory. This allows for a gentle extension of NextJS's built-in capabilities.

There are 2 examples for custom routing.

In the first example, we pre-load some data, and bake it into the page before serving it to the user. This may be useful to avoid an extra HTTP roundtrip after the page renders, and is difficult to pull off w/o a monolithic API+frontend setup as presented here.

In the second example, we render different variants of a page depending on an integer value in the route - or an error, if the input is invalid. Perhaps a real application may fetch user data, and render it differently depending on some condition (e.g. the viewer's relationship with them) - and render an error if the user is not found.

Using the template

I licensed the code under MIT - which means you are free to use the template in closed-source and commercial products, and make any modifications you want. Please attribute/give credit if you are able to!

It's also a template on github, which means you can just click a button and start a new repo based on https://github.com/alexey-dc/nextjs_express_template

Running

The instructions for running are in the github repo.

Iterating

You'll probably want to delete the sample custom endpoint and associated pages I provided - and start replacing them with your own!

I included a sample organization for pages as well - the page roots are in pages as nextjs mandates, but all the reusable jsx is in views - for the demo, I was using a common layout for pages, and the Layout component is housed in views.

Javascript repl console

Alexey — Sun, 18 Apr 2021 17:25:27 +0000

Premise

I often want to run ad hoc Javascript commands that rely on npm packages and custom classes I've written that work with a database/filesystem or wrap common logic.

Node comes with repl (Read-Eval-Print Loop), and you can launch a simple repl console by just running node with no arguments - the commands for it are documented in e.g. https://nodejs.org/api/repl.html#repl_design_and_features. That's quite handy - but falls short of a full-featured interactive shell that has access to all necessary packages.

The solution

Luckily, repl is available in node as a package ( https://nodejs.org/api/repl.html#repl_repl ) - so all that's necessary is to write a small script that starts a repl instance and pulls in everything you need.

You'll need to inject all the packages you want to use interactively into the repl console via a launcher script. It's also handy to configure repl in the script, and I show some examples below:

/*
  Opinionated example on how
  to make the repl console environment aware.
*/
require("dotenv").config()
/*
  If you intend to launch the console
  across multiple environments (development/production/staging) -
  it's helpful print the environment
  to avoid unfortunate mistakes.
*/
console.log(`Starting console - ${process.env.NODE_ENV}`)

const repl = require("repl")
const util = require("util")

const startConsole = async () => {
  /*
    The lines below configure output formatting for repl.

    W/o specifying any output options, you'd get
    formatting like
    > a = {a: {b: {c: {d: {e: {f: {g: {h: 1}}}}}}}}
    { a: { b: { c: [Object] } } }

    With these options, you'd get
    > a = {a: {b: {c: {d: {e: {f: {g: {h: 1}}}}}}}}
    { a: { b: { c: { d: { e: { f: { g: { h: 1 } } } } } } } }

    Note these options are the same as the options passed to inspect
    https://nodejs.org/api/util.html#util_util_inspect_object_options
  */
  util.inspect.defaultOptions.depth = 20
  util.inspect.defaultOptions.colors = true
  util.inspect.defaultOptions.getters = true
  util.inspect.defaultOptions.compact = true

  /*
    repl is supposed to use util.inspect to format by default.
    However, w/o explicitly specifying {writer: util.inspect},
    I was not able to get the options above to be successfully applied
    for eval results formatting. They _do_ get applied to
    console.log formatting, though, in either case.

    You may want to specify other options - see
    https://nodejs.org/api/repl.html#repl_repl_start_options
    for a comprehensive list - e.g. {prompt: "xyz>"} is a handy one.
  */
  const replServer = repl.start({writer: util.inspect})
  /*
    Pull in any number of modules here - these are the
    modules that will be available to you in the repl instance.
  */
  const modules = ["util", "fs"]
  modules.forEach((moduleName) => {
    replServer.context[moduleName] = require(moduleName)
  })
  /*
    This is not necessary in newer versions of node,
    but in older versions I wasn't able to pull in
    ad-hoc modules to a running repl instance w/o it.
  */
  replServer.context.require = require
}

startConsole()

The way I personally set it up is by having all the things my application cares about available as a single module defined in my application - including both npm packages and my own library/reusable code.

I use this single module in application code, scripts, background jobs, and also in the repl console - that way accessing functionality looks the same in all contexts, and I can easily memorize commands and have them at my fingertips.

My script ends up looking more like this:

require("dotenv").config()
console.log(`Starting console - ${process.env.NODE_ENV}`)

const repl = require("repl")
const util = require("util")
/*
  This contains all the modules I want to pull in
*/
const lib = require("../lib.js")

const startConsole = async () => {
  /*
    E.g. establish connections to various databases...
  */
  await lib.init()

  util.inspect.defaultOptions.depth = 20
  util.inspect.defaultOptions.colors = true
  util.inspect.defaultOptions.getters = true
  util.inspect.defaultOptions.compact = true
  const replServer = repl.start({writer: util.inspect})

  for(key of Object.keys(lib)) {
    replServer.context[key] = lib[key]
  }
}

startConsole()

Starting the console

I usually start the script through npm/yarn, via package.json:

...
  "scripts": {
    ...
    "console": "node --experimental-repl-await ./scripts/console.js"
    ...
  },
...

I love --experimental-repl-await (https://nodejs.org/api/cli.html#cli_experimental_repl_await - added in Node.js 10.0.0), and I hope it makes its way out of experimental soon. It allows awaiting on async commands in the repl console. Without it, working with promises is quite annoying.

After that's in, it's just yarn run console or npm run console.

Working with the console

yarn run console
> console.log("Hello world")
Hello world
undefined

Note how console.log("...") produces 2 lines as output. It performs its side effect of printing and returns a value - and repl will print the result of each expression it evaluates. For example, variable declarations return undefined, but variable assignments return the assigned value:

> let a = 1
undefined
> a = 2
2

That's handy to know if you want to skip printing the output of some expression.

In most cases, I tend to avoid using variable declarations in repl, since you can assign a variable without declaring it. The reason is that I often copy-paste sections of code from a text editor, and variable declarations are not re-runnable. In application code I'll usually use const, but in repl that locks you out from fixing mistakes, especially with e.g. function declarations.

> let a = 1
undefined
> let a = 1
Uncaught SyntaxError: Identifier 'a' has already been declared
> b = 1
1
> b = 1
1

Persistent history

Repl supports bi-directional reverse-i-search similar to zsh. I.e. you can search back through your history by pressing ctrl+r (or ctrl+s to search forward) - which makes preserving history between runs potentially very worth it.

History is preserved in a file, so you'll need to choose where to store it. I store it in a .gitignored folder in my project. E.g. the default node.js repl console stores history by default, in your home folder in .node_repl_history ( https://nodejs.org/api/repl.html#repl_persistent_history ).

Here's the code for enabling persistent command history - the path is relative to the root of the project ( https://nodejs.org/api/repl.html#repl_replserver_setuphistory_historypath_callback ):

replServer.setupHistory("./no_commit/repl_history", () => {
  console.log("Loaded history!")
})

I add this at the end of the startConsole() function above, adding the environment as the filename suffix:

require("dotenv").config()
console.log(`Starting console - ${process.env.NODE_ENV}`)

const repl = require("repl")
const lib = require("../index.js")
const util = require("util")

const startConsole = async () => {
  await lib.init()

  util.inspect.defaultOptions.depth = 20
  util.inspect.defaultOptions.colors = true
  util.inspect.defaultOptions.getters = true
  util.inspect.defaultOptions.compact = true
  const replServer = repl.start({
    writer: util.inspect,
    prompt: "> "
  })

  for(key of Object.keys(lib)) {
    replServer.context[key] = lib[key]
  } 

  const historyPath = `./no_commit/repl_history_${process.env.NODE_ENV}`
  replServer.setupHistory(historyPath, () => {})
}

startConsole()

Conclusion

It's quite easy to set up an interactive Javascript shell based on Node's REPL module. It can be configured flexibly, have access to application logic, and any installed npm modules.

Unlike a Chrome console, it can be used to run arbitrary commands on your computer (or a remote computer), and not just for working with a particular application.

Quick manual deploy to EC2 with github

Alexey — Fri, 12 Mar 2021 03:54:52 +0000

This article is about my preferred way to bootstrap a deploy process ASAP (with AWS/EC2). It is most useful for prototyping ideas or publishing solo projects. For more mature projects, I've used a variety of other deploys styles - which are all more expensive to set up.

I use a simple on-demand instance in EC2 with SSH, and use git clone/git pull via deploy keys - then directly running the server the same way I work with it locally (except with a different set of environment variables).

I pair this with an nginx/SSL configuration with free SSL certificates via certbot: https://dev.to/alexeydc/modern-https-configuration-2h86

Deploy keys

Github has a notion of deploy keys, designed for this very purpose.

Step 1: Generate keys

On the machine you want to deploy to, run:

ssh-keygen -t ed25519 -f ~/.ssh/my_project_deploy_key -C "your_email@example.com"

Note the -t ed25519 - an upgrade from RSA to elliptic curve cryptography. More info: https://medium.com/risan/upgrade-your-ssh-key-to-ed25519-c6e8d60d3c54.

In this case, we're only going to use the key to pull the repo on a deploy - so you don't really need the -C flag. But e.g. if you develop on a remote machine and plan to commit with the key - it's helpful to tie it to your github email.

Step 2: Add the public key to github

In the target repository, go to Settings -> Deploy keys.

Click Add Deploy Key.

Give it a descriptive title, like "Production MyProject EC2". Copy-paste the contents of ~/.ssh/my_project_deploy_key.pub into the key field, don't tick Allow write access (or do - but for deploys you shouldn't want to).

Step 3: Configure git to use the key

Actually, if we had named the key id_rsa, git would already have access to it (just make sure to clone via the git: URL, not https:).

However, then you are limited to just one repository for this machine, and... are forced to use the default key name.

Instead of being handcuffed by defaults, let's set up the config ( https://snipe.net/2013/04/11/multiple-github-deploy-keys-single-server/ ):

vim ~/.ssh/config

# Then add this type of config to the file
Host github.com-any_string_identifying_our_service
HostName github.com
IdentityFile /home/ubuntu/.ssh/my_project_deploy_key
User git

# You can have entries for multiple repos
Host github.com-another_string_for_another_project
HostName github.com
IdentityFile /home/ubuntu/.ssh/other_deploy_key
User git

Then you need a special clone command, using the host defined above. Let's say our github organization name is shiny_deploys_inc, and the repository name is deploy_alpha. The web URL for the repo would be https://github.com/shiny_deploys_inc/deploy_alpha. Here is the clone command, given the example config above:

git clone git@github.com-any_string_identifying_our_service:shiny_deploys_inc/deploy_alpha.git

After that, git pull should work as usual to update the project.

What next?

This barebones setup can work quite well to get started, but it's far from being complete.

Each deploy will involve a bit of downtime. One way to get around that is with pm2 - check out my article on setting up a rolling deploy with pm2 that helps mitigate downtime due to backend updates https://dev.to/alexeydc/pm2-express-nextjs-with-github-source-zero-downtime-deploys-n71.

Modern HTTPS configuration

Alexey — Wed, 10 Mar 2021 01:34:58 +0000

I've tried many ways of setting up HTTPS servers, and I've finally found a favorite method.

Instead of paying for production certificates, it's easy to verify your own certificates via cerbot https://certbot.eff.org/ and LetsEncrypt https://letsencrypt.org/.

The flow below is for Ubuntu, and involves using nginx to serve a file - as opposed to having the file served by your actual backend. I find this solution to be more elegant if you have full access to your server.

Installing cerbot and receiving a certificate

1. Install certbot

For ubuntu 20:

sudo snap install core; sudo snap refresh core
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot

Earlier versions:

sudo add-apt-repository ppa:certbot/certbo
sudo apt-get update
sudo apt-get install certbot

2. Run certbot

sudo certbot certonly --manual

This will stop at a prompt; keep it open and follow the next steps.

3. Set up nginx to serve the right data for certbot

# Snap didn't have nginx when I was doing this setup, so:
sudo apt install nginx
sudo ufw allow 'Nginx HTTP'

(with reference to https://docs.nginx.com/nginx/admin-guide/web-server/serving-static-content/ ):

# By default nginx will serve files from /var/www/html
# Put the cert there by default, or see what works best for your setup:
sudo mkdir /var/www/html/.well-known
sudo mkdir /var/www/html/.well-known/acme-challenge
sudo vim /var/www/html/.well-known/acme-challenge/<filename from certbot>
<copy in certbot data>
sudo chmod a=r /var/www/html/.well-known/acme-challenge/<filename from certbot>

# We don't need to change anything with the above folder structure.
# Alternatively, we can change the config
sudo vim /etc/nginx/sites-enabled/default
# If you do change the config, reload nginx
sudo systemctl reload nginx

4. Finalizing verification

Go back to certbot; it should be prompting to hit Enter. Do that, and the verification should complete:

 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/yourdomain.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/yourdomain.com/privkey.pem
   Your certificate will expire on 2021-06-07. To obtain a new or
   tweaked version of this certificate in the future, simply run
   certbot again. To non-interactively renew *all* of your
   certificates, run "certbot renew"

Using the certificate

The newly created certificates will only be available to root https://certbot.eff.org/docs/using.html#where-are-my-certificates

sudo chmod 0755 /etc/letsencrypt/{live,archive}
# In the doc above, this isn't mentioned as necessary, but I couldn't get access to the privkey w/o being explicit
sudo chmod 0755 /etc/letsencrypt/live/yourdomain.com/privkey.pem

Now, you can either choose to use these certificates directly by your service, or let nginx deal with that layer.

Configuring nginx

To make a server available on HTTPS w/o a port suffix, it's necessary to run on port 443. That requires elevated privileges in linux, and it's not a good idea to run Node.js that way - though nginx is perfectly suited just for this.

A good way to set up port-less access is to configure port forwarding via nginx: from 443 to e.g. 8080 - you can connect from nginx to your service directly via HTTP w/o SSL.

I usually create a file in sites-available, then symlink it inside of sites-enabled:

sudo vim /etc/nginx/sites-available/yourdomain.com

Let's say you're running your application server on port 8080 (you can use HTTP with your application, since nginx will handle the HTTPS handshake/encryption):

server {
  listen 443 ssl http2;
  server_name yourdomain.com www.yourdomain.com;
    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem; # managed by Certbot

  include ssl_config;
  access_log            /var/log/nginx/yourdomain.access.log;

  location / {
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;

    # Fix the "It appears that your reverse proxy set up is broken" error.
    proxy_pass          http://localhost:8080;
    proxy_read_timeout  90;

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Host $host;
  }

}

server {
    if ($host = yourdomain.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


  listen 80;
  server_name yourdomain.com www.yourdomain.com;
  return 301 https://yourdomain.com$request_uri;
}

Don't forget to replace yourdomain.com with your actual domain.

After the config is in sites-available, it should be symlinked into sites-enabled:

sudo ln -s /etc/nginx/sites-available/your-domain.com  /etc/nginx/sites-enabled/your-domain.com

If you just have 1 site, you could use sudo vim /etc/nginx/sites-available/default - just append the config above to that file.

Once the config is set up,

sudo systemctl reload nginx

HTTPS in local development

Usually, you can rely on HTTP in local development - since browsers special-case behavior for http://localhost, and don't require the same degree of security as with remote hosts.

There are some rare situations where HTTPS locally is useful - e.g.
some OAuth providers that you may need to redirect to your site from may disallow http redirect URLS.

In those cases, mkcert can be very handy ( https://github.com/FiloSottile/mkcert ):

# Only do this once for all mkcert projects
brew install mkcert
brew install nss # for Firefox
mkcert -install
# Set up this repo with mkcert certificates
# I personally just keep my mkcert right in the folder of the repo.
# Don't forget to add the directory to .gitignore!
mkdir mkcert
cd mkcert
mkcert localhost

After that you'll need to get your application server to load those certificates and set up HTTPS - I'll leave that out of scope for this article.

Maintenance

The issued certificates are good for 3 months. To get the time remaining, run:

sudo certbot certificates

To re-issue fresh certificates manually, run:

sudo certbot --force-renewal

This will renew all certificates from certbot (i.e. it's intended to support multiple certificates and services on one machine).

Certbot is built to be automated - so choose your own style, set up a crontab if you like. The general-purpose renew command is

sudo certbot renew

For more information, see https://certbot.eff.org/docs/using.html?highlight=renew#renewing-certificates