DEV Community: Anchor

Developing CLIs

Wesley Beary — Mon, 12 Aug 2024 19:00:00 +0000

Building CLI features often boils down to:

Release new API endpoint(s).
Build new CLI actions that need the API changes.
Realize mistakes were made in the API you just released.
Try to fix the API for this feature without creating more problems in the future.
Try to remember what you were trying to achieve in the CLI and then actually achieve it.
GOTO: realize mistakes were made.

Each step requires the one before and oops we've reinvented waterfall project management. You get worn down by the pain trying to gracefully pave over mistakes until you limp to functional, but bow out well before exceptional. And don’t get me started on maintaining the resulting cluster of ad-hoc “fixes” and warts.

Been there, done that. We knew we needed to move past the waterfall-ish approach.

Here is the story of how we got there and some of the tools that helped along the way.

Off to a Sketchy Start

We wanted cheap, fast iteration until we understood the feature, and only then commit to expensive implementation and long-term support. As a small team, I was often doing this process end to end, and wanted to focus on each part in turn. We wanted to fake implementation parts until we felt confident enough to make it.

Getting back to the process, it starts with proposing features. We wanted to get out of the abstract, but not if it meant half-baked implementation. We faked it with "sketches", inspired by the Google Docs CLI sketch approach that Github describes here.

Unfortunately, static sketches didn't quite give us the feedback we wanted. Our CLI changes output over time, more like animation than a drawing. To achieve higher fidelity, I wrote little Ruby programs to take basic inputs and respond by printing appropriate canned responses.

Since then we've found an even better way to capture animated CLI output, but to explain that requires a little detour.

Do You Even Test?

As we started to flesh out our CLI, we also wanted to test edge cases and detect regressions. I surveyed public cobra/bubbletea based CLIs to look for ideas, and found frustratingly few tests. Then we stumbled upon Charm's teatest which gave us a starting point.

Teatest focuses on golden tests, capturing a known good output and then asserting that future outputs continue to match it. Which brought us back, once again, to high fidelity capture of animated CLI outputs. Teatest gave us the great idea of a frame-based solution, like a flipbook, which we have built upon:

─── SigninHeader ───────────────────────────────────────────────────────────────
# Signin To Your CLI Account `cli auth signin`
─── SigninInput --──────────────────────────────────────────────────────────────
# Signin To Your CLI Account `cli auth signin`
    ? What is your username?
    ? user
─── SigninInput ────────────────────────────────────────────────────────────────
# Signin To Your CLI Account `cli auth signin`
    * Signing in to your CLI account… ⠋
─── SigninInput ────────────────────────────────────────────────────────────────
# Signin To Your CLI Account `cli auth signin`
    * Signed in to your CLI account: user@example.com

This simplified example shows what a golden output might look like for a basic authorization command. The horizontal lines delineate frames, with labels indicating the active model. Taken together we get a high fidelity capture of output even as lines are added, removed, or replaced.

We use a flag in our test suite to update files with the golden outputs, and otherwise the tests fail if the output doesn't match the files. This keeps us aware of output changes and facilitates PR reviews by allowing us to understand what the output should look like and if it has changed. We like it so much that we plan to replace our sketch programs with golden style output in Github style google docs so we can capture both animation and style ideas.

With our once and future sketches in hand, let's return to getting started with new API endpoints.

Designing API and CLI Together

We work on the API and CLI simultaneously, because the best designs for these grow out of tight integration. We are able to do this, while still avoiding the perils of waterfall, by iterating on designs in cheaper contexts and waiting to implement until the requirements solidify. For our APIs, this means sketching with OpenAPI:

openapi: 3.1.0
info:
  contact:
    email: contact@example.com
  description: An example API.
  title: Example API
  version: 0.0.1
servers:
  - url: https://api.example.com
tags:
  - description: account operations
    name: account
paths:
  '/v0/auth/signin':
    post:
      description: Signin to CLI.
      operationId: auth_signin
      responses:
        '200':
          content:
            'application/json':
              schema:
                additionalProperties: false
                properties:
                  email:
                    description: Email address for authenticated user.
                    example: username@example.com
                    type: string
                required:
                  - email
                type: object
          description: Successful signin.
      summary: Signin to CLI.
      tags:
        - account

This simplified example shows what a schema might look like for a basic authorization command. We use the spectral linter to simplify working on these files.

With a sketch in hand, we then use prism as a mock API server while we implement the CLI. When we inevitably realize mistakes were made, we can just tweak the spec and get back to our CLI iteration. Working at this high level allows us to evolve the API and CLI together and defer costly implementation until we have better knowledge.

Implementing APIs

We also lean into our OpenAPI spec to keep us honest during implementation using committee. assert_schema_conform tests alignment and the middleware notifies us of any live discrepancies. These combine to allow for red green implementation while protecting us from regressions.

Testing with Mocks and Proxies

To round things out, our test suite uses flags to run prism in either mock or proxy mode. By using flags we can focus on writing just one kind of test, though it does mean we skip some tests in one mode or the other. We use mock tests for their speed and on Windows and macOS where our full stack doesn't run in CI. Our proxy tests let us run tests against our entire stack just by adding a flag, making it easy to run end to end testing whenever we deem it necessary.

Pulling it All Together

Sketches and specs help us iterate past the abstract without having to get bogged down in implementation. Then mocks and proxies help us ensure the implementations match the sketches. By continuing to iterate our process, each feature causes less pain, which we have deeply appreciated while building the teams experience we will deliver later this month.

We will keep iterating on our process, I hope you learned something from it and I would love to learn from you. What have you tried, where are you proud and what remains frustrating?

HTTPS on Localhost with Next.js

Ben Burkert — Mon, 29 Apr 2024 16:04:00 +0000

Next.js is a popular framework for building web applications with great development support and tooling. However, its local development story has a plot hole: HTTPS support. While the next dev command has a --experimental-https flag to enable HTTPS locally, it’s rather limited.

Anchor’s new lcl.host service fills this hole and even works with Docker containers. This article will explain what lcl.host is, how it compares to Next.js’ experimental HTTPS flag, and how to configure your Next.js application to work with lcl.host.

What is lcl.host?

lcl.host is an easy way to enable HTTPS in local development environments, which improves the security of the development process, ensures feature parity between development and production environments, and enables features like CORS that behave differently on localhost.

There are multiple ways to use HTTPS on a local machine. You need to get a certificate for your server, renew it when it expires, and ensure your clients trust the CA that issued that certificate. If you get a certificate from a public CA, your clients automatically trust it, but your development server must be public, too. If you set up a local CA, your server can be private, but your clients need updates to trust your CA.

lcl.host takes care of all this for you by providing a CA for development, ensuring your browsers and clients trust this CA, issuing certificates for your apps, and renewing them when they expire.

What are the Benefits of Using lcl.host over Next.js’ Experimental HTTPS Flag?

Next.js offers an experimental flag that enables HTTPS locally. While lcl.host requires a bit more setup, it also brings several benefits over the flag.

lcl.host Works With All Next.js Commands

To use the experimental HTTPS support, add the --experimental-https flag every time you run next dev command.
With lcl.host you can use all next commands without a flag, and HTTPS works transparently with them.

lcl.host Works With HTTP and HTTPS

While the experimental HTTPS flag only listens on HTTPS, lcl.host allows using HTTP and HTTPS simultaneously, so you can switch without restarting your server.

lcl.host Works With Docker Containers

The experimental HTTPS flag relies on mkcert, designed for a single development system. If you run a Docker container, the flag won’t configure your local browser to trust its certificate.

lcl.host configures your browser on your system to trust the certificates in all containers you run locally.

lcl.host Works With Subdomains

Experimental HTTPS only works with localhost; if you have multiple applications, they all run on localhost with different ports. lcl.host doesn’t limit you to https://localhost:<PORT>/; you can use any subdomain of lcl.host, allowing you to run multiple apps simultaneously.

lcl.host Renews Certificates Automatically

Experimental HTTPS might not take expired certificates into account.

lcl.host handles automatic certificate renewal for you.

lcl.host Integrates With the Experimental HTTPS Flag

The experimental HTTPS flag works with existing certificates to import the certificates generated by lcl.host with the --experimental-https-key and --experimental-https-cert flags.

How to Use lcl.host in a Next.js Project?

Now that you understand what lcl.host can do for you, the next step is to see how to integrate it with a Next.js app.

Prerequisites

1. Installing the Anchor CLI

First, you must install the Anchor CLI; this will handle the lcl.host setup for your project.

With Homebrew (MacOS and Linux):

brew install anchordotdev/tap/anchor

With Chocolatey (Windows):

choco install anchor --version=0.0.22

Build from source with Go:

go install github.com/anchordotdev/cli/cmd/anchor@latest

2. Using Anchor’s Next.js Starter Template

For a new project, you can use the Next.js-lcl.host starter template provided by Anchor. It comes preinstalled with dependencies for automatic certificate renewal and an update next.config.mjs file.

You can clone it from GitHub with this command:

git clone https://github.com/anchordotdev/nextjs-lclhost-starter.git

Using this template, you can skip to step 5.

3. Installing Dependencies

For the automatic certificate renewal, you need the latest anchor-pki package. Install it as a development dependency with the following command:

npm i anchor-pki -D

lcl.host works without this package, but when your certificates expire you must manually renew them.

4. Updating the Next.js Configuration

Now, you must tell Next.js to use the anchor-pki package every time you run your app. To do so, update the content of your next.config.mjs file:

import autoCert from "anchor-pki/auto-cert/integrations/next"
const withAutoCert = autoCert({enabledEnv: "development"})
const nextConfig = {
  // Your configurations
}
export default withAutoCert(nextConfig)

You can configure your Next.js app as usual; just ensure the config object is wrapped in the withAutoCert() call before exporting it.

5. Connecting the Project with lcl.host

To make your project available via lcl.host, open the project directory in a shell and run the following command:

anchor lcl

After the command completes, you should see a URL to a guide for the automatic renewal:

| Next Steps
  | Now that you have local HTTPS setup, let's automate certificate provisioning
  | so you never have to manually provision future certificates again.
  | 
  | We've generated an Anchor.dev setup guide for your application with
  | instructions for automating certificate provisioning inside your
  application.
  - Opened https://anchor.dev/<ACCOUNT>/services/<APP_NAME>/guide.

Open this guide in your browser, then skip to the second step, as we already performed the first two here.

Click the “Generate new tokens” button, as seen in Figure 1, and copy the tokens and other environment variables into a new .env.local file.

The content of the env file should look similar to this:

ACME_KID=aae…zWD
ACME_HMAC_KEY=72D…dMr
ACME_DIRECTORY_URL='https://anchor.dev/<ACCOUNT>/localhost/x509/ca/acme'
ACME_CONTACT="<ACCOUNT_EMAIL>+${ACME_KID}"
ACME_ALLOW_IDENTIFIERS="<APP_NAME>.lcl.host,<APP_NAME>.localhost"
HTTPS_PORT='<RANDOM_PORT>'

Next.js will use the .env.local file to set up environment variables before starting the web server.

6. Testing the Project

Finally, to test the whole setup, you can start the development server with this command:

npm run dev

If everything goes well, you should get an output similar to this:

[AutoCert] Starting auto-cert proxy ...
[anchor-pki] The 'sni-callback' Configuration instance is enabled
[AutoCert] Listening on https://<APP_NAME>.lcl.host:44356 -> http://127.0.0.1:3000
[AutoCert] Listening on https://<APP_NAME>.localhost:44356 -> http://127.0.0.1:3000
[anchor-pki] No Terms of Service exist so agreed : true
[anchor-pki] No Terms of Service exist so agreed : true
[anchor-pki] Certificate for <APP_NAME>.lcl.host provisioned.
[anchor-pki] Certificate for <APP_NAME>.lcl.host cached in (tmp/acme).

As the output tells, you can now access your app via two HTTPS-enabled URLs.
If you get this error instead, it indicates a missing or incomplete .env.local file:

[AutoCert] HTTPS_PORT is not set, assuming 4333
/path/to/your/project/node_modules/anchor-pki/src/auto-cert/configuration.js:185

Return to step 5 and double-check you didn’t miss any environment variables.

Summary

Next.js has a nifty HTTPS feature, which is nice for quickly testing an app on your local machine. However, its tricky to use with Docker or multiple applications.

lcl.host is a complete solution for HTTPS-enabled development environments. It handles automatic certificate renewal and works transparently with several setups, including Docker, multiple local applications, and all Next.js commands.

Resources

Squashing Mixed Content in Development with lcl.host

Ben Burkert — Fri, 22 Mar 2024 17:24:13 +0000

Mixed content occurs when a webpage served over HTTPS loads a resource over HTTP. Browsers block these mixed content resources to protect your security, but blocked resources can break your pages. These bugs often slip through to production because browsers specifically allow mixed content on http://localhost pages. Switching to HTTPS in local development will make our browser block mixed content and behave the same for the local and deployed version.

Getting HTTPS setup and working with an app in local development is tricky. There were two options: acquire a publicly-trusted certificate from a CA, or make your own self-signed certificate from the command line. Neither of these options are simple, that's why most developers skip HTTPS in their development environment. But lcl.host now makes this quick and easy.

lcl.host is a free developer tool from Anchor.dev to setup a local HTTPS development environment in minutes. It combines the best parts of both public CA certificates and self-signed certificates, and it's built for novice and expert web developers alike. Read more about how lcl.host works on the Anchor blog.

Now we will look at mixed content issues in practice with an example app which attempts to load a script asset over HTTP. Our browser will allow the script to load via http://localhost but will block the script over HTTPS, just as it does with deployed apps. Using HTTPS in local development let's us catch mixed content warnings and errors before we deploy and cause issues for users.

Mixed Content Example App

anchordotdev/mixed-content-example is a nodejs app that triggers a mixed content error when served over HTTPS. We'll start by setting up the app and running it with just HTTP, which won't trigger any errors at all.

$ git clone https://github.com/anchordotdev/mixed-content-example
$ cd mixed-content-example
$ npm install
$ node ./index.js
mixed-content-example app over HTTP: http://localhost:3000

Browse to http://localhost:3000/ and you'll see "INSECURE SCRIPT LOADED AND RUN!!!", demonstrating that the insecure script loaded without any errors.

This script will always be insecure because it's always served over HTTP, yet our browser allowed it through! Next let's setup and use lcl.host to serve the page over HTTPS and secure our page.

lcl.host

Setting up local HTTPS used to be a pain, but lcl.host makes it easy: install the CLI, run anchor lcl, and work through the guided steps:

$ brew install anchordotdev/tap/anchor
$ anchor lcl

For the example, we use the suggested values for all the prompts. Once you are done, restart the app, and you'll see it is now listening on both HTTP and HTTPS:

$ node ./index.js
mixed-content-example app over HTTP: http://localhost:3000
mixed-content-example app over HTTPS: https://mixed-content-example.lcl.host:44375

Now open the HTTPS version in your browser, and be sure to use the port number returned when you run the app, which may differ from what we depict here. Notice this time the "INSECURE" message doesn't appear, because your browser blocked the script for you.

If we look in the devtools console, we also can see there is now a "Mixed Content" error for the insecure simple-example.js script.

Working through this example helped us reproduce mixed content errors and catch them in our dev tools console. You can use lcl.host to catch mixed content bugs in all your web applications too.

Real World Apps

Using lcl.host with real development apps works the same as the example: run anchor lcl in the project directory and follow the guide. Check out the lcl.host docs to learn more about other issues that local HTTPS helps mitigate.