3 Commandments for CLI Design

In the beginning…

In Neal Stephenson’s 1999 essay “In the Beginning Was The Command Line”, he describes how using Command-Line Interface (CLI) utilities is simpler and requires less code than a Graphical User Interface (GUI). They’re tiny, they do one thing really well, and they are meant to work in conjunction with other small, single-purpose tools to accomplish complicated tasks quickly.

However, the old-school Unix utilities are not without their problems. The passage of time and inevitable feature creep have led to situations like this:

usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]

The humble ls command now has more options than there are letters in the alphabet! Plus, despite the proliferation of features, many advanced and useful options are off by default to preserve backwards compatibility. Other core utilities are in similar situations: find and tar are stuck with arcane, user-hostile syntaxes, and the man pages for modern distributions are littered with comments like this one from the BSD-derived macOS 10.15 grep manual: “This implementation supports those options; however, their use is strongly discouraged.” (Discouraged? By whom? Why?)

Fast Forward

Fortunately, though much remains constant in the world of command-line tools, much has changed. There’s a buzz of activity in the open-source world around re-examining the core user experience of CLIs with fresh eyes.

Relay.sh is an automation platform for event-driven devops workflows and we’re deeply committed to building awesome command-line experiences. I started writing down these principles as we began rethinking the command line tool for Relay. To me, the key UX principles that underpin the best of the new breed of projects in this vein are:

1. Enable Progressive Discovery — users are trying to solve a problem, but might start out knowing very little about the tool. The tool should guide them to a solution in iterative steps and provide plain-language help along the way. Users resorting to Google or StackOverflow is an anti-pattern here.
2. Go Upscale, Intelligently — the best tools infer the context they’re running in take advantage of modern capabilities if they can. For example, colorized text can greatly add to usability and aesthetics, but if the tool is being run in a pipeline, colors should be off to avoid polluting the output stream.
3. Be a Good CLI Citizen — there are a number of hard-fought CLI best practices that good tools follow, both to match users’ expectations and to peacefully co-exist with the rest of the ecosystem. It’s important for tool authors to be aware of these conventions, comply with as many as possible, and be deliberate about any significant divergence.

In the rest of the post I’ll provide some examples of each of these principles using CLI tools I’ve fallen in love with.

Enable Progressive Discovery

Pulumi is an infrastructure-as-code tool that allows developers to create and manage cloud infrastructure using general-purpose programming languages. While Pulumi programs can be written in Python, Typescript, .Net and so forth, the user interaction loop centers on using the pulumi CLI tool to interface between the local code repository and the Pulumi service. It’s a fantastic example of modern CLI design that exemplifies all of the principles I mentioned, but I’m going to focus on the way it enables progressive discovery.

First, installing it is a breeze because the website has helpful instructions for all the major OSes with a one-click copy-to-clipboard command to get you started.

Once you’ve got it installed, simply running the pulumi command with no arguments gives you helpful hints on what to do next in order to begin working with it. I’ve annotated the output of running pulumi with no arguments to illustrate this principle.

The output not only shows, right at the top, the first thing you ought to run if you’re new, it provides some hints as to what the second and third steps might be after you’ve begun. It links to the detailed docs on the website, invites exploration and discovery by giving the complete list of available commands, and has a reference section to show the flags which modify the program’s behavior in consistent ways, regardless of what subcommand you’re using.

Once you’ve gotten started and are working with a “stack” (a collection of resources that make up one application instance), the idea of progressive discovery shows up again. Running the pulumi stack subcommand without additional arguments doesn’t error out; instead it provides a reasonable guess as to what you meant: “show me information about the current stack”.

In addition to providing metadata about the stack and a nicely-formatted hierarchical representation of its resources, the output has links to the web app for more info and again shows hints about what to do next if this isn’t what you were looking for.

All in all, this experience is pretty delightful. It makes working with pulumi positive and enjoyable, and rather than feeling like a cut-down version of a web app it feels like a powerful first-class way of interacting with the service.

Go Upscale, Intelligently

My former colleague from Danger and all-around good human C J Silverio tweeted recently:

Ceej is sheltering under cats

@ceejbot

The rustlang ecosystem scene that’s making me happy right now: the retakes on ancient unix cli tools, with new approaches to info display.

bat
dua
exa

03:19 AM - 22 Feb 2020

3 30

This sent me down a rabbit hole of investigation and discovery. There’s indeed a vibrant community working in this space, and the ones CJ mentions are replacements for (respectively) cat, du, and ls. (There’s even a rewrite of the wc command from the Stephenson essay!) I came out the other side of the rabbit hole with a new set of aliases for my .zshrc:

# if rust stuff is found, use it
RBIN=$HOME/.cargo/bin
if [[ -d $RBIN ]]; then
  [[ -f $RBIN/bat ]] && for f in less more cat ; do 
     alias $f=bat ; 
  done
  [[ -f $RBIN/dua ]] && alias du=dua
  [[ -f $RBIN/rg  ]] && alias grep=rg
  [[ -f $RBIN/exa ]] && alias ls=exa
  [[ -f $RBIN/fd  ]] && alias fd=find
fi

These tools vary in their implementation but all of them exemplify the “Go Upscale” principle: they are aware of the affordances that modern terminal programs have and make use of them to provide upgraded experiences. To focus on one example, the lowly cat program is one of the oldest and most useful Unix utilities. Its name is short for “concatenate” and it can be used for all kinds of input and output stream manipulations, but it’s pretty primitive. bat (here’s its homepage on github) retains the usefulness of the original but adds colorized and line-numbered output, syntax highlighting for over a hundred file types, and cool features like git awareness. So bat README.md in my local hiera repository shows me:

The subtly shaded line numbers and line-drawn formatting, the + markers for uncommitted lines, and the helpful Markdown syntax highlighting are all super cool. But the thing that allowed me to alias cat=bat in my shell without fear of breaking scripts is that bat detects when it’s not printing to a terminal and turns all of that pretty-printing off so it retains compatibility.

Be a Good CLI Citizen

There’s a set of conventions and practices that high quality CLI tools share, which enable them to interact seamlessly with each other and, more importantly, allow a user to extrapolate lessons and habits learned from one tool into new contexts. This leads to a quicker sense of mastery and user delight because things feel “intuitive” (in reality, nothing about computers is truly intuitive to humans — there’s just experiences that confirm or confound our expectations based on previous encounters).

It’s tough to focus on a single tool as an example. Some of these conventions are defined in Jeff Dickey’s great post “12 Factor CLI Apps” about the Heroku CLI design philosophy, others from the GNU coreutils doc “Opening the software tool box”, and some are arguably matters of taste. Please comment if you think I’ve gotten something wrong or missed an important one!

• Expect to operate as part of a pipeline and adjust behavior accordingly. Like the bat example above, turn off any fancy formatting if output isn’t a terminal so downstream tools see plain-text output for easy processing.

• Don’t cross the streams! Closely related to pipeline operation, output should go to stdout and errors to stderr, so errors won’t be swallowed by redirection or intermingled with actual output.

• Provide tabular and structured output formats for lists of data. For extracting only parts of a record, make it easy for users to split on whitespace to get just single fields. And no modern tool that outputs a lot of records in a table ought to be without a json output option.

• For sophisticated CLIs, such as those which interact with remote services like Pulumi, think deeply about your command’s information architecture, then stick with it. A complex command invocation is like a grammatical sentence: it’s usually got a subject, a verb, and an object. For nontrivial commands there’s a hierarchy of these nouns and verbs that users put together to solve a problem. Either use a verb-object construction like kubectl get pods, where get is a top-level verb and the targets vary, or subject-verb construction like gcloud container clusters list (“clusters” is a subcommand of “container” and “list” is the verb).

• Observe control conventions. Users frequently get things wrong, and sometimes that happens after they’ve hit the enter key on a command. The convention is that ^C (control-C) interrupts execution and exits the program, cleanly if possible. Similarly, ^Z (control-Z) suspends a program so that it can be paused or put in the background. Good CLI commands should honor these signals and behave predictably when they come in.

• Don’t litter the filesystem. Use package managers and follow their conventions for the target operating system wherever possible. Similarly, follow conventions like XDG-Spec for guidance on where to install configuration files, docs, and supporting libraries. This kind of consistent packaging may take some additional work up front but it pays off in the long run when your software can be installed, upgraded, and config-managed alongside the rest of the user’s toolchain.

END OF FILE

When you’re done inputting a file into cat or its equivalents, a ^D (control-D) character indicates to the terminal that the end of the file’s been reached. Similarly, when ending a blog post it’s customary to summarize the points you’ve made and provide a stirring call to action for the reader. So, to recap the three top level design principles for modern command line interaction design:

1. Enable progressive discovery to lead users down a low-friction path to solving their problem with your tool.
2. Upgrade capabilities to take advantage of modern terminal emulation, so users get the advantage of colors, graphics, and interaction.
3. Understand and implement the conventions for well-behaved tools in your space, users can get to a sense of mastery quicker and avoid unpleasant surprises.

As I mentioned at the top, I’m working on a product called Relay that hopefully embodies these principles. Check out the introductory blog post to learn more and follow along as we develop our open-source CLI at github.com/puppetlabs/relay.