DEV Community: Daniel G. Taylor

Reducing Go Dependencies

Daniel G. Taylor — Wed, 07 Feb 2024 17:20:26 +0000

A case study of dependency reduction in Huma.

Intro

This article is a practical look at reducing dependencies in Go libraries. We'll start by looking at how Go dependencies work, then go into a few ideas around reducing dependencies. Finally, we'll go into a few ways I've implemented these ideas in Huma and the results. Hopefully you can use some of the same techniques in your own projects.

Why Reduce Dependencies

Why would you want to reduce the number of dependencies in your Go library? There are a few reasons:

Reduced Complexity: Fewer dependencies means less overall code to understand and maintain.
Reduced Risk: Fewer dependencies means fewer potential security vulnerabilities.
Reduced Build Times: Fewer dependencies can mean faster builds.
Reduced Binary Size: Fewer dependencies can mean smaller binaries.

A bit more subjective, perhaps, is that reducing dependencies can make your library more approachable to potential users. If they see that your library has fewer dependencies, they might be more likely to use it.

It can also mean the difference between using your library and not using it for certain constrained organizations, like governments, NGOs, or large corporations with strict security policies around third party code.

How Go Dependencies Work

Go dependencies are managed using the go.mod file. This file lists the dependencies of your project, and is used by the go tool to download and build your dependencies. When a user of your package runs go get, the go tool will download your package and some or all of its dependencies.

As of Go 1.17+, the go tool does what is called dependency graph pruning, meaning it will only add and download the dependencies that are actually used by your project. This means that if you use a module that has a dependency that is not used by your project, it will not be downloaded by the go tool.

Huma is a great example of this, as it uses an adapter concept to support many different router packages. The Huma go.mod contains a dependency for each router, but because each one is in its own package within the Huma module, only the one that is actually used by the project will be downloaded.

Another great example is the popular stretchr/testify module, which provides a package for test assertions. Many, many libraries use it for testing, but consumers of the library do not need to download that package because they are not using it in their own code.

This is important to remember: not every dependency you see in a library's go.mod will wind up in your project's go.mod because of this pruning!

Ideas Around Reducing Dependencies

Here are a few general ideas around reducing dependencies in Go libraries:

Use The Standard Library: Use the Go standard library as much as possible. It's well-tested, well-documented, and has a lot of functionality built in. Sometimes a dependency is just a convenience wrapper around the standard library and isn't strictly necessary.
Build It Yourself: Sometimes it's easier to build a small piece of functionality yourself than to bring in a dependency. This can be especially true for small, well-defined pieces of functionality.
It's Okay To Copy: If you only need a small part of a library, consider copying the code into your own project. This can be a good idea if the library is large or has a lot of dependencies.
Reframe The Problem: Sometimes you can reframe the problem you're trying to solve to avoid a dependency. For example, if you're using a library to marshal to a specific format, you might be able to use a simpler format or a different approach to avoid the dependency.
Move Examples: It's common and helpful to include examples or demos in your codebase to make it easy for new users to adopt and experiment. However, these examples can sometimes bring in a lot of dependencies. Consider moving them to a separate package or directory to avoid adding unnecessary dependencies to your main package.

A Side Note On Versioning

Keep in mind that as you work to reduce dependencies you have two options related to versioning (assuming you use semantic versioning):

Try your best to remain backward-compatible. You cannot break existing users of the library. This is the most desirable outcome and should result in a new minor version.
Make a new major version with breaking changes. This will require all users to change their own code, but it may be worth the benefits.

A third option might be to deprecate, announce, and later remove (or move) features without a new major version. It's not strictly semantic versioning then, but more like library evolution and may or may not work for your users.

A Case Study: Huma

Huma is a micro web framework for Go that is designed to be easy to use and provide OpenAPI & JSON Schema support on top of existing routers. It has a number of adapters for different routers, including http.ServeMux, chi, fiber, gorilla/mux, gin, etc.

When originally designing Huma v2.0.0 significant time was spent on the library's interface, usability, and feature set. I think that was the right call, and various dependencies allowed me to move faster and try various approaches easily. However, at release Huma had a lot of dependencies (both direct and indirect), and I wanted to reduce them in part because of feedback from the community and in part because I wanted to make the library more approachable to potential users.

Pull request #223 significanly reduces the dependencies between Huma versions v2.3.0 and v2.4.0. Read on for a detailed analysis.

Environment Variables

Huma depended on spf13/viper as a way to bind environment variables to command-line options so that a service can be configured either via env vars or flags on the command line. This was a convenience feature, but it was also a significant dependency. I realized that I was pulling in a dozen dependencies just for this, when instead I could write a small amount of code using the standard library to accomplish essentially the same thing. Whereas before I had code similar to this:



cfg := viper.New()
cfg.SetEnvPrefix("SERVICE")
cfg.SetEnvKeyReplacer(strings.NewReplacer("-", "_"))
cfg.AutomaticEnv()

// ... later on ...
cfg.SetDefault(name, defaultValue)
cfg.BindPFlag(name, flag)

Instead, I could write a small amount of code like this:



envName := "SERVICE_" + casing.Snake(name, strings.ToUpper)
defaultValue := field.Tag.Get("default")
if v := os.Getenv(envName); v != "" {
    // Env vars will override the default value, which is used to document
    // what the value is if no options are passed.
    defaultValue = v
}

Now, instead of pulling in a dozen dependencies, I was only using the standard library. Command-line options would either have the default set from the service code or be overridden by the environment variable, causing the help text to be accurate for that particular run.

This worked great as I really didn't need the full power of viper and was able to replace it with a small amount of code. You should consider whether you really need the full power of a library you are about to pull into your project.

YAML Generation

Huma supports generating OpenAPI 3.1 from your service, and it's common for OpenAPI to use both JSON and YAML for the spec document. It's also common for people to use extensions to the OpenAPI spec, meaning they put their own fields into the resulting JSON/YAML document. This is also the only part of my code that needs to use YAML, and it only needs to be marshaled (i.e. no parsing).

I was originally using goccy/go-yaml as it has built-in support for both ,inline to merge in extension fields, as well as a JSON formatter during serialization (which works because JSON is a subset of YAML). This worked great to support both JSON and YAML using one library, but it was a significant dependency with its own validation and parsing code, colorized output support using additional dependencies, special error dependencies, etc.

Rethinking and reframing this problem, I realized that I could use the standard library to generate JSON, and then use a small amount of code to merge in the extension fields using custom MarshalJSON() ([]byte, error) methods on my OpenAPI and JSON Schema structs (the marshalJSON utility is omitted for brevity):



func (i *Info) MarshalJSON() ([]byte, error) {
    return marshalJSON([]jsonFieldInfo{
        {"title", i.Title, omitNever},
        {"description", i.Description, omitEmpty},
        {"termsOfService", i.TermsOfService, omitEmpty},
        {"contact", i.Contact, omitEmpty},
        {"license", i.License, omitEmpty},
        {"version", i.Version, omitNever},
    }, i.Extensions)
}

Once I had the JSON, it's actually not that difficult to convert it to YAML. I began to write a small library to do so, but found someone else had already done it in just a few lines and was able to copy it into my codebase.

This removed a significant dependency and replaced it with mostly standard library code. I kept the ,inline field tags so if someone using Huma wanted to augment or serialize using a proper YAML library everything will still just work as expected.

JSON Schema Validation

Huma has its own JSON Schema request validator built-in, but for some validations it depended on third-party libraries, particularly for validating the format of strings.

For example, it used google/uuid to validate that a UUID is in the correct format. But, that library does a lot more than simple validation. It also has parsing, string generation, and other features that I didn't need. It's a mature library. I was able just copy a small amount of code from it to validate UUIDs in my own codebase.

Text Manipulation

As part of the OpenAPI generation, Huma uses a registry of schemas which are created from your Go operation handler input/output structs. These schemas are named using the struct name and support unicode, generics, and scalar types too.

For consistency the code which generates these names converts the input to PascalCase, which is common for public (exported) structs in Go code but not used everywhere, and generics using scalars result in type names like MyGeneric[int] which we want converted to MyGenericInt. This means you need to title case the parts of the name after removing any generics brackets.

Unfortunately strings.Title is deprecated, and the suggestion is to use golang.org/x/text/cases instead. That brings in over 40mb of source as a dependency due to all the unicode handling and other features it provides! This makes it hard to justify using it for just a single line of code, slows down compilation, potentially increases the binary size, and makes it impossible for users to share runnable snippets on the Go Playground as the builds will timeout.

Instead, I was able to write a tiny amount of code to handle this case and avoid the dependency entirely. It's not as featureful as the golang.org/x/text/cases package, but it's good enough for my use case and avoids a significant dependency while still supporting unicode and generics:



result := ""
for _, part := range strings.FieldsFunc(name, func(r rune) bool {
    // Split on special characters. Note that `,` is used when there are
    // multiple inputs to a generic type.
    return r == '[' || r == ']' || r == '*' || r == ','
}) {
    // Split fully qualified names like `github.com/foo/bar.Baz` into `Baz`.
    fqn := strings.Split(part, ".")
    base := fqn[len(fqn)-1]

    // Add to result, and uppercase for better scalar support (`int` -> `Int`).
    // Use unicode-aware uppercase to support non-ASCII characters.
    r, size := utf8.DecodeRuneInString(base)
    result += strings.ToUpper(string(r)) + base[size:]
}

It continues to pass a pretty extensive suite of test cases, and I'm happy with the result.

Code Examples

Huma has a "transformer" concept, which enables you to write a piece of code which modifies the response of an HTTP request before it is marshaled to e.g. JSON. It works with Go instances, and is a powerful feature used to e.g. insert $schema JSON Schema references which enabled code completion and linting as you type in editors like VSCode.

Since it's not a concept found in other routers or web frameworks, I wanted to provide more examples of what might be possible with it. My mistake was including them in the main package, causing additional dependencies like danielgtaylor/shorthand and danielgtaylor/mexpr to get pulled in.

Moving such examples into an examples directory with its own go.mod makes things much cleaner and avoids pulling in unnecessary dependencies. It also makes it easier for users to understand the core library and what is just an example or toy to show it off.

Results

The results speak for themselves. Using the Huma tutorial code as a benchmark, and replacing the Chi router with Go 1.22's improved http.ServeMux router to truly show the required dependencies of Huma's core library, we can see the following results from Huma v2.3.0 (before the changes):



module example.com/demo

go 1.22

require github.com/danielgtaylor/huma/v2 v2.3.0

require (
    github.com/danielgtaylor/casing v0.0.0-20210126043903-4e55e6373ac3 // indirect
    github.com/danielgtaylor/mexpr v1.8.0 // indirect
    github.com/danielgtaylor/shorthand/v2 v2.1.1 // indirect
    github.com/fatih/color v1.15.0 // indirect
    github.com/fsnotify/fsnotify v1.6.0 // indirect
    github.com/fxamacker/cbor/v2 v2.5.0 // indirect
    github.com/goccy/go-yaml v1.11.0 // indirect
    github.com/google/uuid v1.3.1 // indirect
    github.com/hashicorp/hcl v1.0.0 // indirect
    github.com/inconshreveable/mousetrap v1.1.0 // indirect
    github.com/magiconair/properties v1.8.7 // indirect
    github.com/mattn/go-colorable v0.1.13 // indirect
    github.com/mattn/go-isatty v0.0.20 // indirect
    github.com/mitchellh/mapstructure v1.5.0 // indirect
    github.com/pelletier/go-toml/v2 v2.0.8 // indirect
    github.com/spf13/afero v1.9.5 // indirect
    github.com/spf13/cast v1.5.1 // indirect
    github.com/spf13/cobra v1.8.0 // indirect
    github.com/spf13/jwalterweatherman v1.1.0 // indirect
    github.com/spf13/pflag v1.0.5 // indirect
    github.com/spf13/viper v1.15.0 // indirect
    github.com/subosito/gotenv v1.4.2 // indirect
    github.com/x448/float16 v0.8.4 // indirect
    golang.org/x/exp v0.0.0-20230515195305-f3d0a9c9a5cc // indirect
    golang.org/x/net v0.19.0 // indirect
    golang.org/x/sys v0.15.0 // indirect
    golang.org/x/text v0.14.0 // indirect
    golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
    gopkg.in/ini.v1 v1.67.0 // indirect
    gopkg.in/yaml.v3 v3.0.1 // indirect
)

After the changes above from PR #223 which are released in Huma v2.4.0, the dependencies are significantly reduced:



module example.com/demo

go 1.22

require github.com/danielgtaylor/huma/v2 v2.3.0

require (
    github.com/danielgtaylor/casing v1.0.0 // indirect
    github.com/fxamacker/cbor/v2 v2.5.0 // indirect
    github.com/inconshreveable/mousetrap v1.1.0 // indirect
    github.com/spf13/cobra v1.8.0 // indirect
    github.com/spf13/pflag v1.0.5 // indirect
    github.com/x448/float16 v0.8.4 // indirect
)

It's now so small and fast that it's easy to show off various features in the Go Playground, for example: https://go.dev/play/p/Qmv5VwNTg-L.

There could be further improvement, but not without breaking changes. For example, the optional CLI functionality could be moved to its own package and that would get rid of the spf13/cobra, spf13/pflag, and inconshreveable/mousetrap dependencies if not using that functionality. In practice, every service I've seen does use it so the utility of doing so is limited. Maybe this can be deprecated and removed over time, but for now I'm extremely happy with the results.

Aside from the smaller go.mod, build times were reduced by 12% and compiled binary size was reduced by 20% just from having to parse/compile/link less code.

Hopefully this dive into reducing dependencies in Huma has given you some ideas for your own projects. It's a good exercise to go through your dependencies and see if you really need them all, and if not, to consider whether you can replace them with a small amount of code or a different approach.

APIs in Go with Huma 2.0

Daniel G. Taylor — Wed, 06 Dec 2023 17:13:53 +0000

A modern, simple, fast & flexible micro framework for building HTTP REST/RPC APIs in Go backed by OpenAPI 3 and JSON Schema.

History

Back in 2016 I was working for a small-ish company doing live video streaming services which would eventually become part of Warner Bros Discovery through a complex chain of acquisitions and a "spin-merge." We were working on pulling apart a monolithic live video transcoding application meant to run on individual physical server racks into a set of web services run and dynamically scaled in the cloud in an effort to improve reliability, resiliency, and to realize potential cost savings.

We started to dabble in using Go, and I dove deep into the way the language works and how to write idiomatic Go code using the tools it provides. My confidence with Go quickly improved, and I started to help others to pick it up as well.

I wound up on a different team with pre-existing Python code so temporarily shelved my use of Go for a bit, and we used Sanic (an async Python framework built on top of the excellent uvloop & libuv that also powers Node.js) to build some APIs for live channel management & operations. We hand-wrote our OpenAPI and used it to generate documentation and a CLI, which was an improvement over what was there (or not) before. Other teams used the OpenAPI document to generate SDKs to interact with our service.

Unfortunately design-first was hard for us, as it is for most teams. The OpenAPI description document quickly got out of sync with the actual service, and clients ran into problems with missing or incorrect features in the CLI/SDK. Design first is great in concept, but without the robust framework of enforced API governance along with excellent scenario test coverage utilizing the OpenAPI document it's extremely difficult for teams to be successful. There is a high barrier of entry, especially for what should be a quick & simple API.

When it came time to evaluate replacing the now aging service, I prototyped a new version using the rapidly growing FastAPI framework, which automatically generates OpenAPI from the code for you, eliminating the possibility of the code & OpenAPI being out of sync, with the downstream effect of the docs, CLI, and SDKs also staying in sync properly. You can also still use a design-first approach, it just involves some structures written in code to be reviewed first. Design vs. code first is a false dichotomy by which we needn't be constrained.

Separately in my personal time I had been continuing my use of Go on Restish, a generic CLI for web APIs. I then evaluated the Go landscape for something like FastAPI in Go, but came away disappointed. I wondered if I could make something similar, and started to prototype what this might look like. With some excellent mentorship and a lot of rewrites, I wound up with Huma as a mostly idiomatic version of something like FastAPI in Golang.

It was then announced at work that all new services would be written in Go. Having had experience with it before, I was excited to get started. I evaluated the landscape of packages again and pitched using Huma to the team, which was an easy sell after the benefits we saw in the FastAPI prototype and the pain we had felt for years hand-writing a separate OpenAPI document.

And so, Huma was born and rapidly got production usage handling the configuration & live operations of thousands and thousands of live channels. Third party contributions started to come in and several organizations began to use it for production systems, resulting in a stable v1.0.0 followed by many fixes and additions. Here's an unsolicited quote from Reddit user Jeb_Jenky:

[Huma] is by far my favorite web framework for Go. It is inspired by FastAPI, which is also amazing, and conforms to many RFCs for common web things...
Anyway I really like the feature set, the fact that it uses Chi, and the fact that it is still somehow relatively simple to use. I've tried other frameworks and they do not spark joy for me.

The value of a software package's ability to spark joy cannot be understated!

Huma 2.0

Since that initial release I have gotten a ton of great feedback, and Go 1.18/1.20 added new interesting features to the language which could help prevent some common mistakes via strongly-typed compile-time checks.

I created a Huma v2 branch to rewrite it for Go 1.20+ improving on almost every aspect of the framework, and the 2.0.0 release is available now alongside a brand new Huma Documentation site.

Features

What makes Huma special and why should you use it? How does 2.0.0 differ from the 1.x.x versions?

Standards compliant: built-in support for OpenAPI 3.1, JSON Schema 2020-12, JSON, CBOR, RFC 7807 errors, client-driven content negotiation, conditional requests, PATCH support, describedby link relations & more.
Up to date: docs & generated tooling are always up to date with the server interface, no matter how many teams work on it or how old the project gets.
Router agnostic: use whatever router you want. Integrate it into existing projects easily for incremental adoption.
Strongly typed: request inputs and response outputs are strongly typed and just use standard Go structs. Generics ensure compile-time checks for fast feedback.
Idiomatic Go: handlers use context.Context, your input/output structs, and error. There is less interface{} or any magic.
Errors: Huma uses standardized (RFC 7807), exhaustive errors so users can know what is wrong right away and fix it, rather than making request after request with a different error response each time.
Extensible: all of the OpenAPI is editable, including extensions. You have direct access to the router for serving static content, HTML rendering, websockets, or other features without needing to open pull requests. It's easy to wrap or replace built-in functionality, like error models. It's easy to add custom validation rules, too.
Fast: reflection is pre-computed and cached at startup; validation is zero or low-allocation and blazing fast, and buffer pools are shared to amortize allocation cost whenever possible. Go is also significantly faster than Node.js or Python.
Well tested: Huma is used in production services to run large successful live linear and live event channels. Code coverage is 93%, and some optional pieces like the shorthand PATCH syntax are additionally fuzz tested for potentially problematic inputs.

See https://huma.rocks/features/ for more information and a deep dive on how the various features work.

Example

Here is a basic "Hello, world!" style example that implements a single API endpoint /greeting/{name} and replies with a JSON object like:

{
  "message": "Hello, {name}!"
}

The {name} is replaced with whatever value is sent to the API. This example is taken directly from the Huma Tutorial.

package main

import (
    "context"
    "fmt"
    "net/http"

    "github.com/danielgtaylor/huma/v2"
    "github.com/danielgtaylor/huma/v2/adapters/humachi"
    "github.com/go-chi/chi/v5"
)

// GreetingInput represents the greeting operation request.
type GreetingInput struct {
    Name string `path:"name" maxLength:"30" example:"world" doc:"Name to greet"`
}

// GreetingOutput represents the greeting operation response.
type GreetingOutput struct {
    Body struct {
        Message string `json:"message" example:"Hello, world!" doc:"Greeting message"`
    }
}

func main() {
    // Create a new router & API
    router := chi.NewMux()
    api := humachi.New(router, huma.DefaultConfig("My API", "1.0.0"))

    // Register GET /greeting/{name}
    huma.Register(api, huma.Operation{
        OperationID: "get-greeting",
        Summary:     "Get a greeting",
        Method:      http.MethodGet,
        Path:        "/greeting/{name}",
    }, func(ctx context.Context, input *GreetingInput) (*GreetingOutput, error) {
        resp := &GreetingOutput{}
        resp.Body.Message = fmt.Sprintf("Hello, %s!", input.Name)
        return resp, nil
    })

    // Start the server!
    http.ListenAndServe("127.0.0.1:8888", router)
}

And now you can test it out, e.g. with Restish:

# In one tab, run the service:
$ go run main.go

# In another tab, make a request
$ restish :8888/greeting/world

You can even see generated interactive documentation powered by Huma's OpenAPI support at http://localhost:8888/docs:

With just a few lines of code you now have a fully functional API with an OpenAPI document and generated documentation!

Q & A

What about DBs and other features?

Huma is deliberately a lightweight micro-framework. It is a building block to build on top of, and will never be something like Django. Others can definitely take it and make something like Django for Go, it's just not what this projects is about.
What about gRPC?

gRPC is an HTTP/2-based protocol built around binary protobuf messages. It has excellent tooling for Go and uses code generation from a description format similar to OpenAPI. I use it at work alongside Huma services, and I've written one of the more popular Python protobuf libraries, so you could say I know a little bit about it. Overall in concept it's quite similar to OpenAPI and can be a great choice for your organization or team.

I tend to prefer OpenAPI for interfacing with third-party customers, as they are most likely already familiar and comfortable with HTTP and/or REST services (and probably use e.g. AWS, Google Cloud, Azure, etc). There are good HTTP gateways for gRPC, but they have quirks and don't always map well to REST concepts if that's something you care about. Additionally protobuf has no built-in validation, though there are third-party libraries for it.

We actually expose some of our gRPC services through Huma rather than another gateway using the protoc-gen-huma plugin. You'll have to evaluate what works best for your team and product.

It's worth noting that Huma also supports HTTP/2 and the CBOR binary format out of the box, so you get some of the same benefits (connection multiplexing, smaller wire size, faster encoding) of using gRPC. It's also possible to use protobuf with Huma.

Mapping OpenAPI to the CLI

Daniel G. Taylor — Fri, 15 Apr 2022 20:02:45 +0000

In this post we'll explore Restish, a CLI for APIs with built-in OpenAPI support. How does it go from an OpenAPI service description to CLI commands & arguments? Read on to find out!

Autodiscovery

Restish supports OpenAPI autodiscovery using several different mechanisms. You can provide an RFC 8631 service-desc link relation, an RFC 5988 describedby link relation, or provide an /openapi.yaml or /openapi.json with your API.

Link Relation Header

When using the service-desc or describedby link relation, Restish follows the link to find the OpenAPI. For example, if you get https://api.example.com/ it might return the following header:

Link: </path/to/openapi.yaml>; rel="service-desc"

Restish would then fetch https://api.example.com/path/to/openapi.yaml and load that into operations, arguments, flags, etc.

Fallback Mechanism

If not link relation header is present, a fallback mechanism is used. Restish looks for https://api.example.com/openapi.yaml or https://api.example.com/openapi.json. If found, it will load it.

Anatomy of a CLI Operation

A CLI operation can consist of several parts:

Name	Description
API	Short-name of the API, configured when registering the API with Restish (can be anything you want).
Operation	OpenAPI Operation ID
Options	Optional operation query or header parameter(s)
Arguments	Required operation path parameter(s)
Request Body	Optional request body, which can be passed in via stdin or via CLI Shorthand in the command.

Aside from those, there is also the act of generating --help output: markdown descriptions and output JSON Schemas which need to be handled.

This is how those would map into an OpenAPI 3 YAML:

Github API Example

Let's take a look at a real-world example from the Github V3 OpenAPI. Here is a truncated version of the code search operation:

"/search/code":
  get:
    summary: Search code
    operationId: search/code
    parameters:
    - name: q
      description: The query contains one or more search keywords...
      in: query
      required: true
      schema:
        type: string
    - name: sort
      description: Sorts the results of your query...
      in: query
      required: false
      schema:
        type: string
        enum:
        - indexed
    - "$ref": "#/components/parameters/order"
    - "$ref": "#/components/parameters/per-page"
    - "$ref": "#/components/parameters/page"
    responses:
      '200':
        description: Response
        content:
          application/json:
            schema:
              type: object
              required:
              - total_count
              - incomplete_results
              - items
              properties:
                total_count:
                  type: integer
                incomplete_results:
                  type: boolean
                items:
                  type: array
                  items:
                    "$ref": "#/components/schemas/code-search-result-item"
            examples:
              default:
                "$ref": "#/components/examples/code-search-result-item-paginated"
      '304':
        "$ref": "#/components/responses/not_modified"
      '503':
        "$ref": "#/components/responses/service_unavailable"
      '422':
        "$ref": "#/components/responses/validation_failed"
      '403':
        "$ref": "#/components/responses/forbidden"

This translates into the following command help in Restish showing you how to use it. Note the operation ID search-code and the parameters like --sort and --per-page from the $refs above. The response is also used to generate a terminal-friendly representation of the response schema so users know what to expect as output.

$ restish github search-code --help
Description truncated for example...

## Response 200 (application/json)

`schema
{
  incomplete_results: (boolean)
  items: [
    {
      git_url: (string)
      html_url: (string)
      name: (string)
      path: (string)
      repository: {
        archive_url: (string)
        assignees_url: (string)
        blobs_url: (string)
        branches_url: (string)
        collaborators_url: (string)
        comments_url: (string)
        commits_url: (string)
        compare_url: (string)
        contents_url: (string)
        contributors_url: (string)
        description: (string)
        downloads_url: (string)
        events_url: (string)
        fork: (boolean)
        forks_url: (string)
        full_name: (string)
        git_commits_url: (string)
        git_refs_url: (string)
        git_tags_url: (string)
        hooks_url: (string)
        html_url: (string)
        id: (number)
        issue_comment_url: (string)
        issue_events_url: (string)
        issues_url: (string)
        keys_url: (string)
        labels_url: (string)
        languages_url: (string)
        merges_url: (string)
        milestones_url: (string)
        name: (string)
        node_id: (string)
        notifications_url: (string)
        owner: {
          avatar_url: (string)
          events_url: (string)
          followers_url: (string)
          following_url: (string)
          gists_url: (string)
          gravatar_id: (string)
          html_url: (string)
          id: (number)
          login: (string)
          node_id: (string)
          organizations_url: (string)
          received_events_url: (string)
          repos_url: (string)
          site_admin: (boolean)
          starred_url: (string)
          subscriptions_url: (string)
          type: (string)
          url: (string)
        }
        private: (boolean)
        pulls_url: (string)
        stargazers_url: (string)
        statuses_url: (string)
        subscribers_url: (string)
        subscription_url: (string)
        tags_url: (string)
        teams_url: (string)
        trees_url: (string)
        url: (string)
      }
      score: (number)
      sha: (string)
      url: (string)
    }
  ]
  total_count: (number)
}
`

Usage:
  restish github search-code [flags]

Flags:
      --accept application/vnd.github.v3+json   Setting to...
  -h, --help                                    help for search-code
      --order desc                              Determines whether the first...
      --page int                                Page number of the results to fetch. (default 1)
      --per-page int                            Results per page (max 100) (default 30)
      --q string                                The query contains one or more search...
      --sort indexed                            Sorts the results of your query...

Global Flags:
...

Note also that all parameters use double slashes (--), since single slashes are reserved for Restish use. Conversely, all Restish parameters are prefixed with --rsh- in order to prevent collisions.

For operations with required arguments and/or bodies, Restish is able to generate usage and example documentation as well, including example CLI Shorthand input. For example, when creating a new repo fork:

## Request Schema (application/json)

`schema
{
  organization: (string) Optional parameter to specify the organization name if forking into an organization.
}
`

...

Usage:
  restish github repos-create-fork owner repo [flags]

Examples:
  restish repos-create-fork owner repo organization: string
  restish repos-create-fork owner repo <input.json

Overrides

Sometimes you might want the CLI operation name or parameter name to be different from what the official name is in the API, or hide a particular deprecated parameter, or even tell Restish how to automatically configure auth. These are all possible with Restish OpenAPI Extensions.

Name	Description
`x-cli-aliases`	Sets up command aliases for operations.
`x-cli-config`	Automatic CLI configuration settings.
`x-cli-description`	Provide an alternate description for the CLI.
`x-cli-ignore`	Ignore this path, operation, or parameter.
`x-cli-hidden`	Hide this path, or operation.
`x-cli-name`	Provide an alternate name for the CLI.

For example, in the search operation above, the query parameter is named q and would show up in Restish as --q which is not very friendly. You might rename it via x-cli-name: query so that people can use --query instead.

Conclusion

Hopefully this has shed some light on how Restish is able to dynamically generate CLI commands from OpenAPI specifications, and how you can expect those commands to operate if you are already familiar with the backend API.

A CLI for REST APIs

Daniel G. Taylor — Wed, 03 Jun 2020 17:58:27 +0000

Standardizing how power users and scripts interact with your service

Sorry about the acronym soup in the title! 😅

Today I want to talk about an oft-neglected aspect of building REST or HTTP APIs. There are plenty of articles about API design, API description formats like OpenAPI, and lots of libraries and code generators used to talk to the API from code. What is missing in my opinion is a high-quality command line client that’s generic enough to be used for most HTTP APIs.

But there’s curl, you say. Many service docs even include examples specifically for calling the service’s API via curl commands. I love curl and HTTPie, but these are just generic HTTP clients. There’s not enough specialization for REST or HTTP APIs. HTTPie comes close since it speaks JSON, the lingua franca of web APIs, but falls short in other ways, such as describing complex inputs for API operations.

What fancy-pants features would make me happy, you ask? My list would include at least the following:

Easy to install & fast startup
Default to HTTP/2 (with TLS), which is now 5 years old! 😮
Speaks JSON, YAML, and their binary relatives natively
Caching of responses when appropriate headers are set
Built-in common API authentication mechanisms like OAuth 2
Automatically handle pagination of collections
An easy way to input complex nested data
Pretty colorized output with the ability to filter built-in
An understanding of API specs like OpenAPI with auto-discovery
Always having access to the latest API features & docs

I think that would be a good start to getting where we should be in 2020. With that in mind, I’d like to introduce my new CLI tool for REST & HTTP APIs called Restish (https://rest.sh/).

Restish can do all of the above wish list, and more. Installing it is easy if you have Homebrew:



$ brew tap danielgtaylor/restish && brew install restish

Windows users will need to download a release. At its most basic it can be used very similar to how you might use curl:



$ restish -H Authorization:abc123 api.example.com/items/1
HTTP/2.0 200 OK
Content-Encoding: br
Content-Type: application/json
Date: ...

{
  name: "Item number one"
  cost: 12.34
  created: "2020-01-01T12:00:00Z"
  tags: ["grocery"]
}

The real magic happens when you register an API base URI with Restish through the interactive configure sub-command:



$ restish api configure example
? Base URI [? for help]: https://api.example.com
Setting up a `default` profile
? Select option for profile `default` Add header
? Header name Authorization
? Header value (optional) abc123
? Select option Save and exit

$ restish get example/items/1
...

$ restish post example/items name: Another, cost: 2.50, tags[]: household
HTTP/2.0 201 Created
Location: /items/2

If the API provides an OpenAPI spec via a known link relation header or some common path like /openapi.json then things get even more interesting because you can directly call described API operations rather than specific URIs:



$ restish example create-item name: Another, cost: 2.50, tags[]: household
HTTP/2.0 201 Created
Location: /items/3

The popular FastAPI Python library provides exactly that, so any services written with it will just work out of the box with Restish. And because Restish loads the OpenAPI spec on the fly, whenever you push updates to your FastAPI service then your CLI users will already have the new features available.

Even better, CLI users can query the API for information on structure, including any updates you push to the API. For example:



$ restish example create-item --help
Create a new item in the inventory

# Request Schema (application/json)

{
  name*: (string) The item name
  cost*: (number min:0) The cost of the item
  tags: [
    (string maxLen:12) Tag name
  ]
}

# Response 201

A new item has been created. The `Location` header links to the item.

Usage:
  restish example create-item ...

Example:
  restish example create-item name: string, cost: 1.0, tags[]: string
  restish example create-item <input.json
...

This is win-win for everyone. Users get a high quality CLI tool with advanced features built-in to interact with your service. Your developers don’t have to waste time building and maintaining a custom CLI or getting users to upgrade to access the newest API features.

You should consider adopting Restish for your APIs and adding the “Works with Restish” badge to your documentation.

There is a ton more breadth and depth to Restish. This quick introduction is just the first of a multi-part series to go into the various features and use-cases with lots of examples, so stay tuned and thanks for reading!