DEV Community: Frédéric G. MARAND

CLI commands with google/subcommands

Frédéric G. MARAND — Sun, 06 Mar 2022 14:04:50 +0000

How to create CLI programs with multiple commands, flags and subcommands, and do it fast and simply?

google/subcommands makes it a breeze. Let us see how.

(original location: https://osinet.fr/go/en/articles/cli-google-subcommands/)

The problem scope

First, let us agree on the topics covered here.

Any Go program is - at least in concept - run from a command line, whether it be an actual shell command line or some automation tool, using the execve(2) system call.

To define the specific words used in this tutorial, let us consider a typical *NIX command:

docker -D container inspect -s some_container

docker is the program, or application
-D is a global flag
container is a root or top-level command
inspect is a subcommand of the container command; more generally, it is also a command too, but a level 2 one.
-s is a local flag for the container subcommand
some_container is an argument to the inspect subcommand

In order to interpret that kind of structure, Go code needs to work from a single global variable, os.Args, the first element of which contains the absolute path to the program being run, while the other elements are copies of the arguments passed when starting the program.

To avoid having to redo that kind of very formalized work in every other program, various libraries/modules exist, with the most popular being:

Module	☆	⑂	Direct deps.
spf13/cobra	25400	2200	5
urfave/cli	17400	1500	3
alecthomas/kingpin	3200	240	5
google/subcommands	594	51	0

The first three libraries are by far the best known and the richest in terms of functionality, but that comes with multiple dependencies, sometimes quite a lot (3136 for Cobra 1.3). While this is easily justified in humongous projects like Docker or Kubernetes, such a complexity is not as acceptable for typical projects like microservices or small, sharp tools for the CLI.

This is where google/subcommands shines, providing support for most commonly needed command creation tools without requiring any dependency, in a single module around 500 lines only, very easy to use in any application and with a negligible size cost even for the tiniest projects. This tutorial is about how to use it from the simplest to the most advanced scenario.

All along the successive refinement levels in this demo, we are going to be adding features to the same example code, which you can find on https://github.com/fgm/subcommands_demo, with every branch matching the eponymous section in this tutorial, containing 100% of the source code for the level being discussed, as well as a Makefile configured to demonstrate just the specifics of that level.

This tutorial covers 100% of the functionality in google/subcommands, hence its overall length ; you will however be able to create a complete application as early as level 1.2 : later levels describe features meant for projects ever more complex and demanding.

Level 1 : simple commands

We shall start by creating a couple of top-level commands, laying out our code in two directories: the project root and cmd/, with the following main files :

Path	Contents
`go.mod`	the module description file
`main.go`	the program entry point
`Makefile`	the build tasks file
`cmd/`	the commands directory
`cmd/root.go`	the code running the commands

A word of advice: in each branch, the make command runs a different default sequence of tasks, meant to illustrate the changes brought by the current branch: you might wish to use that command on your own machine any time you change branch, to see the impact the code changes actually have.

1.1 Enabling builtin commands

The code for that level is available on level1.1-builtin_commands.

At this step, the code is just made up of two files: the application entry point in main.go:

func main() {
    ctx := context.Background()
    sts := cmd.Execute(ctx) // Runs the designated command on the CLI
    os.Exit(int(sts))       // Returns the command result to the program caller
}

…and the code triggering execution of the commands, in cmd/root.go :

// Execute sets up the command chain and runs it.
func Execute(ctx context.Context) subcommands.ExitStatus {
    for _, command := range [...]subcommands.Command{
        subcommands.CommandsCommand(), // Implement "commands"
        subcommands.FlagsCommand(),    // Implement "flags"
        subcommands.HelpCommand(),     // Implement "help"
    } {
        subcommands.Register(command, "")
    }

    flag.Parse()
    return subcommands.Execute(ctx)
}

The above code registers the three optional commands provided by subcommands proper, which enables listing existing commands and flags, as well as providing CLI help, then it parses global flags and ends up passing control to the actual command selected by the CLI arguments, and returning its result.

Our application scaffold is now ready and supports commands commands, flags, et help.

$ go run . help
Usage: subcommands_demo <flags> <subcommand> <subcommand args>

      Subcommands:
      commands         list all command names
      flags            describe all known top-level flags
      help             describe subcommands and their syntax
$

We can now create our first custom commands.

1.2 Writing custom commands

The code for that level is available on [level1.2-custom_commands](https://github.com/fgm/subcommands_demo/tree/level1.2-custom_commands.

The best practices with custom commands is to have one file per command. This eases code navigation and comparison of commands between each other to ensure consistency of practices. We shall therefore add two files, one for each of the two top-level commands we are adding:

Path	Contents
`cmd/top1.go`	the `top1` command, showing a fixed message
`cmd/top2.go`	the `top2` command, showing a message based on the CLI arguments

For google/subcommands, a command is provided by a variable, the type of which implements the subcommands.Command interface. Our two files will therefore be very similar to each other, based on the structure in file cmd/top2.go :

type top2 struct{}

func (cmd *top2) Name() string {
    return "top2"
}

func (cmd *top2) Synopsis() string {
    return "top2 is an example top-level custom command with arguments"
}

func (cmd *top2) Usage() string {
    return fmt.Sprintf("%s arg1 arg2 ...", cmd.Name())
}

func (cmd *top2) SetFlags(fs *flag.FlagSet) {}

func (cmd *top2) Execute(_ context.Context, fs *flag.FlagSet, _ ...interface{}) subcommands.ExitStatus {
    // Notice how the command line arguments are taken on the flag set, not on the variadic.
    fmt.Printf("In %s %v\n", cmd.Name(), fs.Args())
    return subcommands.ExitSuccess
}

For now, top2 just implements subcommands.Command and does nothing more:

top2.Name() returns the name of the command
top2.Synopsis returns a single-line summary of the command description
top2.Usage() returns an example of how to use the command, and may spread over multiple text lines. The returned value is used by the help builtin command.
top2.SetFlags() defines the local flags on the command: none for now.
top2.Execute() is the actual command implementation.
- the context.Context arguments supports passing context values, and using timeout and cancellation capabilities, which will be available to subcommands, just like in a http.Handler.
- the flag.FlagSet holds the local flag definitions, actual values, and the command arguments.
- when running that method, the name of the program, the global flags, as well as the command local flags have been parsed, and fs.Args() holds the command arguments with no extras.
- if we run the program as demo top2 hello world, fs.Args() will contain []string{"hello", "world"}.
- the third parameter is not used in this scenario.

Now that we have created our two commands, we can register all three of them with google/subcommands: top1, top2, and a 1 alias for top1. We can perform this by adding instances of the commands to the subcommands.Register
call besides commands, flags and help, in root.go :

func Execute(ctx context.Context) subcommands.ExitStatus {
    for _, command := range [...]subcommands.Command{
        subcommands.CommandsCommand(),   // Implement "commands"
        subcommands.FlagsCommand(),      // Implement "flags"
        subcommands.HelpCommand(),       // Implement "help"
        &top1{},                         // Our first top-level command, without args
        &top2{},                         // Our second top-level command, with args
        subcommands.Alias("1", &top1{}), // An alias for our top1 command
    } {
        subcommands.Register(command, "")
    }

Once this is done, our program supports these two commands, as well as a "1" alias for command top1.

Many applications will not anything more when it comes to commands handling.

$ go run . commands
commands
flags
help
top1
top2
1
$ go run . top2 hello world
In top2 [hello world]
$

1.3 Passing non-CLI arguments to commands

The code for that level is available on level1.3-non_cli_arguments.

In the previous example, the top2 command takes arguments passed from the CLI.

Command code may however need extra arguments passed to all commands regardless of the CLI arguments, like a logger, an authentication service, etc.

That is where the variadic argument to subcommands.Execute enters into play, taking any number of values of any type, to be received by the Command.Execute() implementations on their variadic parameter. We shall be using it to pass two values :

func Execute(ctx context.Context) subcommands.ExitStatus {
    for _, command := range [...]subcommands.Command{
        /* ...snip... */
    flag.Parse()
    return subcommands.Execute(ctx, "meaning", 42)

Here they are handled in top1 :

func (cmd *top1) Execute(_ context.Context, _ *flag.FlagSet, args ...interface{}) subcommands.ExitStatus {
    // The variadic arguments are the ones passed to subcommands.Execute().
    // Unlike the CLI args, they are always a []interface{}.
    fmt.Printf("In %s.\nNon-CLI args: %#v\n", cmd.Name(), args)
    return subcommands.ExitSuccess
}

$ go run . top1
In top1.
Non-CLI args: []interface {}{"meaning", 42}
$

The salient issue here is how, as a command author, you need to coordinates the types and order of values passed in that variadic arguments, since they are passed as interface{} (aka any starting with Go 1.18), which will usually mean a type assertion in the Execute method to get a ready-to-use value.

1.4 Grouping commands

The code for that level is available on level1.4-command_groups.

When a project starts to grow and get ever more commands, grouping them by purpose may become useful. This is what the second, string, parameter of subcommands.Register is for. We use it in root.go, to associate each registered command to a given subcommands.Group.

In this example, we shall separate commands provided by google/subcommands, in the help group, from our custom commands, in the top group.

func Execute(ctx context.Context) subcommands.ExitStatus {
    for _, command := range [...]struct {
        group string
        subcommands.Command
    }{
        {"help", subcommands.CommandsCommand()},  // Implement "commands"
        {"help", subcommands.FlagsCommand()},     // Implement "flags"
        {"help", subcommands.HelpCommand()},      // Implement "help"
        {"top", &top1{}},                         // Our first top-level command, without args
        {"top", &top2{}},                         // Our second top-level command, with args
        {"top", subcommands.Alias("1", &top1{})}, // An alias for our top1 command

    } {
        subcommands.Register(command.Command, command.group)
    }

The help command now displays our commands in these two different groups :

$ go run . help
Usage: subcommands_demo <flags> <subcommand> <subcommand args>

Subcommands for help:
        commands         list all command names
        flags            describe all known top-level flags
        help             describe subcommands and their syntax

Subcommands for top:
        top1, 1          top1 is an example top-level custom command without arguments
        top2             top2 is an example top-level custom command with arguments
$

1.5 Adding global or local flags

The code for that level is available on level1.5-flags.

Many programs use a few global flags, like a boolean -v used to increase the verbosity of all commands ; and some also use local flags allowing individual commands to adjust their own behaviour.

Let us start with the typical global boolean -v flag, and implementing it in our two existing commands. We can do this with the stdlib package flag, with which google/subcommands has a hidden integration.

To make sure that flag will be available to all commands and subcommands, we add its value to the context. Following best Go practices, we define an unexported specific context key type, and a variable of that type, in our root.go file :

type verboseKey struct{}

var VerboseKey = verboseKey{}

Now, in our Execute function, we define that the flag, parse the program CLI arguments, and add the parsed flag value to the context using that key :

func Execute(ctx context.Context) subcommands.ExitStatus {
    for _, command := range [...]struct {
        /* ...snip... */
  }

    verbose := flag.Bool("v", false, "Be more verbose")
    flag.Parse()
    ctx = context.WithValue(ctx, VerboseKey, *verbose)

Commands can now fetch the flag value from the context, as in this example taken from top1.go :

func (cmd *top1) Execute(ctx context.Context, _ *flag.FlagSet, _ ...interface{}) subcommands.ExitStatus {
    if ctx.Value(VerboseKey).(bool) {
        fmt.Printf("In %s.\n", cmd.Name())
    }

Since we defined the flag type as boolean, and the key is of a unique unexported type, there can be no collision on the context value even in a large program, which means the type assertion will never fail.

For that global flag, we did not knowingly use google/subcommands at all, and it is correctly recognized by builtin command flags, but does not appear in the builtin help provided by the help command :

$ go run . help
Usage: subcommands_demo <flags> <subcommand> <subcommand args>

Subcommands for help:
        commands         list all command names
        flags            describe all known top-level flags
        help             describe subcommands and their syntax

Subcommands for top:
        top1, 1          top1 is an example top-level custom command without arguments
        top2             top2 is an example top-level custom command with arguments


Use "subcommands_demo flags" for a list of top-level flags
$ go run . flags
  -v    Be more verbose
$

Now, let us define a local flag on command top1. Since that flag will be specific to the command, it will be listed after the command name in the command line arguments, preventing a simple use of the flag package procedural API like we just did for global flags.

Local flags are defined on each command by implementing their SetFlags method to store the parsed flag value on a command instance field.

To that effect, we shall add a field to the top1 type implementing our command, in the top1.go file, to store the parsed flag value, which will be a string field in this example.

type top1 struct {
    prefix string
}

We then implement the top1.SetFlags method to define that flag and store its parsed value on the command instance :

func (cmd *top1) SetFlags(fs *flag.FlagSet) {
    fs.StringVar(&cmd.prefix, "prefix", "", "Add a prefix to the result")
}

func (cmd *top1) Execute(ctx context.Context, _ *flag.FlagSet, _ ...interface{}) subcommands.ExitStatus {
    if ctx.Value(VerboseKey).(bool) {
        fmt.Printf("In %s.\n", cmd.Name())
    }
    fmt.Println(strings.Join(append([]string{cmd.prefix}, "hello"), ": "))
    return subcommands.ExitSuccess
}

The help command is now aware of our local flag :

$ go run . help top1
top1  -prefix string
        Add a prefix to the result
$

When subcommands.Execute prepares to invoke our top1 command instance, it starts by creating a new empty flag.FlagSet from the remaining arguments on the current one, which it passed to the SetFlags method on the command instance. That defines flags pointing to fields on said command instance, so that parsing will field values on it. These fields are thereafter available on the instance
when eventually running the top1.Execute method, allowing the method code to access their values as plain typed fields on the method receiver, as demonstrated on line 40 for cmd.prefix.

$ go run . top1 -prefix today
today: hello
$

Another possible mechanism enabled by this process would be for the method implementation to just drop the flag, not pointing it anywhere, then extract the value from the fs *FlagSet argument in top1.Execute, as done below, but that leads to code which is much less readable than the recommended method above.

func (cmd *top1) SetFlags(fs *flag.FlagSet) {
  _ = fs.String("prefix", "", "Add a prefix to the result")
}

func (cmd *top1) Execute(ctx context.Context, fs *flag.FlagSet, _ ...interface{}) subcommands.ExitStatus {
    prefix := fs.Lookup("prefix").Value.(fmt.Stringer).String()
    fmt.Println(strings.Join(append([]string{cmd.prefix}, "hello"), ": "))
    return subcommands.ExitSuccess
}

1.6 Labeling flags as important

The code for that level is available on level1.6-important_flags.

Some flags may be frequently used, making them worthy of being reported in the help command output.

Let us describe our -v global flag as an important one, and our code also include an unimportant boolean -debug flag. We can inform google/subcommands of that difference when declaring the flags in our root.go file :

func Execute(ctx context.Context) subcommands.ExitStatus {
    for _, command := range [...]struct {
        /* ...snip... */
    }

    debug := flag.Bool("debug", false, "Show debug information")
    verbose := flag.Bool("v", false, "Be more verbose")
    subcommands.ImportantFlag("v")
    flag.Parse()

Without the subcommands.ImportantFlag("v") statement, the help command does not document any top-level flag, as we saw in the level1.5 example above, where the help command mentions that top-level flags exist, but does not list any.

When we add this statement, however, the now important -v flag is included in the help output, while the ordinary -debug flag remains unlisted.

$ go run . help
Usage: subcommands_demo <flags> <subcommand> <subcommand args>

Subcommands for help:
        commands         list all command names
        flags            describe all known top-level flags
        help             describe subcommands and their syntax

Subcommands for top:
        top1, 1          top1 is an example top-level custom command without arguments
        top2             top2 is an example top-level custom command with arguments


Top-level flags (use "subcommands_demo flags" for a full list):
  -v=false: Be more verbose
$

Both flags are listed indiscriminately by command flags, since its express purpose is to document flags :

$ go run . flags
  -debug
        Show debug information
  -v    Be more verbose
$

With all the features we have now examined, google/subcommands covers the needs of most small applications. Later levels will mostly be relevant for more complex projects, especially those which require unit testing coverage for commands themselves, instead of just requiring it for the service-level code the commands call upon, which is a more frequent requirement.

Level 2 : Reusing command code

The code for that level is available on level2.1-reuse.

Until this point we have been using one defined type per command, not counting aliases, as suggested by the minimal documentation provided by the google/subcommands repository. Doing so allows storing command-specific properties of commands, like local flags, on the instance of that type which is registered with google/subcommands.

In practice, however, many commands either do not use any flag, or share whichever flags they use ; and repeating identical code for each of these types is not entirely satisfying. But there is no actual need to use different types for different commands, especially if they share the same structure.

The main reason for different types is the Command.Execute method, which has to differ for each command since commands implement different features. Sharing types means using different methods for each command instance.

Various techniques are available for that purpose. For example, using a single top type, its Execute method could look up a property on the command instance to select and invoke a concrete runner function, as in this fragment :

// In top.go
type top struct {
    // ...shared fields
    name string // command name
}

// Add the Name, Synopsis, and SetFlags methods to implement subcommands.Command.

// NewTop replaces both NewTop1 and NewTop2.
func NewTop(name string) {
    return &top{name: name}
}

func (t *top) Execute(ctx context.Context, fs *flag.FlagSet, args ...any) {
    switch t.name {
    case "top1":
        Top1Execute(ctx, fs, args) // A function in top1.go
  case "top2":
      Top2Execute(ctx, fs, args) // A function in top2.go
    default:
        panic("Unexpected command %s, should never happen", t.name)
    }
}

// In root.go,
func Execute(ctx context.Context) subcommands.ExitStatus {
    for _, command := range [...]struct {
        group string
        subcommands.Command
    }{
        {"help", subcommands.CommandsCommand()},    // Implement "commands"
        {"help", subcommands.FlagsCommand()},       // Implement "flags"
        {"help", subcommands.HelpCommand()},        // Implement "help"
        {"top", NewTop("top1")},                    // Our first top-level command, without args
        {"top", NewTop("top2")},                    // Our second top-level command, with args

Onw downside with this approach is how it introduces coupling between

the code in the shared Execute method and the various commands.
the code in the shared NewTop factory function, which could requires modifications to accommodate the specifics of various commands

A better model is more direct and prevents such coupling. It keeps per-command factory functions, and uses a function field to store the runner function on the instance. That way the dedicated factory functions are used in root.go :

func Execute(ctx context.Context) subcommands.ExitStatus {
    for _, command := range [...]struct {
        group string
        subcommands.Command
    }{
        {"help", subcommands.CommandsCommand()},    // Implement "commands"
        {"help", subcommands.FlagsCommand()},       // Implement "flags"
        {"help", subcommands.HelpCommand()},        // Implement "help"
        {"top", NewTop1()},                         // Our first top-level command, without args
        {"top", NewTop2()},                         // Our second top-level command, with args

The top.go contains the reusable top type , with various fields to store :

the constant values for which the Name, Synopsis and Usage will just be getters;
the local flags shared by both our commands, in this example prefix, as declared by SetFlags
a function field storing the runner function for the Execute method.

type top struct {
    name, synopsis, usage string // Reuse support
    prefix                string // Actual features
    run                   func(context.Context, *top, *flag.FlagSet, ...interface{}) subcommands.ExitStatus
}

func (cmd top) Name() string {
    return cmd.name
}

func (cmd top) Synopsis() string {
    return cmd.synopsis
}

func (cmd top) Usage() string {
    return cmd.usage
}

func (cmd *top) SetFlags(fs *flag.FlagSet) {
    fs.StringVar(&cmd.prefix, "prefix", "", "Add a prefix to the result")
}

func (cmd *top) Execute(ctx context.Context, fs *flag.FlagSet, args ...interface{}) subcommands.ExitStatus {
    if cmd.run == nil {
        log.Printf("command %s is not runnable", cmd.name)
        return subcommands.ExitFailure
    }
    return cmd.run(ctx, cmd, fs, args)
}

A limitation of function fields vs methods lies in the fact that they do not receive the instance as a hidden first parameter like methods do, meaning that with the same signature, they cannot access the instance, which Execute needs to access the local flag values.
The command instance must therefore be passed as an explicit extra argument, hence the *top parameter in the function signature, on line 15.

To remain as close as possible to a standard method expression, that parameter should be the first, but it would conflict with the Go usage of having context always be the first parameter of any function, hence the placement as the second parameter.

Commands using the shared type, like top2, define a runner function with that signature, which provides the runner with access to the instance as if they were methods; and their factory function assigns that runner to the instance during the instance initialization process, like in this code in top2.go :

func top2Execute(ctx context.Context, cmd *top, fs *flag.FlagSet, _ ...interface{}) subcommands.ExitStatus {
    if ctx.Value(VerboseKey).(bool) {
        fmt.Printf("In %s.\n", cmd.Name())
    }
    fmt.Println(strings.Join(append([]string{cmd.prefix}, fs.Args()...), ": "))
    return subcommands.ExitSuccess
}

func NewTop2() *top {
    const name = "top2"
    return &top{
        name:     name,
        synopsis: fmt.Sprintf("%s is an exemple top-level custom command with arguments", name),
        usage:    fmt.Sprintf("%s arg1 arg2 ...", name),
        prefix:   "",
        run:      top2Execute,
    }
}

This process removes coupling between commands and the Execute function. An unlimited number of commands can now reuse the shared command type, without having to modify the implementation in top.Execute.

$ go run . top1 -prefix today
today: hello
$ go run . top2 -prefix "the answer is" 42
the answer is: 42
$

Level 3 : Commanders

3.1 Procedural API vs object API

The code for that level is available on level3.1-object_api.

Up until now, we have been using the public functions exported by google/subcommands :

subcommands.Alias and subcommands.Register to register command instances
subcommands.ImportantFlag to label flags as flobal
subcommands.CommandsCommand, subcommands.FlagsCommand and subcommands.HelpCommand for the builtin help and documentation features

That procedure-based API is actually only a facade simplifying the underlying object API : exepct for Alias, which does not depend on a Commander, each of these functions is actually a method call to the method with the same name on a default instance of the subcommands.Commander type, held in the exported mutable global variable subcommands.DefaultCommander, which is initialized when importing github.com/google/subcommands by the init pseudo-function, using the subcommands.go fragment below :

// DefaultCommander is the default commander using flag.CommandLine for flags
// and os.Args[0] for the command name.
var DefaultCommander *Commander

func init() {
    DefaultCommander = NewCommander(flag.CommandLine, path.Base(os.Args[0]))
}

// Register adds a subcommand to the supported subcommands in the
// specified group. (Help output is sorted and arranged by group
// name.)  The empty string is an acceptable group name; such
// subcommands are explained first before named groups. It is a
// wrapper around DefaultCommander.Register.
func Register(cmd Command, group string) {
    DefaultCommander.Register(cmd, group)
}

This is but one case of a model frequently used throughout the Go standard library, for instance by packages flag (flag.CommandLine) or http (http.DefaultClient).

We can switch to that object model by only changing our root.go file, in which we replace every procedural call (except Alias) by a method call on that default Commander instance :

func Execute(ctx context.Context) subcommands.ExitStatus {
    commander := subcommands.DefaultCommander

    for _, command := range [...]struct {
        group string
        subcommands.Command
    }{
        {"help", commander.CommandsCommand()},      // Implement "commands"
        {"help", commander.FlagsCommand()},         // Implement "flags"
        {"help", commander.HelpCommand()},          // Implement "help"
        {"top", NewTop1()},                         // Our first top-level command, without args
        {"top", NewTop2()},                         // Our second top-level command, with args
        {"top", subcommands.Alias("1", NewTop1())}, // An alias for our top1 command

    } {
        commander.Register(command.Command, command.group)
    }

    debug := flag.Bool("debug", false, "Show debug information")
    verbose := flag.Bool("v", false, "Be more verbose")
    commander.ImportantFlag("v")
    flag.Parse()

    ctx = context.WithValue(ctx, DebugKey, *debug)
    ctx = context.WithValue(ctx, VerboseKey, *verbose)

    return commander.Execute(ctx, "meaning")

The program keeps operating in the exact same way as the previous version.

3.2 Using custom commanders

The code for that level is available on level3.2-custom_commander.

Our next step will be to create our own Commander instance, rather that using the shared mutable instance exported by google/subcommands, so we can keep its initialization and mutations under control. We will therefore create it in the Execute function in our root.go file :

func Execute(ctx context.Context) subcommands.ExitStatus {
    // See subcommands.DefaultCommander
    commander := subcommands.NewCommander(flag.CommandLine, os.Args[0])

Nothing else deviates from the previous version, but we are now one step closer to a testable code structure, by removing our dependency on the global subcommands.DefaultCommander.

Time to finish the work and make Execute completely testable by removing all remaining dependencies to other globals.

3.3 Creating a testable command structure

The code for that level is available on level3.3.

Our code is still encumbered by a number of obstacles before it can be unit tested easily :

Problem	Solution
global standard and error outputs	inject from `main`
global `os.Args`	inject from `main`
global `flag.CommandLine`	replace by a custom `flag.FlagSet`
global `flag.Bool`, `flag.Parse`	replace by equivalent `flag.FlagSet` methods
hidden use of global `log.std`	remplace by a custom `log.Logger`
`flag` calling `os.Exit`	modify `flag.FlagSet` creation options
command creation in `Execute`	inject from a factory

Once we apply these changes, our latest version of main.go should become the only place where globals are still accessed :

func main() {
    sts := cmd.Execute(context.Background(), os.Stdout, os.Stderr, os.Args, log.LstdFlags, cmd.Describe)
    os.Exit(int(sts))
}

We also need to create a Describe factory instantiating our commands in root.go, so that Execute now receives them instead of creating them itself :

type description struct {
  group   string
  command subcommands.Command
}

func Describe(outW io.Writer, logger *log.Logger) []description {
    return []description{
        {"top", NewTop1(outW, logger)},                         // Our first top-level command, without args
        {"top", NewTop2(outW, logger)},                         // Our second top-level command, with args
        {"top", subcommands.Alias("1", NewTop1(outW, logger))}, // An alias for our top1 command
    }
}

…and our new version of Execute now receives all that data it needs by injection instead of referencing imported variables or creating instances itself.

func Execute(ctx context.Context,
    outW io.Writer, // Standard output for command results
    errW io.Writer, // Error output for logs
    args []string,  // CLI args to avoid depending on the flag global
    logFlags int,   // Log flags to make error message testing easier
    describe func(outW io.Writer, logger *log.Logger) []description, // Command registration descriptions
) subcommands.ExitStatus {
    // Do not depend on log.Default().
    logger := log.New(errW, "", logFlags)

    // Create a flag.FlagSet from args to avoid depending on global os.Args.
    // Continue on error to support testing instead of the ExitOnError on flag.CommandLine
    if len(args) < 1 {
        logger.Printf("Expected at least one argument for the program name, got none")
        return subcommands.ExitFailure
    }
    fs := flag.NewFlagSet(args[0], flag.ContinueOnError)

    // Create a custom commander to avoid depending on global flag.CommandLine and os.Args
    commander := subcommands.NewCommander(fs, args[0])

    descriptions := []description{
        {"help", commander.CommandsCommand()}, // Implement "commands"
        {"help", commander.FlagsCommand()},    // Implement "flags"
        {"help", commander.HelpCommand()},     // Implement "help"
    }
    if describe != nil {
        descriptions = append(descriptions, describe(outW, logger)...)
    }
    for _, command := range descriptions {
        commander.Register(command.command, command.group)
    }

    debug := fs.Bool("debug", false, "Show debug information")
    verbose := fs.Bool("v", false, "Be more verbose")
    commander.ImportantFlag("v")

    // Parse must not receive the program name, hence the slice.
    if err := fs.Parse(args[1:]); err != nil {
        // Our logger has been configured above.
        logger.Printf("Error parsing CLI flags: %v", err)
        return subcommands.ExitUsageError
    }

    ctx = context.WithValue(ctx, DebugKey, *debug)
    ctx = context.WithValue(ctx, VerboseKey, *verbose)

    return commander.Execute(ctx, "meaning", 42)
}

40-41: outW and errW are the two io.Writer passed by main().
41,47: errW is used to create a log.Logger
44: describe is the command description factory, passed by main()
55: a new FlagSet is created from received parameters, to break the dependency on os.Args
58: a Commander is create from that new FlagSet and received arguments, to break the dependency on flag.CommandLine, and indirect dependency to os.Args
66: outW and our new logger are passed to describe so it can pass them into the command factory functions, allowing command instances to work on injected writers instead of global variables.
72, 73: global flags are defined on the new FlagSet instead of global flag.CommandLine
77: flags parsing is run on the FlagSet instead of global flag.CommandLine

Execute is no longer using any global variable, and passes injected dependencies to the top1 et top2 command instances. We have to modify these too, so that they use the injected writer and logger instead of shared globals. This is how it applies in top1.go :

func top1Execute(ctx context.Context, cmd top, fs *flag.FlagSet, _ ...any) subcommands.ExitStatus {
    if ctx.Value(VerboseKey).(bool) {
        cmd.logger.Printf("In %s.\n", cmd.Name())
    }
    if l := fs.NArg(); l != 0 {
        cmd.logger.Printf("%s expects no arguments, called with %d: %v", cmd.Name(), l, fs.Args())
        return subcommands.ExitFailure
    }
    message := "hello"
    if cmd.prefix != "" {
        message = strings.Join(append([]string{cmd.prefix}, message), ": ")
    }
    fmt.Fprintln(cmd.outW, message)
    return subcommands.ExitSuccess
}

func NewTop1(outW io.Writer, logger *log.Logger) *top {
    const name = "top1"
    return &top{
        logger:   logger,
        name:     name,
        outW:     outW,
        prefix:   "",
        run:      top1Execute,
        synopsis: fmt.Sprintf("%s is an exemple top-level custom command without arguments", name),
        usage:    name,
    }
}

The code in top2 is very similar, and the top type in top.go has been extended as show at lines 34-40 above to store the injected dependencies.

This enables us to establish a 100% test coverage rate very simply, without needing any mocking tool.

$ go test -race -count=1 -cover ./cmd
ok      github.com/fgm/subcommands_demo/cmd 0.035s  coverage: 100.0% of statements
$

Level 4 : adding nested subcommands

The code for that level is available on level4.1-nesting.

With more complex applications, some commands will need a hierarchy of subcommands, but google/subcommands does not describe how to implement them, leading to a false impression that it does not support them.

That support is actually a consequence of our use of a custom commander created from received parameters passed to subcommands.NewCommander : since flags are now defined on a local FlagSet and arguments received, all it takes to implement a command accepting subcommands is to create a Commander instance from the arguments not consumed after resolving the command itself.

This is how our new top3 works, with its sub31 and sub32 nested commands, to be found in top3.go. Feature-wise, it is similar to top1 and top2, but its runner function top31Execute now includes the subcommands handling logic in addition to the base top3 logic.

func top3Execute(ctx context.Context, cmd top, topFS *flag.FlagSet, args ...any) subcommands.ExitStatus {
    name := cmd.Name()
    if ctx.Value(VerboseKey).(bool) {
        cmd.logger.Printf("In %s.\n", cmd.Name())
    }
    // Handle command called without subcommands.
    if topFS.NArg() == 0 {
        return top3Internal(ctx, cmd, topFS, args)
    }

    // Create a flag.FlagSet from args to use only remaining args
    // Continue on error to support testing.
    fs := flag.NewFlagSet(cmd.Name(), flag.ContinueOnError)

    // Create a custom commander to restart evaluation below this command.
    commander := subcommands.NewCommander(fs, name)

    descriptions := []description{
        {name, commander.CommandsCommand()}, // Implement "commands"
        {name, commander.FlagsCommand()},    // Implement "flags"
        {name, commander.HelpCommand()},     // Implement "help"
        {name, NewSub31(cmd.outW, cmd.logger)},
        {name, NewSub32(cmd.outW, cmd.logger)},
    }
    for _, command := range descriptions {
        commander.Register(command.command, command.group)
    }

    // Parse must not receive the command name.
    if err := fs.Parse(topFS.Args()); err != nil {
        cmd.logger.Printf("Error parsing %s flags: %v", name, err)
        return subcommands.ExitUsageError
    }

    return commander.Execute(ctx, fs)
}

near the beginning (lines 29-31), the command checks whether it receives arguments, which could be subcommand names. If there are none, it then transfers control to its own internal logic, which performs the work desired for top3 without a subcommand.
otherwise, it implements a logic similar to the one in our root Execute function :
- create a local FlagSet,
- create a local Commander from that FlagSet and available arguments,
- register subcommands on the Commander, including builtin commnds,
- parse available arguments for local flags and arguments on the nested commands
- run the subcommand designated by arguments, if any.

$ go run . top3
hello top3
$ go run . top3 commands
commands
flags
help
sub31
sub32
$ go run . top3 help sub31
sub31  -prefix string
        Add a prefix to the result
$ go run . top3 sub31 -prefix today
today: hello sub31
$

Level 5 : beyond `NewCommander`

5.1 Controlling outputs beyond our code

The code for that level is available on level5.1-newcommander.

Although we have injected everything for our own logic to support unit testing, some error outputs remain, from the subcommands and flag packages themselves, in error situations, and can be seen when running tests. We could have foreseen it, because NewCommander does not take any argument for the standard or output error to use by the instance it returns.

$ go test -race -count=1 -v  ./cmd 2>&1 | grep -vE '(CONT|RUN|PASS|PAUSE)'
flag provided but not defined: -bad
Usage: Test_Execute <flags> <subcommand> <subcommand args>

Subcommands for help:

  ...snip (33 lines total)...

flag provided but not defined: -bad
top1  -prefix string
        Add a prefix to the result
flag provided but not defined: -bad
c  -prefix string
        Add a prefix to the result
ok      github.com/fgm/subcommands_demo/cmd 0.036s
$

However just because that redirection is not available at instance creation, it is still available after creation, as we do here in root.go for top-level commands and global flags :

func Execute(ctx context.Context,
    outW io.Writer, // Standard output for command results
    errW io.Writer, // Error output for logs
    args []string, // CLI args to avoid depending on the flag global
    logFlags int, // Log flags to make error message testing easier
    describe func(outW io.Writer, logger *log.Logger) []description, // Command registration descriptions
) subcommands.ExitStatus {
  /* ...snip... */
    fs := flag.NewFlagSet(args[0], flag.ContinueOnError)
    fs.SetOutput(errW)

    // Create a custom commander to avoid depending on global flag.CommandLine and os.Args
    commander := subcommands.NewCommander(fs, args[0])
    commander.Output = outW
    commander.Error = errW

Redirecting the additional properties removes most of the parasitic output we were seeing during tests :

$ go test -race -count=1 -v  ./cmd 2>&1 | grep -vE '(CONT|RUN|PASS|PAUSE)'
flag provided but not defined: -bad
flag provided but not defined: -bad
ok      github.com/fgm/subcommands_demo/cmd 0.045s
$

The two remaining uncaptured messages appear to be a bug in version 1.2.0 of google/subcommands, for which a pull request exists.

We have now covered all the useful features of the google/subcommands package. All that remains is two related sets of features which are unlikely to be used in most practical cases.

5.2 Commanders introspection

The code for that level is available on level5.2-visit.

The subcommands.Commander type includes exported methods allowing code to examine its internal flags and commands data.

Let us create a top-level visit command, similar to those we have been writing, and which demonstrates all these methods. It is located in the visit.go file.

The VisitAll methods implements the Visitor pattern, allowing introspection of the flags on the Commander instances:

func visitAll(commander *subcommands.Commander, w io.Writer) {
    fmt.Fprintln(w, "VisitAll show all the commander flags:")
    tw := tabwriter.NewWriter(w, 0, 0, 1, ' ', tabwriter.Debug)
    fmt.Fprintln(tw, "\tName\tDefault\tValue\tUsage\t")
    commander.VisitAll(func(f *flag.Flag) {
        fmt.Fprintf(tw, "\t%s\t%s\t%s\t%s\t\n", f.Name, f.DefValue, f.Value, f.Usage)
    })
    tw.Flush()
    fmt.Fprintln(w)
}

Running "demo visit":
VisitAll show all the commander flags:
 |Name  |Default |Value |Usage                  |
 |debug |false   |false |Show debug information |
 |v     |false   |true  |Be more verbose        |

The VisitAllImportant methods is almost identical, but only visits flags labeled as important:

VisitAllImportant only shows the "important" flags:
 |Name |Default |Value |Usage           |
 |v    |false   |true  |Be more verbose |

The VisitGroups method implements the Visitor pattern, allowing introspection of the CommandGroup instances created from the group names used when registering commands. It is quite limited, as the public methods on the CommandGroup type do not provide access to the grouped commands:

func visitGroups(commander *subcommands.Commander, w io.Writer) {
    fmt.Fprintln(w, "VisitGroups only visits the command groups, not the commands:")
    tw := tabwriter.NewWriter(w, 0, 0, 1, ' ', tabwriter.Debug)
    fmt.Fprintln(tw, "\tName\tLen\t")
    commander.VisitGroups(func(group *subcommands.CommandGroup) {
        fmt.Fprintf(tw, "\t%s\t%d\t\n", group.Name(), group.Len())
    })
    tw.Flush()
    fmt.Fprintln(w)
}

VisitGroups only visits the command groups, not the commands:
 |Name |Len |
 |help |3   |
 |top  |5   |

The VisitCommands method implement the Visitor pattern in a much more useful way, providing access to the Command instances registered on the Commander, and from there to their CommandGroup and local flags:

func visitCommands(commander *subcommands.Commander, w io.Writer) {
    fmt.Fprintln(w, "VisitCommands visits the commands themselves:")
    tw := tabwriter.NewWriter(w, 0, 0, 1, ' ', tabwriter.Debug)
    fmt.Fprintln(tw, "\tGroup\tName\tSynopsis\tFlags\t")
    commander.VisitCommands(func(group *subcommands.CommandGroup, c subcommands.Command) {
        fs := flag.NewFlagSet("visit", flag.ContinueOnError)
        c.SetFlags(fs)
        var flags []string
        fs.VisitAll(func(f *flag.Flag) { flags = append(flags, f.Name) })
        fmt.Fprintf(tw, "\t%s\t%s\t%s\t%s\t\n", group.Name(), c.Name(), c.Synopsis(), strings.Join(flags, ", "))
    })
    tw.Flush()
    fmt.Fprintln(w, "(command usage omitted for readability)")
}

VisitCommands visits the commands themselves:
 |Group |Name     |Synopsis                                                            |Flags  |
 |help  |commands |list all command names                                              |       |
 |help  |flags    |describe all known top-level flags                                  |       |
 |help  |help     |describe subcommands and their syntax                               |       |
 |top   |top1     |top1 is an exemple top-level custom command without arguments       |prefix |
 |top   |top2     |top2 is an exemple top-level custom command with arguments          |prefix |
 |top   |1        |top1 is an exemple top-level custom command without arguments       |prefix |
 |top   |top3     |top3 is an exemple top-level custom command with nested subcommands |prefix |
 |top   |visit    |demoes commander Visit* functions                                   |       |
(command usage omitted for readability)

Because these visitor callbacks do not receive the Commander instance, the visit command will need to carry an injected commander instance, as commands are not aware of the Commander invoking them. We perform this with a new feature in root.go :

type CommanderAware interface {
    SetCommander(commander *subcommands.Commander)
}

func Execute(ctx context.Context,
  /* ...snip... */
    for _, command := range descriptions {
        if vc, ok := command.command.(CommanderAware); ok {
            vc.SetCommander(commander)
        }
        commander.Register(command.command, command.group)
    }

Our visit command is an instance of the visitCmd type, which includes a subcommands.Commander field, and an associated visitCmd.SetCommander setter, making it a CommanderAware implementation, allowing the Execute function to inject it with the active Commander as show on lines 48-49.

5.3 Overriding builtin output

The code for that level is available on level5.3-explain.

The last and most esoteric feature in google/subcommands is the ability to swap the implementation of the builtin CommandsCommand, FlagsCommand and HelpCommand, using matching function fieled on the subcommands.Commander type :

Explain is a func(io.Writer) field, initialized in NewCommander with the unexported method expression cdr.writer
ExplainGroup is a func(io.Writer, *CommandGroup) field, initialized in NewCommander with unexported function explainGroup
ExplainCommand is a func(w io.Writer, c subcommands.Command) field, initialized in NewCommander with unexported function explain

Since these function fields are missing receiver access although they need it, our new explain command, defined in cmd/explain.go, has to be CommanderAware, so it can replace the fields with functions having access to the Commander instance, using the one instance it was injected with :

Examples below first show the default builtin output, then the output of a customized version

ExplainCommand : builtin output, then YAML API version:

Running "demo explain":

Demoes overriding ExplainCommand to describe top3.
- Builtin version using unexported explain:
top3  -prefix string
        Add a prefix to the result
- Custom version in YAML format:
top3:
    flags:
        - name: prefix
          default: ""
          usage: Add a prefix to the result
    synopsis: top3 is an exemple top-level custom command with nested subcommands
    usage: top3

ExplainGroup : builtin output, then YAML API version:

Demoes overriding ExplainGroup.
- Builtin version using private explainGroup:
Subcommands for help:
    commands         list all command names
    flags            describe all known top-level flags
    help             describe subcommands and their syntax  
...snip...
- Custom version in YAML format, without access to group contents:
help: 3
top: 6

Explain : builtin output, then neutral version suggesting how to write a custom description

Demoes overriding Explain.
- Builtin version using private commander.explain:
Usage: demo <flags> <subcommand> <subcommand args>


Top-level flags (use "demo flags" for a full list):
  -v=false: Be more verbose

- Custom version, build from commander methods:
Use any commander.(Explain|Visit)* methods

Conclusions

Pros
- google/subcommands is an excellent fit for small, simple projects
- it is uncommonly lightweight, and very stable due to its having no dependencies at all
- starting with it does not prevent evolving from a simple version to a multi-level command hierarchy with a mix of local and global flags
- it does not make it difficult to provide unit test coverage for the commands themselves instead of limiting one to the service code the commands call upon.
Cons
- it does not natively merge CLI flags and environment variables
- it does not natively include merging configuration files with flags
- creating complex commands with validation hooks has no direct support: all steps will have to be implemented together in the Execute method
- it does not natively include the persistent flags concept, that is flags defined at one level and also applying to subcommands. These can however be passed manually, for example in the context like we did for global flags.
- creating multi-level command hierarchies requires a bit more work than with some other tools.

To summarize, that package is likely to be your tool of choice for your next microservice or CLI tool, but not if you are building your next giant monolith.

Tip of the day: how to edit Ansible Jinja2 templates in JetBrains IDEs

Frédéric G. MARAND — Sun, 26 Dec 2021 13:36:29 +0000

One annoying missing feature in most JetBrains IDEs is the lack of builtin support for Jinja2 templates, used notably as the Ansible template engine.

One paid solution exists, the OrchidE plugin, but there is a free workaround. Can you guess which one ?

One template engine which originally got a lot of inspiration from Jinja is the PHP Twig engine made popular by Symfony.

Even today, in spite of the divergent evolutions of Jinja 3 and Twig 3, it is close enough in basic syntax for the Twig plugin to be used to edit Ansible Jinja templates. And the JetBrains Twig plugin has a very useful feature in the form of "sub-templates". That is, providing syntax support for both the template engine itself (the braces content) and the template destination (YAML, INI, Apache Config, etc).

This relies on having template files named (file).(target type).twig : on such files the Twig plugin applies Twig (Jinja) syntax to the braces content and target type syntax to the rest of the file. For example, we could create a myvhost.apacheconf.twig file and have it be edited as a "Jinja" template targeting an Apache config file, with the following steps:

Go to Preferences / Editor / File Types, add *.apacheconf to the Apache config type; validate.
Create your Apache vhost template as my_vhost.j2.
Symlink it to my_vhost.apacheconf.twig.
Enjoy !

If you don't mind going for a more unusual, but even simpler approach, you can even forego the *.j2 extension altogether, and name your templates that way, also when referencing them as template.src arguments: no symlinks, no problems.

Just don't forget to document why your templates are named that way in your project README file: other devs could be surprised to see that unusual extension.

GoLand tip of the day: clean up GoLand leftover cache and log folders

Frédéric G. MARAND — Thu, 25 Mar 2021 08:15:05 +0000

Goland

When GoLand is upgraded from one version to the next, the deployment process adds a new set of configuration, cache, and log directories. But it does not remove the versions associated to the previous versions of the IDE, to ease rolling back to the previous version if an upgrade happens to cause an issue.

So after some time, removing these now-unused folders will regain some disk space, possibly more than 1 GB. Here are their respective locations and content on macOS:

Role	Size indication	Configuration variable	Default location
Configuration	3 to 10 MB	`idea.config.path`	`~/Library/Preferences/Gogland1.0` `~/Library/Preferences/Goland.(version)`
Plugins	50 to 100 MB	`idea.plugins.path`	`~/Library/Application Support/Gogland1.0` `~/Library/Application Support/GoLand-EAP` `~/Library/Application Support/GoLand ~/Library/Application Support/GoLand.(version)`
Caches	200 MB or (much) more	`idea.system.path`	`~/Library/Caches/Gogland1.0` `~/Library/Caches/GoLand2018.(version)`
Logs	1 to 10 MB	`idea.log.path`	`~/Library/Logs/Gogland1.0` `~/Library/Logs/GoLand2018.(version)`

On Windows and Linux, lookup the location using the name of the configuration variable in the third column.

Notice that the Toolbox app uses similar locations

Tip of the day: patching legacy Drupal 7 projects with Composer

Frédéric G. MARAND — Tue, 26 Jan 2021 09:30:58 +0000

Some late Drupal 7 projects use Composer for project structure and tooling, even though they don't use Composer for the Drupal requirements proper. In that case, the normal Drupal 8/9 patching process using cweagans/composer-patches is not available, since dependencies are not handled with Composer.

So, is there a way to apply patches cleanly from Composer ? Sure there is! Let's see how.

The problem

A typical situation happens like this:

Legacy Drupal 7 project keeps on being used, but not upgraded to Drupal 8 / Drupal 9 or BackdropCMS/li>
Since it has to be maintainable, it is best on the then-current technique of git submodules, with each contrib being brought in as a separate submodule
But then, more recent updates bring in Composer, e.g. to start using modern PHP components, starting the autoloader from settings.local.php; or using it for additional tools like Drush or PHPunit.
When a module needs to be patched, the legacy drush-based process becomes an issue with submodules and Composer, and yet, because components are not downloaded by Composer, using cweagans/composer-patches is not an options

Now, how can the patch process be handled in composer.json, just as is would be in any modern Composer deployment ?

How to use Composer patches with zero plugin

There is actually a very simple mechanism for this: using Composer hooks on command events.

That's all we need. Let's take an example: how to add the PHP 7.4 compatibility patch for Gmap, available on the https://www.drupal.org/project/gmap/issues/3118279 issue ?

First step: creating a `composer patch` command

First we need to defined our patches and how to apply them. Here is what needs to be added to composer.json for a start:

{  
  ...misc sections...  
  "require": { ... },  
  "require-dev: { ... },  
  "scripts": {    
    "patch": [      
      "cd www/sites/all/modules/contrib/gmap; curl https://www.drupal.org/files/issues/2020-03-06/3118279-3.patch | patch -p1"   
    ],  
  },  
  "type": "project"
}

The scripts section allows adding custom commands to Composer
In this case, we add a new patch command, to apply all patches
Since commands can be just shell scripts, we create one line for each patch, made of two parts:
- cd (the module directory)
- apply the patch straight from its download URL, taking advantage of the fact that the patch command takes patches form its standard input
These two have to be kept as a single shell command, because the working directory is only retained for the direction of the shell subprocess, hence the ;

At this step, we can now use composer patch to apply our patches on demand. But we can do better: apply the command automatically.

Second step: applying patches on composer install/update

Remember, Composer includes these command hooks for install and update. To quote the Composer documentation:

post-install-cmd: occurs after the install command has been executed with a lock file present.
post-update-cmd: occurs after the update command has been executed, or after the install command has been executed without a lock file present.

Beyond invoking shell scripts, Composer can also invoke its custom commands by prepending them with an @ like Symfony services

{
  ...snip...
  "scripts": {    
    "patch": [      
      "cd www/sites/all/modules/contrib/gmap; curl https://www.drupal.org/files/issues/2020-03-06/3118279-3.patch | patch -p1"   
    ],    
    "post-install-cmd": ["@patch"],    
    "post-update-cmd": ["@patch"]  
  },`
   ...snip...
}

Now, any time we run composer install or composer update, Composer will apply the patches without having to type composer patch manually.

However, there are still two issues, which is especially annoying if the site uses submodules, but would also apply if all code is committed to the project:

After one of composer install or composer update, the files are patches. So any subsequent install or update will find them modified, and attempt to preserve them or rollback, requiring many interactions
The modified files are visible to git, which could invite us to commit them, which is probably not desired.

Third step: cleaning up

So we can just add an automatic cleanup before the install/update commands, so that our patch command, which runs after these tasks, will find the git checkout pristine

For projects committing contrib dependencies, a simple git reset --hard is all it takes, but that will not work with submodules, so we need to be a bit smarter:

{
  ...snip...
  "scripts": {
    "patch": [
      "cd www/sites/all/modules/contrib/gmap; curl https://www.drupal.org/files/issues/2020-03-06/3118279-3.patch | patch -p1"
    ],
    "subclean": ["git submodule update --init -f"],
    "pre-install-cmd": ["@subclean"],
    "pre-update-cmd": ["@subclean"],
    "post-install-cmd": ["@patch"],
    "post-update-cmd": ["@patch"]
  },
  ...snip...
}

This time we define a composer subclean command to clean up submodules, and we register it to run before both install and update.

Now, we can run composer install and composer update multiple times, and find the project clean everytime.

Finally, come commit time, we can just run composer subclean before committing, or even as a git pre-commit hook, and the patch-induced changes in our contributions will be reset, at no risk of getting committed.

Gérer les erreurs en Go 1.14+

Frédéric G. MARAND — Thu, 27 Feb 2020 07:33:23 +0000

La gestion des erreurs a été beaucoup enrichie en Go 1.13 et ses améliorations ont commencé à être appliquées en Go 1.14.

Cette présentation, donnée au meetup Golang Paris chez Deezer le 26/02/2020 présente les mécanismes standard error, les tentatives d'évolution check / handle et try / defer, et le mécanisme d'error wrapping finalement validé depuis Go 1.13.

Showcase: Migrating FranceTVSport.fr to Drupal 8 and Symfony 4

Frédéric G. MARAND — Mon, 18 Mar 2019 13:03:18 +0000

The opening talk as DrupalCamp Paris 2019 was a presentation given by Thomas Jolliet (FranceTV) and yours truly about how we rebuilt FranceTV Sport to a Symfony 4 / headless Drupal 8 combo.

The most salient points of the talk are probably the "defense in depth" mechanisms we built for scalability and fault tolerance, and the business results, like -85% full page load time, -65% speed index, or +50% iOS app traffic.

Go internals unofficial wiki

Frédéric G. MARAND — Mon, 04 Mar 2019 11:09:22 +0000

Google Go = Golang

The "Go internals unofficial wiki" on goin.wikispot.org used to be a valuable resource for developers interesting in the internal operation of the Go runtime, but disappeared when wikispot.org went down in 04/2014.

I spent a couple of hours this morning to restore and reformat its contents to the Golang section on the Audean wiki.

Enjoy... but responsibly : its content was fresh in 2014, not last week...

GoLand tip of the day: when Make run configurations don't find go

Frédéric G. MARAND — Fri, 12 Oct 2018 11:31:06 +0000

The problem

In a GoLand 2018.3 EAP run configuration, a Makefile run configuration finds the go binary on macOS, but not on Ubuntu, causing make targets like this one to fail:

# Makefile
client/test.wasm: client/main.go  # Building WASM
        GOARCH=wasm GOOS=js go build -o client/test.wasm client/main.go

$ /usr/bin/make -f (some edited path)/Makefile
client/test.wasm/bin/sh: 1: go: not found
make: *** [client/test.wasm] Error 127
$

Simple enough to fix...

The diagnostic

On the command line, the same GOARCH=wasm GOOS=js go build -o client/test.wasm client/main.go line works normally. Looks like a path issue in Goland or the plugin. Checking in the Makefile:

# Makefile
client/test.wasm: client/main.go    
        # "${PATH}" 
        # Building WASM 
        GOARCH=wasm GOOS=js go build -o client/test.wasm client/main.go

$ # Building WASM
make: *** [client/test.wasm] Error 127# "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
$

So indeed, on at least this version, the PATH is not taken into account. Let's fix this.

The solution

Of course, one could just add the Go bin path to the tasks in these targets, but this is uncalled for on other environments, so we can just remind GoLand to add this path to the run configuration.

Edit the Run configuration
Edit the Environment variables field
Add a definition for the PATH variable taken from an echo $PATH on the command line
Apply and close

Oddities

As usual, Make has its own peculiar charm, requiring the variable name to use the braced format shown above instead of a plain $PATH.

Trying to extend the PATH variable by just adding something like $PATH:/usr/local/go/bin or ${PATH}:/usr/local/go/bin won't work:

in the first case, Make will receive the result of evaluation $P, then the ATH string, so other binaries beyond the ones in /usr/local/go/bin will be missing
in the second case, Make detects a variable referencing itself and aborts the task

Tip of the day: how to compare Guitar Pro files

Frédéric G. MARAND — Sun, 07 Oct 2018 09:21:56 +0000

Music

The problem

When working on compositions in a multi-member band (currently 5 in AnotherDay), each member tends to create their own Guitar Pro files to avoid overwriting the previous versions created by other band members. And .gp files tend to accumulate for the same song. At which point I start to wonder: "I know there's this riff in one of the versions, but which one ?" or "Are these two files really different or could I just merge them by editing a few notes, or remove one altogether ?"

The problem, of course, is that Guitar Pro, even in version 7.5, has no builtin file comparison tool. How can one compare these files ?

Diagnostic

If GP7 doesn't have a buitin solution, how about comparing the files themselves ? Problem: the file format is not documented. But a file score.gp command returns Zip archive data, at least v2.0 to extract. So let's unzip the file to see what it looks like:

$ unzip score.gp
Archive:  score.gp   
creating: Content/  
  inflating: Content/BinaryStylesheet
  inflating: Content/LayoutConfiguration
  inflating: Content/PartConfiguration
  inflating: Content/Preferences.json  
  inflating: Content/score.gpif
  inflating: VERSION
$ cat VERSION
7.0
$ cd Content
$ file *
BinaryStylesheet:    data
LayoutConfiguration: X11 SNF font data, MSB
firstPartConfiguration:   data
Preferences.json:    ASCII text, with no line terminators
score.gpif:          XML 1.0 document text, UTF-8 Unicode text, with very long lines$`

A quick look at the content of the files confirms that the actual score data is contained in the score.gpif file.

$ tidy -q -xml -indent score.gpif | head -20  
<?xml version="1.0" encoding="utf-8"?>
<GPIF>
  <GPVersion>7</GPVersion>
  <GPRevision required="12021" recommended="12023">
  12024</GPRevision>
  <Encoding>
    <EncodingDescription>GP7</EncodingDescription>
  </Encoding>
  <Score>
    <Title>
      <![CDATA[]]>
</Title>
    <SubTitle>
      <![CDATA[]]>
</SubTitle>
    <Artist>
      <![CDATA[]]>
</Artist>
    <Album>
      <![CDATA[]]>

Ahah... so it's an XML document with a GPIF top-level element. A quick search with that keyword lands us on a file called "The Guitarpro 7 (".GP") file format" by the MuseScore developers, currently available at https://musescore.org/sites/musescore.org/files/2017-08/the_guitarpro_7_file%20format.doc. And its content allows one to understand the contents of these scores. I bet it would even come in handy to manually edit/repair a file corrupted by the frequent crashes of Guitar Pro.

But that's not the goal for today, we can now compare these scores more or less easily.

The solutions

First solution is to use the process described above and compare the score.gpif files using your favorite text comparison software. Works well, but can we do it with less command-line manipulation ?

Turns out, most of the process can be replicated in Guitar Pro itself, by using the new (?) MusicXML export: just navigate to File / Export / MusicXML... to export the scores to compare, and you directly get a pair of XML files which you can directly compare in your visual text comparison tool. Problem solved.

Munin plugins for Beanstalkd in PHP are now 1.0.0

Frédéric G. MARAND — Sun, 12 Aug 2018 19:16:07 +0000

In 2014, I created a PHP version of the Munin plugins for Beanstalkd introduced by AirBnB, but originally created in Python.

Fast forward four years, and after being adapted for compliance with the PHP-FIG PSR1, PSR2 and PSR12 standards, as well as the Zend coding standards, and with extra documentation, I published the first "stable" version 1.0.0 of the package today under the Apache APL-2.0 license.

To use it and ensure observability of your favorite job queue, follow the instructions in the INSTALL.md file. In short, it's just 3 steps:

clone the repository
install its dependencies with composer install
symlink the plugins to /etc/munin/plugins

That's it, you can now plot command rate, connections, jobs rate, age in queue, queue size by type, and timeouts frequency.

Blog tags:

Lighthouse 3.0 is out: Drupal 8.5 has killer performance

Frédéric G. MARAND — Sun, 05 Aug 2018 16:08:11 +0000

Lighthouse 3.0 is out: Drupal 8.5 has killer performanceFrederic MarandSun, 2018-08-05 18:08

Computing

As you may have noticed if you are a regular here, I just refreshed the theme on this site to a custom theme built on Bartik. And since Chrome had just been updated, I took the newly updated performance audit.

What more to say ?

How to debug Drupal 8 Functional tests with XDebug

Frédéric G. MARAND — Sun, 29 Jul 2018 19:12:18 +0000

How to debug Drupal 8 Functional tests with XDebug

The problem

While converting the old-style functional tests in the MongoDB project from WebTestBase to BrowserTestBase, I stumbled upon a problem: after following the drupal.org instructions to run PHPunit tests from PhpStorm, doing step-by-step debugging in the IDE worked nicely, but then stopped at the first $this->drupalLogin(); call, returning a 404 after a very long delay, although the same call running without debugging worked normally. What could be going on ?

The diagnostic

Stepping into UiHelperTrait::drupalLogin(), I noticed the delay occurent during the drupalGet('user/login') call early in the method, before submitting the account identification. Could it be the issue ?

Indeed, evaluating the result of the call via $this->getSession()->getStatusCode(); after that call returned show a 504 Timeout HTTP status code. And indeed, dumping the page with $this->getSession()->getPage()->getContent() show it to contain a typically timeout page.

Examining the logs showed the timeout happened at the PHP level, during the opposite direction call performed by Xdebug towards the debugging client, although such calls obviously worked since I'd been stepping through code for hours before that.

The solution

The solution was suggested by the always helpful and knowledgeable @berdir : PhpStorm happens to set a limit to the number of simultaneous connections it will accept for debugging, and that limit is 1 by default.

All of a sudden, it made sense: during classical debugging, the PHP interpreter from the host site establishes one connection to the client (the PhpStorm IDE), but when invoking the child site during such a call, a second instance of the interpreter is started and tries to connect too... but is blocked by the single connection limit of the default configuration.

So all it takes is increase the number of connections!

In PhpStorm 2018 and 2019 macOS, the settings can be found at PhpStorm / Preferences / Languages & Frameworks / PHP / Debug in the External connections part of the right pane, as shown below.

No need to restart the IDE or PHP, just restart the debugging run configuration, and it works beautifully.

DEV Community: Frédéric G. MARAND

CLI commands with google/subcommands

The problem scope

Level 1 : simple commands

1.1 Enabling builtin commands

1.2 Writing custom commands

1.3 Passing non-CLI arguments to commands

1.4 Grouping commands

1.5 Adding global or local flags

1.6 Labeling flags as important

Level 2 : Reusing command code

Level 3 : Commanders

3.1 Procedural API vs object API

3.2 Using custom commanders

3.3 Creating a testable command structure

Level 4 : adding nested subcommands

Level 5 : beyond NewCommander

5.1 Controlling outputs beyond our code

5.2 Commanders introspection

5.3 Overriding builtin output

Conclusions

Tip of the day: how to edit Ansible Jinja2 templates in JetBrains IDEs

GoLand tip of the day: clean up GoLand leftover cache and log folders

Tip of the day: patching legacy Drupal 7 projects with Composer

The problem

How to use Composer patches with zero plugin

First step: creating a composer patch command

Second step: applying patches on composer install/update

Third step: cleaning up

Gérer les erreurs en Go 1.14+

Showcase: Migrating FranceTVSport.fr to Drupal 8 and Symfony 4

Go internals unofficial wiki

GoLand tip of the day: when Make run configurations don't find go

The problem

The diagnostic

The solution

Oddities

Tip of the day: how to compare Guitar Pro files

The problem

Diagnostic

The solutions

Munin plugins for Beanstalkd in PHP are now 1.0.0

Lighthouse 3.0 is out: Drupal 8.5 has killer performance

How to debug Drupal 8 Functional tests with XDebug

The problem

The diagnostic

The solution

Level 5 : beyond `NewCommander`

First step: creating a `composer patch` command