DEV Community: Kevin K.

Configuration Tetris

Kevin K. — Sat, 03 Feb 2024 15:28:50 +0000

Recently this post titled, "Don't prefill Your config files" was making its round of the various news aggregators I follow. This topic closely tracks the last few articles I wrote about building command line applications and passing around an "Application Context."

I still plan on writing an article about loading configuration specifically for applications written in Rust, but at least for now I wanted to jot down some thoughts and best practices in a language agnostic manner.

Play Field

At it's most basic form the runtime context will contain only application defaults, i.e. the application isn't configurable at all.

There are a multitude of places an application could pull it's configuration from, not all applications support all methods. But if we look at the most common options, we'd see something like this for an application with six (6) configurable options.

NOTE
In the image above the "filled in" (hash checked) boxes are "values" provided.

A runtime context for a configurable option will always have a value, even if it's just an application default and that default is "no value" as expressed in whatever language happens to be used.

Rules

Notice the order of the Y axis is important. Higher rows will override values in lower rows.

The lower the row the more "implicit" the value is considered. Likewise, the higher the row the more "explicit" the value is considered because it is increasingly likely to have been set by the user directly and for this particular run of the program.

Falling Blocks

If we were to "fill in" various cells indicating that the user or system administrator changed some default to a new value we may end up with something that looks like this:

Notice how the final "Runtime Context" contains the "value" of the highest (most explicit) value.

That's really all there is to it. More explicit values should override implicit values.

Using a Trait in Derive CLIs

Kevin K. — Wed, 27 Dec 2023 19:31:24 +0000

In Part 5 we update our trait to work with Derive based CLIs and discuss the tradeoffs in doing so.

Previously On...

First a quick re-cap on how we used a trait in the Build method so we can compare.

Builder Trait Revisited

In the Builder method we defined a trait that enforced updating a context struct, performing some action with only that context struct, and then calling the next subcommand, if any.

A reminder about what our trait looked like:

pub trait Cmd {
    fn update_ctx(&self, _args: &clap::ArgMatches, _ctx: &mut Ctx) -> anyhow::Result<()> {
        Ok(())
    }
    fn run(&self, _ctx: &mut Ctx) -> anyhow::Result<()> {
        Ok(())
    }
    fn next_cmd<'a>(
        &self,
        _args: &'a clap::ArgMatches,
    ) -> Option<(Box<dyn Cmd>, &'a clap::ArgMatches)> {
        None
    }
}

We then implemented this trait on Zero Sized Type dummy structs:

struct Bustup;

impl Cmd on Bustup {
    // impl here...
}

We had a helper method implemented on the trait object itself that would do the walking and enforcing of update_ctx->run->next_cmd.

impl dyn Cmd + 'static {
    pub fn walk_exec(&self, args: &clap::ArgMatches, ctx: &mut Ctx) -> anyhow::Result<()> {
        self.update_ctx(args, ctx)?;
        self.run(ctx)?;
        if let Some((c, m)) = self.next_cmd(args) {
            return c.walk_exec(m, ctx);
        }
        Ok(())
    }
}

Finally, we kicked off this process in main by creating a Trait Object out of top level dummy struct. Because we used ZSTs, creating a Box<dyn Cmd> did not actually need to heap allocate anything.

let mut ctx = Ctx::default();
let cmd: Box<dyn Cmd> = Box::new(Bustup);
cmd.walk_exec(&args, &mut ctx)

Back to present day

Now that we're using a Derive based method we have a few issues if we were to try and port that trait directly as is to our current code.

CLI Value Structs

We are no longer using dummy structs and no longer passing around a clap::ArgMatches as the CLI values.

If we tried to use ZST structs and instead of clap::ArgMatches pass in our CLI structs we run into not being able to define the type of our CLI struct since they're all distinct types.

pub trait Cmd {
    fn update_ctx(
        &self,
        _args: /* ... */, // <--- What goes here?
        _ctx: &mut Ctx
    ) -> anyhow::Result<()> { }
}

Since we actually want to go from our CLI struct to the context struct in that method, it seems perfectly reasonable to just use our CLI struct instead of a ZST dummy struct.

Single Source of Truth in `run`

Deciding against a ZST dummy struct also has the consequence of now we have access to our CLI struct and the context struct during run which can break our "single source of truth rule."

There is no good answer this issue, and just like in Builder method we "just live with" the possibility of "stringly-typed" bugs or other oddities of the ArgMatches API; this is something we can live with.

For starters, we take self by shared reference (&self) which keeps us from being able to easily mutate anything and encourages us to use the Ctx struct if we needd to mutate a field such as for normalization or validation.

Additionally, I've just lived with allowing the CLI struct to be the source of truth for any values that meet all of the following rules:

No normalization is required for this CLI value (e.g. no need to deconflict or check multiple CLI value fields to determine what the user "really wanted")
This CLI value has no complex environment variable equivalents (clap can handle basic environment variables which is the reason for for "complex" qualifier)
The CLI value has no configuration file equivalents
THe CLI argument is not global or shared between other commands

If all of these rules hold, I allow myself to use the CLI struct's values, otherwise use the Context struct.

In practice this rarely an issue.

Using `Box<dyn Cmd>` will now allocate...a lot

Because we got rid of ZST dummy structs and instead decided to use the CLI value structs, that means creating a Box<dyn Cmd> out of them will allocate. These structs can also end up being quite large depending on the number of CLI arguments your program contains.

Luckily, all is not lost. It turns out we can just as easily use a &dyn Cmd as well since our CLI struct is created at program startup and kept around until after the end of the final run command.

This has the slight downside that you must keep the CLI struct around in memory, but I have yet to find a program where that is a problem as even though I said these structs can be quite large, it's still typically in well below a page size (4k) even for enormous CLIs.

NOTE
See [Appendix A][appendix_a] for a variation on this trait that I've used in the past where I wanted to be able to drop the CLI structs but allow the program to continue to run.

Switching from Box<dyn Cmd> to a &dyn Cmd is trivial in our case:

// src/cli.rs
pub trait Cmd {
    fn update_ctx(&self, _ctx: &mut Ctx) -> anyhow::Result<()> {
        Ok(())
    }
    fn run(&self, _ctx: &mut Ctx) -> anyhow::Result<()> {
        Ok(())
    }
    //                          👇 new
    fn next_cmd(&self) -> Option<&dyn Cmd> {
        None
    }
}

Our impl signature for Cmd changes slightly:

// src/cli.rs
impl<'a> dyn Cmd + 'a { /* .. */ }

And finally our main function changes ever so slightly:

// src/main.rs
fn main() -> Result<()> {
    let args = Bustup::parse();

    let mut ctx = Ctx::default();
    // 👇 new
    let cmd: &dyn Cmd = &args;
    cmd.walk_exec(&mut ctx)
}

Context

Alright, so we've adjusted our trait objects and our trait, but what about our Ctx? Since we are allowing ourselves to use the CLI structs as a partial context only for fields that are trivial, that leaves us with only the more complex, or global values that we need to move into our Ctx. For bustup that happens to only be the toolchain field used globally by all bustup target ....

// src/context.rs
#[derive(Default)]
pub struct Ctx {
    pub target_toolchain: Option<String>,
}

Our First Hard Decision

What about bustup update? It takes a [toolchain] argument, but it meets all the rules for us to just use the CLI struct instead of the context, right? Should we have just named that context field a more general toolchain and shared it?

My answer is, no.

Although those two arguments share a name, they may in the future not share anything else. I prefer to qualify fields with their path, or use child contexts.

Child Contexts

Just a quick aside, our toy example bustup isn't complex enough to really demo this, but in a real CLI it's not a bad idea to instead of qualifying all Ctx fields with their path, instead to use actual sub-contexts. For example something like:

struct Ctx {
    ctx_target: TargetCtx,
    ctx_update: UpdateCtx,
}

PRO TIP
You an also use things like Lazy initialized fields because due to how CLI command work, you typically only walk down a single command path, e.g. bustup update won't ever require anything out of bustup target .... Of course, there are some minor exceptions depending on what you're doing and how things are structured, but in general this is the case.

Enums and Subcommands

Because of how clap's Derive methods parse the subcommands we get back an enum representing one possible subcommand.

And then when we want to implement Cmd::next_cmd we have to do this tedious dance to match:

// src/cli.rs

/// not rustup
#[derive(Parser)]
pub struct Bustup {
    #[command(subcommand)]
    pub cmd: BustupCmd,
}

#[derive(Clone, Subcommand)]
pub enum BustupCmd {
    Update(update::BustupUpdate),
    Target(target::BustupTarget),
}

impl Cmd for Bustup {
    fn next_cmd(&self) -> Option<&dyn Cmd> {
        match self.cmd {
            BustupCmd::Update(cmd) => Some(cmd),
            BustupCmd::Target(cmd) => Some(cmd),
        }
    }
}

While this works, it's tedious to update all the various levels we could have subcommands and also must be updated any time we add or remove a subcommand!

If you don't mind taking on another dependency it turns out we can just implement Cmd directly on our enum in a low drama way!

We'll use the enum_delegate crate to reduce some boiler plate for us:

$ cargo add enum_delegate

Now we change our code slightly to register our trait, and utilize it on our BustupCmd enum:

// src/cli.rs

// 👇 new
#[enum_delegate::register]
pub trait Cmd { /* .. unchanged */ }

// 👇 new
#[enum_delegate::implement(Cmd)]
#[derive(Clone, Subcommand)]
pub enum BustupCmd {
    Update(update::BustupUpdate),
    Target(target::BustupTarget),
}

What does that buy us? Well all of the variants of BustupCmd implement Cmd so we actually just want to delegate directly to them.

Why that matters is when we call <Bustup as Cmd>::next_cmd what we really want is to call the that method on our inner value of whatever variant BustupCmd happens to currently have.

Well now that we have some boiler plate free method do that delegation, we just have to return the enum value itself!

impl Cmd for Bustup {
    // 👇 new
    fn next_cmd(&self) -> Option<&dyn Cmd> {
        Some(&self.cmd)
    }
}

Yes, this will call Cmd::update_ctx and Cmd::run on the enum itself as well, but since we have default implementation that do nothing it doesn't actually matter.

The only thing we have to do is write those three lines for each CLI struct that contains subcommands. No more changes even when we add and remove subcommands!

NOTE
This IMO is a big improvement over the builder method

The Rest of the Owl

Back to code-dump time! We have all the bits we need to implement our remaining commands and see how it all fits together.

Starting with bustup update:

// src/cli/cmds/update.rs
use crate::{cli::Cmd, context::Ctx};

// All else unchanged...

impl Cmd for BustupUpdate {
    fn run(&self, _ctx: &mut Ctx) -> anyhow::Result<()> {
        println!(
            "updating toolchain {}{}",
            self.toolchain,
            if self.force { " (forced)" } else { "" }
        );
        Ok(())
    }
}

// src/cli/cmds/target.rs
use crate::{cli::Cmd, context::Ctx};

// All else unchanged...

impl Cmd for BustupTarget {
    fn update_ctx(&self, ctx: &mut Ctx) -> anyhow::Result<()> {
        ctx.target_toolchain = Some(self.toolchain.clone());
        Ok(())
    }
    fn next_cmd(&self) -> Option<&dyn Cmd> {
        Some(&self.cmd)
    }
}

// src/cli/cmds/target/add.rs
use crate::{cli::Cmd, context::Ctx};

// All else unchanged...

impl Cmd for BustupTargetAdd {
    fn run(&self, ctx: &mut Ctx) -> anyhow::Result<()> {
        println!(
            "Adding the {} target to the {} toolchain",
            self.target,
            ctx.target_toolchain.as_ref().unwrap(),
        );
        Ok(())
    }
}

// src/cli/cmds/target/list.rs
use crate::{cli::Cmd, context::Ctx};

// All else unchanged...

impl Cmd for BustupTargetList {
    fn run(&self, ctx: &mut Ctx) -> anyhow::Result<()> {
        println!(
            "Listing {} targets for the {} toolchain",
            if self.installed { "installed" } else { "all" },
            ctx.target_toolchain.as_ref().unwrap(),
        );
        Ok(())
    }
}

// src/cli/cmds/target/remove.rs
use crate::{cli::Cmd, context::Ctx};

// All else unchanged...

impl Cmd for BustupTargetRemove {
    fn run(&self, ctx: &mut Ctx) -> anyhow::Result<()> {
        println!(
            "Removing the {} target from the {} toolchain",
            self.target,
            ctx.target_toolchain.as_ref().unwrap(),
        );
        Ok(())
    }
}

And that's it!

The full code can be found in the repository under the derive branch.

Summary

By structuring our code in such a way we can have clean delineations between "setup" and "action" code allowing us to split up code to run at the appropriate point of program execution logically.

It also helps us to enforce a little bit of rigor in appropriately using our context structs and keeping the "run" code single purpose.

There will be times when you want to manually "run" one of the commands from within code, and with this type of setup it should be mostly trivial as you only need to ensure the context struct is appropriately populated and then call Cmd::run.

NOTE
It's possible to use an associated type for an "output" of the Cmd::run method if you prefer, for these manual run scenarios. However I've found that it's easier to maintain by just setting a context field if run methods need to pass data between them which is actually quite rare.

Appendix A: Short Lived CLI Structs

If for some reason we do not want our CLI structs to remain in memory for the duration of the program such as either they take up significant memory or perhaps there is sensitive information that must be cleaned up, we can alter our Cmd trait slightly to allow dropping.

In this variation I think of the Cmd trait as an initializer, and thus I either rename run -> runner or remove it altogether.

Addressing these options in reverse order...

Removing `run`

In these scenarios your "real" main/run will just take the final Ctx as it's only argument. This option can get tricky though if you have multiple terminal methods that are distinct, e.g. git clone versus git push. Passing a Ctx to some final run method would probably need to match on some field to figure out what functionality it really needs to execute. Hence this method I typically only use in single purpose daemons where there is esseentially always a single well defined "running" state.

The code would look something like:

pub trait Cmd {
    fn update_ctx(&self, _ctx: &mut Ctx) -> anyhow::Result<()> {
        Ok(())
    }
    fn next_cmd(&self) -> Option<&dyn Cmd> {
        None
    }
}

fn main() -> anyhow::Result<()> {
    let args = Bustup::parse();

    let mut ctx = Ctx::default();
    let cmd: &dyn Cmd = &args;
    cmd.walk_exec(&mut ctx)?;

    // 👇 new
    drop(args);

    // 👇 new
    run(ctx)
}

// 👇 new
fn run(ctx: &mut Ctx) -> Result<()> { /* .. */ }

Renaming `run` to `runner`

In this variation run is more of an object initializer that returns something that can be run.

This means I typically make another trait to represent the terminal Run action that should be performed.

Depending on the use case you can also just not implement runner in any of the commaneds except the terminal one.

// 👇 new
pub trait Runnable {
    fn run(&self, _ctx: &mut Ctx) -> Result<()>;
}

pub trait Cmd {
    fn update_ctx(&self, _ctx: &mut Ctx) -> anyhow::Result<()> {
        Ok(())
    }
    // 👇 renamed                                👇 new
    fn runner(&self, _ctx: &mut Ctx) -> Option<Box<dyn Runable>> {
        None
    }
    fn next_cmd( &self,) -> Option<&dyn Cmd> {
        None
    }
}

impl<'a> dyn Cmd + 'a {
    pub fn walk_exec(&self, ctx: &mut Ctx) -> anyhow::Result<Option<Box<dyn Runnable>> {
        self.update_ctx(args, ctx)?;
        // 👇 new (return if we have something to run)
        if let Some(run) = self.runnere(ctx)? {
            return Ok(Some(run));
        }
        if let Some((c, m)) = self.next_cmd(args) {
            return c.walk_exec(m, ctx);
        }
        Ok(())
    }
}

fn main() -> anyhow::Result<()> {
    let args = Bustup::parse();

    let mut ctx = Ctx::default();
    let cmd: &dyn Cmd = &args;
    let runner = cmd.walk_exec(&mut ctx)?;

    // 👇 new
    drop(args);

    // 👇 new
    runner.run(ctx)
}

Derive Based CLIs

Kevin K. — Wed, 27 Dec 2023 19:31:12 +0000

In Part 4 of this series we go back to the beginning and look at Derive based CLIs, and how we can structure our program with the tradeoffs that a Derive based CLI brings.

Previously On...

Up until this point we'd been using clap's Builder approach to making a CLI and then defining a trait to help with some of the strict enforcement of how we want commands to run.

Builder CLIs are great in that they're the most flexible and fastest to compile, however they're more verbose to create and use at runtime by a significant margin.

This can make them more painful to maintain and use throughout a program and can even introduce subtle bugs when converting from the clap::ArgMatches to your Ctx structs.

Now we go back to the beginning and start anew using clap's Derive mode.

Going Back in Time

First, let's check out the main (empty) branch of our repository and create a new derive branch; we'll then add the dependencies we need:

$ git switch main
$ git swwitch -c derive
$ cargo add anyhow
$ cargo add clap -F derive

Notice we have to activate clap's derive feature to enable the derive macros. This is because they pull in additional dependencies such as syn and quote and will increase the compile time to some extent.

Like before, here's a quick code dump to get us started:

NOTE
Just like in our Builder example, we're enot attempting to show all the cool things you can do with clap.

// src/main.rs
mod cli;

use clap::Parser;

use crate::cli::Bustup;

fn main() -> anyhow::Result<()> {
    let args = Bustup::parse();

    todo!("Run the program!");

    Ok(())
}

And now the actual CLI:

NOTE
I like to break up the commands into individual files, however for a small toy example like this that doesn't normally make sense and adds more noise than necessary. As such I will not be showing files that just contain module definintions or re-exports. Likewise, you're also free to keep everything in a single file if you're following along and wish to do so.

// src/cli.rs
mod cmds;

use clap::{Parser, Subcommand};

use crate::cli::cmds::*;

/// not rustup
#[derive(Parser)]
pub struct Bustup {
    #[command(subcommand)]
    pub cmd: BustupCmd,
}

#[derive(Clone, Subcommand)]
pub enum BustupCmd {
    Update(update::BustupUpdate),
    Target(target::BustupTarget),
}

// src/cli/cmds/update.rs
use clap::Args;

/// update toolchains
#[derive(Clone, Args)]
pub struct BustupUpdate {
    /// toolchain to update
    #[arg(default_value = "default")]
    pub toolchain: String,

    /// forcibly update toolchain
    #[arg(short, long)]
    pub force: bool,
}

// src/cli/cmds/target.rs
mod add;
mod list;
mod remove;

use clap::{Args, Subcommand};

/// manage targets
#[derive(Clone, Args)]
pub struct BustupTarget {
    /// toolchain to update
    #[arg(short, long, default_value = "default")]
    pub toolchain: String,

    #[command(subcommand)]
    pub cmd: BustupTargetCmd,
}

#[derive(Clone, Subcommand)]
pub enum BustupTargetCmd {
    Add(add::BustupTargetAdd),
    List(list::BustupTargetList),
    Remove(remove::BustupTargetRemove),
}

// src/cli/cmds/target/add.rs
use clap::Args;

/// list targets
#[derive(Clone, Args)]
pub struct BustupTargetAdd {
    /// target to add
    #[arg(default_value = "default")]
    pub target: String,
}

// src/cli/cmds/target/list.rs
use clap::Args;

/// list targets
#[derive(Clone, Args)]
pub struct BustupTargetList {
    /// only list installed targets
    #[arg(short, long)]
    pub installed: bool,
}

// src/cli/cmds/target/remove.rs
use clap::Args;

/// remove a target
#[derive(Clone, Args)]
pub struct BustupTargetRemove {
    /// target to remove
    #[arg(default_value = "default")]
    pub target: String,
}

We can see that the CLI build properly by passing the --help flag to the various commands:

$ cargo run -q -- --help
Not rustup

Usage: bustup [COMMAND]

Commands:
  update  update toolchains
  target  manage targets
  help    Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

$ cargo run -q -- update --help
update toolchains

Usage: bustup update [OPTIONS] [toolchain]

Arguments:
  [toolchain]  toolchain to update

Options:
  -f, --force    Forcibly update
  -h, --help     Print help

$ cargo run -q -- target --help
manage targets

Usage: bustup target [OPTIONS] [COMMAND]

Commands:
  add     add a target
  list    list targets
  remove  remove a target
  help    Print this message or the help of the given subcommand(s)

Options:
  -t, --toolchain <toolchain>  toolchain to use [default: default]
  -h, --help                   Print help

$ cargo run -q -- target list --help
list targets

Usage: bustup target list [OPTIONS]

Options:
  -i, --installed              Only list installed targets
  -t, --toolchain <toolchain>  toolchain to use [default: default]
  -h, --help                   Print help

However, if we try to run it, we get a panic due to our todo!():

$ cargo run
thread 'main' panicked at src/main.rs:6:5:
not yet implemented: implement Bustup!
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Let's commit this as our starting point.

$ git commit -am "starting point"

Running the Program

So we have the basic CLI structure, and unlike the Builder method we already kind of have the structure we want at least file system/module wise.

It's also so much easier to just match our way into the code path we want since we have well defined enums that already appear to have their own self-contained context.

WARNING
Don't be fooled into believing the CLI structs are context structs!

Naive Matching and no `Ctx`

The naive method is to match on a particular subcommand, and dispatch to some run-like function that takes the CLI struct by reference as context. This is a common approach, but there are downsides. Let's implement this method for a single command bustup update just so we can contrast it later.

// src/cli/cmds/update.rs
use anyhow::Result;

impl BustupUpdate {
    pub fn run(&self) -> Result<()> {
        println!("updating toolchain...{}", self.toolchain);
        Ok(())
    }
}

And our main.rs:

// src/main.rs

fn main() -> anyhow::Result<()> {
    let args = Bustup::parse();

    match args.cmd {
        cli::BustupCmd::Update(update) => update.run(),
        _ => todo!("implement other subcommands"),
    }
}

We can see that it works by running the update command:

$ cargo run -- update
updating toolchain...default

$ cargo run -- update footoolchain
updating toolchain...footoolchain

This is already way less verbose than the Builder method, and we no longer have to worry about "stringly-typed" bugs.

NOTE
See the Appendix A to this post about how clap actually allows removing a few more layers of indirection that we're using in this example.

For extremely simple CLIs this can work. However, even simple CLIs can suffer by forgoing proper context structs.

A second talk about `Ctx`

It's so tempting to just use our CLI struct as the passed in context like we did above. And for a simple CLI, it'd probably be fine. But we're pretending to build a large and complex CLI.

The CLI structs we defined are only convenient CLI structs. They are not context!

Just like how we mentioned previously a --color and --no-color flag, in a CLI struct that may look something like:

#[derive(Parser)]
struct Args {
    #[arg(long, default_value = "auto", value_enum)]
    color: ColorChoice,

    #[arg(long)]
    no_color: bool,
}

#[derive(ValueEnum)]
enum ColorChoice {
    Auto,
    Always,
    Never,
}

Not only will all code that wants to determine if it should color something deconflict both Args::color and Args::no_color, but remember there are also environment variables and configuration files to handle!

Since we already went over a run-of-the-mill context struct being threaded through the program we will omit that exercise in the Derive based method because there is zero different between it and the Builder version.

This get interesting though when we use a context struct and try to use a trait to define some structure.

Next Time

In the next post we'll see how to modify our trait that we used in the Builder method for our current Derive method and discuss some more tradeoffs.

Appenix A: Removing More Layers

If we trim our example down to just the absolute essentials for a single bustup target add command, it would currently look like this (moving everything into a single file for clarity):

/// not rustup
#[derive(Parser)]
pub struct Bustup {
    #[command(subcommand)]
    pub cmd: BustupCmd,
}

#[derive(Clone, Subcommand)]
pub enum BustupCmd {
    Target(target::BustupTarget),
}

/// manage targets
#[derive(Clone, Args)]
pub struct BustupTarget {
    /// toolchain to update
    #[arg(short, long, default_value = "default")]
    pub toolchain: String,

    #[command(subcommand)]
    pub cmd: BustupTargetCmd,
}

#[derive(Clone, Subcommand)]
pub enum BustupTargetCmd {
    Add(add::BustupTargetAdd),
}

/// list targets
#[derive(Clone, Args)]
pub struct BustupTargetAdd {
    /// target to add
    #[arg(default_value = "default")]
    pub target: String,
}

Because bustup itself has no arguments or state, we can actually implement clap::Parser directly on an enum, and then use struct-variants for our enum which condenses it down even further. This would make the example:

/// not rustup
#[derive(Parser)]
pub enum Bustup {
    #[command(subcommand)]
    Target {
        /// toolchain to update
        #[arg(short, long, default_value = "default")]
        pub toolchain: String,

        /// manage targets
        #[command(subcommand)]
        pub cmd: BustupTargetCmd,
    },
}

#[derive(Subcommand)]
pub enum BustupTargetCmd {
    /// add a target
    Add {
        /// target to add
        #[arg(default_value = "default")]
        pub target: String,
    }
}

Even though the latter is far less verbose, I find that it gets cluttered when the CLI is of any reasonable size. But this is purely subjective, and other find the condensed version easier to grok.

Even if your top level command (i.e. bustup itself in our example) has actual arguments or state, you're still able to use enum struct-variants to remove a layer of indirection if you wish.

If you're interested in this style of de-duplication I'd also suggest looking at the clap documentation for flatten and more generally using the Args trait to define structs of common arguments for multiple command structs.

Using a Trait in Builder CLIs

Kevin K. — Wed, 27 Dec 2023 19:30:48 +0000

In Part 3 of this series we define a trait to help control the chaos and enforce some structure on our CLI programs.

Previously On...

In Part 2 we saw how using context structs was a better idea, but the method in which we niavely applied it has the potential to lead to pain the larger our CLI program grows.

Traits have entered the chat

We can solve this by using a trait to define the context updating, running, and traversing down the subcommand call stack. I call this Cmd and it looks like this:

// src/cli.rs

use clap::ArgMatches;
use anyhow::Result;

pub trait Cmd {
    fn update_ctx(&self, _args: &ArgMatches, _ctx: &mut Ctx) -> Result<()> {
        Ok(())
    }
    fn run(&self, _ctx: &mut Ctx) -> Result<()> {
        Ok(())
    }
    fn next_cmd<'a>(
        &self,
        _args: &'a ArgMatches,
    ) -> Option<(Box<dyn Cmd>, &'a ArgMatches)> {
        None
    }
}

// .. all else unchanged

There is a lot going on here, so let's break it down.

`Cmd::update_ctx`

First we have a method that is responsible for taking a CLI values struct (in our case clap::ArgMattches) and normalizing any values into Ctx.

By default this does nothing because there may be no context to update.

`Cmd::run`

This can be thought of as the entry-point to whatever functionality the command itself does.

Notice that it only takes a Ctx as context.

This is important because it means you can fully test your commands by building a mock Ctx and don't have to worry about mocking CLIs, etc. It also means you can use the same backing code for a CLI, a TUI, or a GUI because the code only need a Ctx to run. It doesn't know anything about CLI values or environments, or config files.

By default this method also does nothing allowing for intermediate subcommands that have no distinct action.

`Cmd::next_cmd`

Finally, we have a method for retrieving the next subcommand to run, if any.

Like all other methods, it also does nothing by default returning that there are no further subcomands to run.

The interesting bit about this method is the return when there is another subcommad to run. It returns a boxed trait object of something else that also implements Cmd.

Traversing the call-stack with `Cmd`

So we've added the definition of Cmd, but there is another bit of magic required to really reap the ergonomic benefits.

We implement a function on the trait object itself for doing the subcommand traversal.

// src/cli.rs

//  ... all else unchanged

impl dyn Cmd + 'static {
    pub fn walk_exec(&self, args: &ArgMatches, ctx: &mut Ctx) -> Result<()> {
        self.update_ctx(args, ctx)?;
        self.run(ctx)?;
        if let Some((c, m)) = self.next_cmd(args) {
            return c.walk_exec(m, ctx);
        }
        Ok(())
    }
}

This updates the context, runs the action, and if there is another subcommand to call it calls walk_exec on it.

WARNING
Some may smell a whiff of recursion and be wary of any such tactics, however by design subcommands cannot, so any such loop would be an inherent error where a stack overflow would be desirable. Additionally, I have never seen subcommands nested so heavily that an overflown stack would be possible. Even on very poorly designed subcommand-based CLIs the nesting does not go above 5-6 levels which is a far cry away from the hundreds that would be required to overflow the stack even on platforms like Windows which use small 1MB stacks by default.

PRO TIP
You can add another method on the trait object that allows for simply updating the context all the way down the call-stack without actually executing any of the code. This is super useful in testing where you don't actually want to run the code for real, but you'd like the context to be accurately updated. I usually call this walk_update_ctx and simply omit the selef.run(..) call while recursively calling walk_update_ctx instead of walk_exec

Our new `main`

Ok, so we have all the glue, now we just need to fill in the implementations. Let's start with main because it contains some interesting notes, here's the entire file:

// src/main.rs
mod cli;
mod context;

use crate::{
    cli::{Bustup, Cmd},
    context::Ctx,
};

fn main() -> anyhow::Result<()> {
    let args = cli::build().get_matches();

    let mut ctx = Ctx::default();
    let cmd: Box<dyn Cmd> = Box::new(Bustup);
    cmd.walk_exec(&args, &mut ctx)
}

Notice we create a boxed trait object and then call walk_exec on it to kick off the entire show.

NOTE
Box has a specialization for Zero-Sized Types (ZSTs) that doesn't actually allocate. cmd however is a pointer to a vtable.

Notice we reference cli::Bustup so let's take a look at that next.

CLI Structs

When using this pattern with clap's builder based CLIs I like to create empty Zero Sized structs which I can actually implement the Cmd trait on.

This forces me to keep no state except in Ctx, and allows the neat no-allocation for Box.

I also prefer to strictly name these structs after their path in the CLI hierarchy. For example Bustup is the top level command, and bustup target add would end up being named BustupTargetAdd. This is handy to always know exactly what command you're operating on in code.

For the top-level command I typically place it in the cli module, but that isn't strictly necessary.

// src/cli.rs
pub struct Bustup;

impl Cmd for Bustup {
    fn next_cmd<'a>(&self, args: &'a ArgMatches) -> Option<(Box<dyn Cmd>, &'a ArgMatches)> {
        match args.subcommand() {
            Some(("update", m)) => Some((Box::new(cmds::update::BustupUpdate {}), m)),
            Some(("target", m)) => Some((Box::new(cmds::target::BustupTarget {}), m)),
            _ => None,
        }
    }
}

Notice I don't need to do anything for run or update_ctx in our toy example, but adding code to execute at those hooks would be trivial.

One annoyance of the builder-pattern based CLIs is that we have manually handle dispatching to another subcommand at each level, but this is a minor complaint that could easily be handled by a macro if we so desired.

Looking over at our update.rs file next we see:

// src/cli/cmds/update.rs
pub struct BustupUpdate;

impl Cmd for BustupUpdate {
    fn update_ctx(&self, args: &ArgMatches, ctx: &mut Ctx) -> Result<()> {
        ctx.toolchain = args.get_one::<String>("toolchain").cloned();
        ctx.force = args.get_flag("force");
        Ok(())
    }

    fn run(&self, ctx: &mut Ctx) -> Result<()> {
        println!(
            "Updating {} toolchain...{}",
            ctx.toolchain.as_ref().unwrap(),
            if ctx.force { " (forced)" } else { "" }
        );
        Ok(())
    }
}

Intermediate Subcommands

In our toy example bustup target ... is an interesting example because it's an intermediate layer subcommand that has no functionality of it's own (although it could), and it defines a "global argument" (an argument that is present in all of it's child subcommands).

So even though we don't have a distinct action to take, we do need to update the context with our global CLI option. We also dispatch to our child subcommands exactly how the top level Bustup command does.

// src/cli/cmds/target.rs

pub struct BustupTarget;

impl Cmd for BustupTarget {
    fn update_ctx(&self, args: &ArgMatches, ctx: &mut Ctx) -> Result<()> {
        ctx.toolchain = args.get_one::<String>("toolchain").cloned();
        Ok(())
    }

    fn next_cmd<'a>(&self, matches: &'a ArgMatches) -> Option<(Box<dyn Cmd>, &'a ArgMatches)> {
        match matches.subcommand() {
            Some(("add", m)) => Some((Box::new(add::BustupTargetAdd), m)),
            Some(("list", m)) => Some((Box::new(list::BustupTargetList), m)),
            Some(("remove", m)) => Some((Box::new(remove::BustupTargetRemove), m)),
            _ => None,
        }
    }
}

The Rest of the Owl

Everything should fairly self-evident at this point, so we'll breeze through the next few files only calling out interesting aspects.

Just like BustupUpdate the following are all leaf subcommands and thus do not need to implement Cmd::next_cmd.

// src/cli/cmds/target/add.rs
pub struct BustupTargetAdd;

impl Cmd for BustupTargetAdd {
    fn update_ctx(&self, args: &ArgMatches, ctx: &mut Ctx) -> Result<()> {
        ctx.target = args.get_one::<String>("target").cloned();
        Ok(())
    }

    fn run(&self, ctx: &mut Ctx) -> Result<()> {
        println!(
            "Adding the {} target to the {} toolchain",
            ctx.target.as_ref().unwrap(),
            ctx.toolchain.as_ref().unwrap(),
        );
        Ok(())
    }
}

// src/cli/cmds/target/list.rs
pub struct BustupTargetList;

impl Cmd for BustupTargetList {
    fn update_ctx(&self, args: &ArgMatches, ctx: &mut Ctx) -> Result<()> {
        ctx.installed = args.get_flag("installed");
        Ok(())
    }

    fn run(&self, ctx: &mut Ctx) -> Result<()> {
        println!(
            "Listing {} targets for the {} toolchain",
            if ctx.installed { "installed" } else { "all" },
            ctx.toolchain.as_ref().unwrap(),
        );
        Ok(())
    }
}

// src/cli/cmds/target/remove.rs
pub struct BustupTargetRemove;

impl Cmd for BustupTargetRemove {
    fn update_ctx(&self, args: &ArgMatches, ctx: &mut Ctx) -> Result<()> {
        ctx.target = args.get_one::<String>("target").cloned();
        Ok(())
    }

    fn run(&self, ctx: &mut Ctx) -> Result<()> {
        println!(
            "Removing the {} target from the {} toolchain",
            ctx.target.as_ref().unwrap(),
            ctx.toolchain.as_ref().unwrap(),
        );
        Ok(())
    }
}

One Final Change

If you look at the repository's final code, you'll notice that one other thing I typically change, although it's not at all mandatory is to move the CLI construction into the respective modules instead of being fully contained at src/cli.rs. I find this lets me better encapsulate shared arguments and keep smaller subsets of the code in my head at any given time. To do this I usually place a fn build() -> clap::Command function in each module that contains a subcommand. For example, the cli.rs would then look this:

// src/cli.rs
pub fn build() -> Command {
    Command::new("bustup")
        .about("Not rustup")
        .subcommand(cmds::update::build())
        .subcommand(cmds::target::build())
}

And for example src/cli/cmds/target.rs:

// src/cli/cmds/target.rs

pub fn build() -> Command {
    Command::new("target")
        .about("manage targets")
        .arg(
            Arg::new("toolchain")
                .help("toolchain to use")
                .long("toolchain")
                .action(ArgAction::Set)
                .default_value("default")
                .short('t')
                .global(true),
        )
        .subcommand(add::build())
        .subcommand(list::build())
        .subcommand(remove::build())
}

And so on and so forth.

The full code can be found in the repository under the builder branch.

Summary for Builder based CLIs

The Builder Pattern is an older and more verbose way to define CLIs in clap. When used with this method for structuring your CLIs it has the benefit of being easy to enforce single-source-of-truth run functions and clap does a lot of the heavy lifting when it comes to providing our ArgMatches structs that we can use to update the Ctx.

Next Time!

That's it for Builder Pattern based CLIs!

In the next post we'll start back over using clap's Derive based approach and see how that changes our trait definitions and concerns.

Builder Based CLIs

Kevin K. — Wed, 27 Dec 2023 19:30:23 +0000

In Part 2 we dive into some actual code to see what it would look like to use these patterns for CLIs that utilize the clap Builder method.

Previously On...

Reminder I said, the commands and arguments of our bustup will only print messages to the terminal (without color...perhaps I'll fully explore Ctx initialization and well defined output coloring in another post).

Also another reminder, we don't care about the tool itself here, so forgive the brevity and code dump.

Let's gooooo

First, some setup:

$ cargo new bustup
$ cd bustup
$ git add .
$ git commit -m "Initial Commit"
$ git switch -c builder
$ cargo add clap anyhow

And now the code:

NOTE
This post is not attempting to show all the cool things you can do with clap, or even trying to use any of the developer niceties. That would get in the way of what we're trying to demo, so the code is naturally a little terse to keep the post ~~shorter~~ bearable.

// src/main.rs
mod cli;

fn main() -> anyhow::Result<()> {
    let args = cli::build().get_matches();

    todo!("Run the program!");

    Ok(())
}

And now the actual CLI:

// src/cli.rs
use clap::{Arg, ArgAction, Command};

pub fn build() -> Command {
    Command::new("bustup")
        .about("Not rustup")
        .subcommand(
            Command::new("update")
                .about("update toolchains")
                .arg(
                    Arg::new("toolchain")
                        .help("toolchain to update")
                        .action(ArgAction::Set)
                        .default_value("default"),
                )
                .arg(
                    Arg::new("force")
                        .short('f')
                        .long("force")
                        .help("Forcibly update")
                        .action(ArgAction::SetTrue),
                ),
        )
        .subcommand(
            Command::new("target")
                .about("manage targets")
                .arg(
                    Arg::new("toolchain")
                        .help("toolchain to use")
                        .long("toolchain")
                        .short('t')
                        .action(ArgAction::Set)
                        .default_value("default"),
                        .global(true),
                )
                .subcommand(
                    Command::new("add")
                        .about("add a target")
                        .arg(
                            Arg::new("target")
                                .help("The target to add")
                                .action(ArgAction::Set)
                            ),
                )
                .subcommand(
                    Command::new("list").about("list targets").arg(
                        Arg::new("installed")
                            .help("Only list installed targets")
                            .long("installed")
                            .short('i')
                            .action(ArgAction::SetTrue),
                    ),
                )
                .subcommand(
                    Command::new("remove")
                        .about("remove a target")
                        .arg(
                            Arg::new("target")
                                .help("The target to remove")
                                .action(ArgAction::Set)
                                .default_value("default"),
                        ),
                ),
        )
}

We can see that the CLI build properly by passing the --help flag to the various commands:

$ cargo run -q -- --help
Not rustup

Usage: bustup [COMMAND]

Commands:
  update  update toolchains
  target  manage targets
  help    Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

$ cargo run -q -- update --help
update toolchains

Usage: bustup update [OPTIONS] [toolchain]

Arguments:
  [toolchain]  toolchain to update

Options:
  -f, --force    Forcibly update
  -h, --help     Print help

$ cargo run -q -- target --help
manage targets

Usage: bustup target [OPTIONS] [COMMAND]

Commands:
  add     add a target
  list    list targets
  remove  remove a target
  help    Print this message or the help of the given subcommand(s)

Options:
  -t, --toolchain <toolchain>  toolchain to use [default: default]
  -h, --help                   Print help

$ cargo run -q -- target list --help
list targets

Usage: bustup target list [OPTIONS]

Options:
  -i, --installed              Only list installed targets
  -t, --toolchain <toolchain>  toolchain to use [default: default]
  -h, --help                   Print help

However, if we try to run it, we get a panic due to our todo!():

$ cargo run
thread 'main' panicked at src/main.rs:6:5:
not yet implemented: Run the program!
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Let's commit this as our starting point.

$ git commit -am "starting point"

Running the Program

So we have the basic CLI structure, now how should we structure our program?

Naive Matching and no `Ctx`

The naive method is to match on a particular subcommand, and dispatch to some run-like function that takes a clap::ArgMatches as a context. This is a common approach, but there are downsides. Let's implement this method for a single command bustup update just so we can contrast it later.

NOTE
I'm going to omit code examples that only contain things like declaring module structure for brevity. The full code is located in the repository if interested.

// src/cli/cmds/update.rs
use anyhow::Result;
use clap::ArgMatches;

pub fn run(args: &ArgMatches) -> Result<()> {
    println!(
        "updating toolchain...{}",
        args.get_one::<String>("toolchain")
    );
    Ok(())
}

And finally, our main.rs:

// src/main.rs
use crate::cli::cmds::update;

fn main() -> anyhow::Result<()> {
    let args = cli::build().get_matches();

    match args.subcommand() {
        Some(("update", args)) => {
            update::run(args)?;
        }
        _ => todo!("implement other subcommands"),
    }

    Ok(())
}

We can see that it works by running the update command:

$ cargo run -- update
updating toolchain...default

$ cargo run -- update footoolchain
updating toolchain...footoolchain

This is a perfectly valid approach! For a small number of subcommands, or CLIs with single-layer subcommands this approach is usually fine. However, it can start to go sideways quickly when using multiple layers of configuration or nested subcommand layers, especially when context/run-actions need to happen at each individual layer.

Adding `Ctx`

It's so tempting to just use clap::ArgMatches as the passed in context like we did above. And for a simple CLI, it'd probably be fine. But we're pretending to build a large and complex CLI.

Based on everything we learned when talking about Ctx above, we're already convinced we should be using a Context Struct. And we know initializing and updating one can be a complex process.

But we're starting small and we won't be adding configuration files or environment variables to complicate things in this post. So let's just create our Ctx and pass that to update::run:

// src/main.rs
mod cli;
// 👇 new
mod context;

//                             👇 new
use crate::{cli::cmds::update, context::Ctx};

fn main() -> anyhow::Result<()> {
    let args = cli::build().get_matches();

    match args.subcommand() {
        Some(("update", args)) => {
            // 👇 new
            let ctx = Ctx::from_update(args);
            update::run(&ctx)?;
        }
        _ => todo!("implement other subcommands"),
    }

    Ok(())
}

// src/context.rs
use clap::ArgMatches;

pub struct Ctx {
    pub toolchain: String,
}

impl Ctx {
    pub fn from_update(args: &ArgMatches) -> Self {
        Self {
            toolchain: args.get_one::<String>("toolchain").unwrap().to_string(),
        }
    }
}

// src/cli/cmds/update.rs
use anyhow::Result;

// 👇 new
use crate::context::Ctx;

//         👇 new
pub fn run(ctx: &Ctx) -> Result<()> {
    //                                 👇 new
    println!("updating toolchain...{}", ctx.toolchain);
    Ok(())
}

This also works!

But for more complex CLIs, this will get tedious and error prone as well for a few reasons:

In the above code we're creating the Ctx from scratch which wouldn't be an option with nested subcommands that each need to update a context
As we nest subcommands the code to update the context and call the next subcommand is going to become pure boilerplate noise.

Next Time

In the next post we'll see how to use traits to perform some magic and enforce structure on what could otherwise become unbridled chaos.

CLI Contexts

Kevin K. — Wed, 27 Dec 2023 19:29:44 +0000

How should one structure their CLI programs? I've seen this question a few times, and over the years I've come up with some patterns that I really like. This post explores the variations of these patterns and trade-offs between them.

Intro

I recently came across this question (and associated answer) on the clap repository. The answer given is a good one. But I wanted to expand with my own findings and practices, which spurred the motivation for this post.

Type of CLI

First things first. We need to know which type of CLI you're building. For these purposes there are two broad categories:

Subcommand based
Single Command Argument based

An example of "subcommand based" would be cargo; each function is represented by a distinct "command" that is a child (or grand-child, or great-grand-child, etc.) of the top level command cargo. i.e:

$ cargo build
$ cargo clean
$ cargo new

Whereas an example of a single command based CLI would be ripgrep, where the tool has a single command, in this case rg that is controlled solely by arguments.

Most often, although not always, subcommand based CLIs each command is a distinct function, and at some level they could have perhaps been distinct tools. For example cargo build builds the Rust program, whereas cargo clean cleans all build artifacts. These could probably have been implemented as cargo-build and cargo-clean respectively. But because those two commands are closely related in purpose (handling Rust projects) it makes far more sense to combine them into a single cargo command which has the added benefit of being able to easily share code, cohesion in things like error messages and arguments, and not cluttering the file system with cargo-* commands, etc.

Similarly, single command based CLIs often, but not always, have a single function. In Ripgrep's case, that is searching through files. This function can be altered in ways via arguments, but ultimately it's still just searching through files.

NOTE
Ripgrep is actually a good example of the "but not always" saying. Technically, Ripgrep can also manage and assist the user with "type definitions" that allow searching exclusively through specific file types. This function is controlled through the --type-* arguments, which when used add, remove, or list various "type definitions" and thus don't actually do the "sole functionality" of searching through files. These could have been made subcommands, such as rg type [OPTIONS] however, this set of functionality is minor enough (and the only other functionality beyond searching files) that making it a subcommand could have detracted away from the primary purpose of searching needlessly. Subcommands can, and do, complicate matters.

Ok, so now we know what kind of CLI we're building, we can start to decide on what pattern to use.

This post will focus almost exclusively on subcommand based, because it's the most interesting and complicated.

NOTE
I plan to write a "BONUS POST" that discusses a single-command based pattern for completeness sake if that's the route you're taking.

The CLI - bad rustup...bustup

So we're making a subcommand based CLI. The example we'll use is a toy example to keep this already long post manageable since the actual tool itself isn't important here. This tool will mimic a tiny tiny subset of rustup extremely poorly...i.e. we're just going to print some messages.

We'll call this bustup.

NOTE
Why rustup? Because it implements a pattern that will be useful to explore, and that's using several nested layers deep of subcommands. For example cargo (and other tools like git) typically only go one layer deep (i.e. git clone, cargo update, etc.) whereas rustup can go several more layers deep, such as rustup toolchain add .... This will be important as the pattern we'll use has additional considerations when nesting more than a layer or two deep.

Parsing Library

We also need to decide on a parsing library. I'm biased towards clap so that's what we'll be using. However, this pattern should more or less be applicable to any command line argument parsing library.

clap like several other parsing libraries supports two different modes of building a CLI. The so-called "Builder Pattern" and the "derive based" method.

We'll cover both methods because the pattern used is different in interesting ways for each.

But this also means we'll have to re-implement bustup twice using the different methods. Another reason to keep bustup minimally simple.

`bustup` Command Structure

Here's what we'll implement:

bustup update [toolchain] [--force]
bustup target add [--toolchain=<TOOLCHAIN>]
bustup target remove [--toolchain=<TOOLCHAIN>]
bustup target list [--toolchain=<TOOLCHAIN>] [--installed]

Context Structs

Something all (or at the very least most) CLI applications deal with the idea of a "runtime context."

When doing an actual action in code, there is normally some defined "context" informing the runtime code on what actions to take and how to take them. For example our bustup update will need some contextual knowledge about which toolchain the user wishes to update.

There are many different ways to store and pass around runtime context. The most common ways are via:

The CLI value structs themselves
Context Structs
Globals

I will omit globals, as it's typically an anti-pattern and frowned upon especially in Rust as these context objects are typically mutable and having global mutable state precludes many multi-threading possibilities and negates many of Rust's safety features.

That leaves us with the CLI value structs, and context structs.

As the heading may suggest, Context Structs are what this post will focus on. But before doing so I will mention below in the appropriate sections about CLI value structs; as the types of value structs and how they're used is different depending on the mode of clap one is using.

For now, suffice it to say that Context Structs are nearly always a better approach, even though they may seem redundant in some circumstances.

Meeting `Ctx`

A Context Struct (usually abbreviated Ctx) is just a local struct containing normalized values from all configuration sources (whereas CLI value structs typically only get their values from the CLI or perhaps from the environment depending on argument parsing library features).

In our bustup update example, using the toolchain as the context we may have a struct similar to:

struct Ctx {
    toolchain: String
}

NOTE
It's common to prefer borrowing of values in some context structs, especially if the value is only temporary. For instance:
struct Ctx<'a> {
    toolchain: &'a str,
}
But for brevity and simplicity I'm leaving out any such concerns in order to focus on the topic of this post.

An important distinction about context structs is that they contain the normalized configuration values.

Single Source of Truth

The primary benefit of using Context Structs is to provide A Single Source of Truth to runtime code.

Although our toy bustup doesn't use overriding flags, if we pretend it does for the sake of argument real quick let's see what could happen.

Imagine something like our bustup update taking a --confirm flag that says to ask the user for confirmation before downloading and installing any updates, and also a --no-confirm flag that says not to ask the use first.

NOTE
Naively it may seem silly to provide both --confirm and --no-confirm, especially if one of those values is the default behavior of the program (for instance if either flag is provided it acts as if --confirm was used). However, it's actually a super useful pattern to provide because it allows users to set up custom aliases. For example a user normally does not want to be asked for confirmation, so they set up alias bustup-up='bustup update --no-confirm'. Now one day, they are on a metered network connection and decide they'd like to confirm before downloading a massive update, so run:
$ bustup-up --confirm

... this expands to ...

$ bustup update --no-confirm --confirm

As it stands, we'd probably start by only passing in the CLI values directly to the run_update function. This would mean it's the function's responsibility for knowing about and deconflicting both --confirm and --no-confirm which is something it should not have to care about. Additionally, if we had other subcommands with similar flags, they too would have to know about and handle these cases.

This typically leads to people passing in duplicate context. e.g. they'd pass in the CLI values raw and a boolean flag of some kind:

fn run_update(cli_values: &Args, confirm: bool) { /* .. */ }

To properly write run_update now one needs to know the difference between the two locations which contain values for whether or not we should prompt the user for confirmation.

And that's not all!

Technically, the CLI values are only a single layer in a delicious cake that is program configuration and on their own make for bad context....

Initializing `Ctx`

Although it's out of scope for this post to fully explore this topic; for completeness sake let's take an ultra-brief look at what it's like to initialize a Ctx struct.

Let's pretend we had a program that could use pretty colors for terminal output. So we have a Context Struct that looks like:

struct Ctx {
    // If `true` pain the terminal
    color: bool
}

Let's further assume our program supports system-wide, user-level, and project-level configuration files, as well as CLI flags and environment variables for controlling settings.

For something as simple as turning color on or off, the ways to control such a setting could be:

A default value of Auto meaning to check if STDOUT is a TTY or not
A system-wide config file setting
A user-level config file setting
A project level config file setting
A set of environment variables
- E.g. could be any of TERM, NO_COLOR, or perhaps PROGRAM_COLOR
A set of CLI flags
- E.g. --color=... or --no-color

This seems like madness! Yet its exactly how a well behaved CLI program should function!

So a full initialization sequence probably looks something like this:

At program startup do something like Ctx::default()
Load system-wide configuration files, if any (e.g. /etc/...) perhaps in a Config<System> struct
- Normalize values if needed
- Update the Ctx with something like a Config::update_ctx(&mut ctx)
- Load user-level configuration files, if any (e.g. ~/.config/...) perhaps in a Config<User> struct
- Normalize values if needed
- Update the Ctx with something like a Config::update_ctx(&mut ctx)
- Load environment variable values, if any
  - This step may be done, or partially done by your CLI parsing library, however things like TERM or NO_COLOR which are only loosely related to your --color and --no-color flags will probably be handled manually via something like a Ctx::update_from_env method
- Parse CLI values
- Normalize values if needed
- Update the Ctx with something like a Config::update_ctx(&mut ctx)
- You now have a fully initialized Ctx

The benefit being any actual functionality in your program no longer needs to worry about all of that and can just get passed an instance of &Ctx as an argument check a quick ctx.color field and trust whatever is written there!

Next Time!

In the next post we'll actually dive in using the clap Builder based method to see what using this pattern looks like in practice.

Generically Bloated

Kevin K. — Mon, 06 Feb 2023 00:17:55 +0000

Rust generics are great, but they can bloat the final code, and have negative effects on compile times. This post discusses a "trick" to help fight those effects.

There exists a decently common pattern to combat these issues given a few preconditions are met. This pattern is used heavily in the standard library and popular crates.

However, when talking to a colleague recently they were unaware of this pattern (to be fair I wasn't fully aware of this pattern either until just a few years ago). This conversation made me wonder if there are others who also haven't yet seen this method which can have pretty big downstream affects!

Prelude

This post only addresses code bloat from generic functions, not generic structs.

Basics

Let's start by just showing a basic set of generics in Rust, and why some problems pop up.

When you use generic parameters in Rust, the compiler will actually generate a copy of your code for each concrete type it finds. For example:

fn genric<T>(param: T) {
  // code...
}

fn main() {
  generic(30);    // T = i32
  generic("foo"); // T = &'static str
}

Turns into something like:

fn genric_i32(param: i32) {
  // code...
}

fn genric_str(param: &'static str) {
  // code...
}

fn main() {
  generic(30);
  generic("foo");
}

While this produces extremely efficient code at runtime, there are a few potential downsides:

If generic<T>() is of substantial size (in compiled binary form), that code is also duplicated leading to potential "bloat"
(Although I haven't gone spelunking into the compiler code to know for sure; I believe that) all the generated (roughly duplicate) code must also be compiled and optimized separately! This is a lot more code for rustc and LLVM to churn through
generic<T>() cannot be fully compiled until the compiler knows all the concrete types that will be used

Library Authors Beware

These concerns are most applicable to library authors, but can affect binary authors as well when their binary is split into a binary-consuming-an-internal-library.

Where this shows up is downstream consumers filing issues related to code bloat (it turned out many concrete types were used for T), or slow compile times due to your library (your library heavily relies on generics that cannot be compiled "earlier" or the compiler is churning through all that extra code).

Preconditions

I mentioned earlier that the pattern we're about to discuss isn't a silver bullet and cannot be used in all circumstances. There is an important precondition that need to be met:

Important
The generic code must have a "preferred type" which usually means the generic
parameter is bounded and that bound represents some kind of type conversion

This should become more clear as we go along.

Immediate Dispatch to the Rescue

The solution is to immediately dispatch to a known type. In hindsight this may seem somewhat obvious, however if it doesn't that's OK! We're about to see this in action!

Note
This example will be extremely contrived in order to demonstrate the effects

In Action

First, lets define a trait that will be used as a generic bound.

trait Speak {
  fn speak(&self) -> String;
}

Next, we define a generic function that can be used with any type that implements the Speak trait.

fn generic_speak<T: Speak>(param: &T) {
  println!("It says: {}", param.speak());
}

This could be thought of as our library boundary. Perhaps the library also has some concrete types it uses internally, but that is not a requirement.

It does not matter if the concrete types are internal, external, or a mix.

Lets define two concrete types that implement said trait as example:

struct Cat;
impl Speak for Cat {
  fn speak(&self) -> String {
    "meow".into()
  }
}

struct Dog;
impl Speak for Dog {
  fn speak(&self) -> String {
    "woof".into()
  }
}

Finally, we use that generic function with both a Cat and a Dog struct:

Note
We're creating a binary because it's easier to demonstrate :)

fn main() {
  let whiskers = Cat;
  let spot = Dog;
  generic_speak(&whiskers);
  generic_speak(&spot);
}

If we run it, we get what you probably expect:

$ cargo run --quiet
It says: meow
It says: woof

Counting Functions

Lets first back up and prove the claims I made at the beginning, that our generic function actually generates two concrete functions. To see that we can either cargo-llvm-lines or cargo-bloat.
I use both pretty extensively, so lets compare the output of both just for fun!

aside: `cargo-llvm-lines`

First, we'll use cargo-llvm-lines to see the amount of LLVM IR generated:

Note
Both tools can generate a lot of output, so we could be greping it down to
size but I'll make some readability edits instead.

$ cargo llvm-lines
Lines               Copies            Function name
-----               ------            -------------
[ .. snip .. ]
    80 (4.9%, 50.3%)   2 (4.9%, 17.1%)  generic_speak
     5 (0.3%, 98.3%)   1 (2.4%, 82.9%)  <Cat as Speak>::speak
     5 (0.3%, 98.6%)   1 (2.4%, 85.4%)  <Dog as Speak>::speak
[ .. snip .. ]

Notice we do, in fact, have two copies of generic_speak (as can be seen by the Copies column) and one implementation function each for the concrete types (e.g. <Cat as Speak>::speak which is the code from where we did impl Speak for Cat).

`cargo-bloat`

Contrasting the above with cargo-bloat which shows the binary size of each function:

Note
cargo-bloat by default only shows the largest 99 functions, but our example
is to tiny we tell it show us the largest 999 functions so we can be sure our
functions show up in the output.

$ cargo bloat -n 999
Analyzing target/debug/blog_demo

File  .text     Size     Crate Name
[ .. snip .. ]
0.0%   0.1%     193B blog_demo generic_speak
0.0%   0.1%     193B blog_demo generic_speak
0.0%   0.0%      44B blog_demo <Dog as Speak>::speak
0.0%   0.0%      44B blog_demo <Cat as Speak>::speak
[ .. snip .. ]

Like carog-llvm-lines we can see that we do, in fact, have two copies of generic_speak and one implementation function each for the concrete types (e.g. <Cat as Speak>::speak which is the code from where we did impl Speak for Cat).

Note
For the rest of the post I'm going to omit the actual trait implementations
(<Dog as Speak>::speak) for brevity, because those don't change, we still
implement those traits with actual code!

With cargo-bloat we see that in the final binary both copies of generic_speak are 193 bytes (for a total of 386 bytes).

Note
As the name implies I tend to prefer cargo-bloat when working on bloat
issues, because it looks at the final compiled binary size as opposed to just
the LLVM IR with cargo-llvm-lines. Although I prefer cargo-llvm-lines
when working on compile times.

Generic Bloat

Although contrived, one thing I like about this example is by pure line count, the generic_speak function looks like almost no code! But this brings up another great source of bloat (especially when combined with the issue
described in this post!): macros!

aside: macros

Using an LSP expand function in my editor (although similar could be done with something like cargo-expand) we see generic_speak actually expands to something like this:

Note
You can't run this directly as it uses private rust internals, but it gives a
good view into the kind things macros expand to

fn generic_speak<T: Speak>(param: &T) {
{
  {
    std::io::_print(std::fmt::Arguments::new_v1(
      &["It says: "],
      &[std::fmt::ArgumentV1::new(
        &(param.speak()),
        std::fmt::Display::fmt,
      )],
    ));
  }
}

Trying Immediate Dispatch

If we take a step back we see that Speak::speak just produces a String and all the code inside generic_speak really only needs that String to operate.

The trick is that we We can add a private internal function that accepts the actual type we actually needed/wanted (i.e. String in this case) instead of just sticking with the generic parameter.

Now generic_speak looks like this:

fn generic_speak<T: Speak>(param: &T) {
  fn generic_speak_string(param: String) {
    println!("It says: {param}");
  }
  generic_speak_string(param.speak());
}

Note
The function could be external as well, it doesn't need to be defined within
the outer function scope. Although if it's not used anywhere else it makes
sense to define it within internal scope.

If we re-run cargo-bloat we now see:

$ cargo bloat -n 999
Analyzing target/debug/blog_demo

File  .text     Size     Crate Name
[ .. snip .. ]
0.0%   0.1%     160B blog_demo generic_speak::generic_speak_string
0.0%   0.0%      37B blog_demo generic_speak
0.0%   0.0%      37B blog_demo generic_speak
[ .. snip .. ]

We still have two concrete implementations of generic_speak since we still have a generic function, however notice the actual code inside has gone down from 193 bytes to just 37 bytes (essentially enough to dispatch the other function). Now, all our "real code" lives in non-generic (and thus not
duplicated) generic_speak_string internal function (160 bytes).

Doing the quick math of our duplicate generic functions + the new internal function we get a total 234 bytes versus the original total of 386 bytes!

This is just a contrived example, but imagine a real library with multiple generics and actually substantial sized functions!

Additionally, that inner function (generic_speak_string) is able to be fully compiled right away since all it's types are fully known.

A Note on Inlining and Release Builds

You may have noticed if you compiled the examples in release mode, the functions don't appear at all!

$ cargo bloat -n 999 --release | grep speak
$

This is because the code is pretty trivial and Rust/LLVM are able to inline all the code and optimize this away. However, in a real world library that's not always possible.

Warning
When using this trick, you may need in some cases tell Rust/LLVM not to
inline your private wrapped function by using #[inline(never)] if it turns
out the function is just getting inlined into all the generic functions
again. But that should only be done when you're sure that's the case, and the
additional code bloat is worse than the performance lost by not-inlining.

Why not just use a `String` as the parameter instead of the generic?

Because this was a contrived example.

Also there are times you'll do something almost exactly like this purely for ergonomics. For example:

// Accepts anything that can be converted to &str cheaply
fn takes_stringish<S: AsRef<str>>(param: S) { /* code */ }

Sure, we could just take &str as the parameter, but some types may be cheaply yet not ergonomically converted to a &str. Using the generic parameter can give our library a nice ergonomic boost.

An Even More Contrived Example

At the risk of making this post too long, to show another example of one where it's less ergonomic to ask the user for exactly what we want.

Cases often comes up around type generics, e.g. we'd prefer to ask for a impl Iterator<Item = AsRef<str>> when we're working with roughly a Vec<&str> internally. Forcing the user to do the conversion isn't very ergonomic.

Let's do exactly this with our previous code, but instead of AsRef<str> we'll use our Speak as the bound.

If we change our generic_speak and main functions:

fn generic_speak<T: Speak>(param: impl Iterator<Item = T>) {
  for item in param {
    println!("It says: {}", item.speak());
  }
}

fn main() {
  use std::collections::HashSet;

  let whiskers = vec![Cat, Cat];
  let mut spots = HashSet::new();
  spots.insert(Dog);
  generic_speak(whiskers.into_iter());
  generic_speak(spots.into_iter());
}

Notice we're taking two totally different collection types, a Vec<Cat> and HashSet<Dog>.

Re-running our example gives what we'd expect:

$ cargo run --quiet
It says: meow
It says: meow
It says: woof

And cargo-bloat now shows:

$ cargo bloat -n 999
Analyzing target/debug/blog_demo

[ .. snip .. ]
0.0%   0.2%     440B  blog_demo generic_speak
0.0%   0.1%     415B  blog_demo generic_speak
[ .. snip .. ]

A total of 855 bytes.

Since you already know the drill, we can do the inner function dispatch thing (with a caveat listed below):

fn generic_speak<T: Speak>(param: impl Iterator<Item = T>) {
  fn generic_speak_strings(params: Vec<String>) {
    for item in params {
      println!("It says: {item}");
    }
  }
  generic_speak_strings(param.map(|s| s.speak()).collect());
}

To which cargo-bloat reports:

$ cargo bloat -n 999
nalyzing target/debug/blog_demo

[ .. snip .. ]
0.0%   0.1%     405B  blog_demo generic_speak::generic_speak_strings
0.0%   0.0%      80B  blog_demo generic_speak
0.0%   0.0%      69B  blog_demo generic_speak
0.0%   0.0%      65B  blog_demo generic_speak::{{closure}}
0.0%   0.0%      65B  blog_demo generic_speak::{{closure}}
[ .. snip .. ]

Even though we have these closures, it's still a total of 684 bytes instead of the original total of 855 bytes. Again, in this contrived example it's not that dramatic, but in the real world it is often quite dramatic as the original generic_speak or later generic_speak_strings could have quite a lot of code!

Warning
The Caveat! Conversion performance and allocations

I mentioned there is a caveat to the above. We created a whole new Vec<String> to pass to the inner function which is another allocation and has it's own performance implications. However, perhaps this is a case were an extra allocation like that is acceptable compared to the code bloat and compile times. You'll have to be the judge in your specific case.

This is also partially only due to the strange contrived API I used for this example as well!

Conclusion

We learned that Rust will duplicate generic functions for all concrete types that use said function, which can cause a decent bit of code duplication and increase compile times.

By turning our generic function into a small wrapping shim, that immediately dispatches to an internal non-generic function only the shim gets duplicated while all our "real code" stays as a single logical function.

Using navi for CLI Cheats

Kevin K. — Sun, 27 Nov 2022 03:00:27 +0000

Diving into navi as an interactive command builder, and using strace as a demo for concise cheat sheet!

In the previous post I spent a short time detailing steps I take to make my command line experience more pleasant and efficient. One of the steps I listed was using the tool navi to quickly recall
commands.

This is incredibly helpful for commands that I don't use often, or for variations of commands with specific options that I use less frequently. I find fuzzy searching through my navi cheats is 100x faster than grepping through
Manpages or --help output. Now, of course there are times where my navi cheats don't cover something I need and I must fall back to older methods, but then I just update my cheats and go about my day.

I recently found a new (to me) way to use navi, as a "command builder" of sorts. It makes sense to use on commands where the flags/options are additive, and not individual commands (which is somewhat of a pet peeve of mine...)

¯\(°_o)/¯

I should make a post on CLI UX about why certain CLI patterns are sub-optimal...

Enter `strace`

A good example is strace, which allows one to display system calls made by particular commands and processes. Several other blog posts have recently mentioned strace, and it's a tool that has come in hand for me more than a
couple times so in it's honor it'll be the demo today.

We won't be adding all the options that strace has, but the ones I find most common to my use cases, which probably covers a pretty vast swath of people as well.

Our goal is to build a navi cheat sheet that walks us through building an strace command with all the options we want to expose.

This post won't cover all the details of strace or even what all the options mean, although most of the ones we're adding are pretty self explanatory.

(ᕗ ͠° ਊ ͠° )ᕗ

Like a good CLI should be!

For a pretty good summary, see "strace The Sysadmins
Microscope".

`navi` Overview

As mentioned in my previous post, I bind navi to <CTRL>-n in my terminal, using the following ZSH function in my ~/.zshrc:



_call_navi() {
  local selected
  if selected="$(printf "$(navi --print --path ${HOME}/Projects/navi-cheats/cheats </dev/tty)")"; then
    LBUFFER="$selected"
  fi
  zle redisplay
}

zle -N _call_navi
bindkey '^n' _call_navi

The post important parts are the --path argument, which tells navi to look in a specific location for cheat sheets (I prefer to use my own directories, instead of the built in ones and "adding repos" feature). You can add multiple
directories separated via : (and technically I have a few directories, including a $work one that I pulled out of the example above).

The path listed above is the cheats/ directory of my public repository. It's just a big list of .cheat files. More on that soon.

Also the --print flag tells navi to simply print the final command, instead of running it. This is useful because often I want to make some additional edits to the command that either my cheat sheet didn't account for, or the features simply don't exist to add the flexibility I wanted.

Normally, --print would just print the command to stdout, however with ZSH you can have the current command line replaced by the output of a command, which is what is happening the remainder of the function.

What is a `.cheat` file?

navi looks for files ending in .cheat with a particular structure. I won't re-hash the entire tutorial from the navi repository since it's quite good and comprehensive, but the TL;DR is:

.cheat files can be split up and named however you like, as the file names aren't used by navi. I tend to use either command names or situations as file names.
Lines starting with ; are ignored (comments)
Lines starting with % are categories and comma separated key words. These make good "tags" when fuzzy matching. I tend to use command names, or variations of the names to assist with matching.
Command descriptions are lines starting with # and used as a short description of a command
Commands templates are on the line below a description with no leading characters.
- Commands templates can be multi-line with \ like regular shell commands
- KEY Commands templates can have "variables" that get expanded later if the command is selected. Variables are words surrounded by angle brackets (i.e. <foo>)
"variable expansions" are optional, but are lines that start with $ followed by the variable name, then : followed by the command to expand the variable (i.e. the command to present options to user for selection). So our <foo> variable could have an expansion $ foo: echo -e "true\nfalse" which would present the two options true and false to the user using fzf, then whatever the user selected would replace the <foo> in the command template.
If no variable expansion line is provided, an fzf text box is presented to the user allowing them to type whatever they want, which will replace the variable in the template.
Files can have as many "categories" as you like
Multiple command templates can share a single expansion. I.e. think of of a bunch of firewall commands, all of which need to allow a user to select a port number. All firewall template command could list a <port> variable, and the file would contain a single $ port: ... expansion that applies to all templates.
Beyond functioning as tags, categories separate out variable expansions. So variables expansions only apply to command templates within the same category.

So a typical .cheat file follows this kind of a pattern:




% foo, bar

# Description of command
command -f -b <qux>

# Description of other command
other_command -b -z <bar> <foo>

$ bar: ...
$ foo: ...

Again, a single .cheat file can have multiple % categories, splitting up the expansions, to allow you to re-use variable names with different concrete expansions. Although, I normally only have a single % per file.

Variable Expansions

This is where the true power of navi comes in to play. By default, navi presents a blank fzf text box for variables without an expansion line. Such as the <qux> variable above.

So let's first build out our first strace .cheat file:



% strace

# Display system calls for a command
strace <CMD>

Running navi, and typing something in the fuzzy match area to select our strace command then brings up a new screen asking us to fill in the CMD variable (since we did not provide an expansion). If we type firefox, navi drops us back to the terminal with the following command ready to run:



$ strace firefox

Static Expansions

We can present a limited set of options for the user (which by default also allows them to type something outside of our presented options).

So if we instead add the following line below our command template, we'll get a list of two options.



$ CMD: echo -e 'firefox\ngoogle-chrome'

NOTE:
the user can still type something outside of our two options, although we can prevent this behavior which we'll see in a minute.

Since fzf by default delimits options via a newline \n, we can actually make the expansion a little more readable using spaces, since the values themselves don't have any spaces.



$ CMD: echo 'firefox google-chrome' | tr ' ' '\n'

This is functionally equivalent to what we had before, but some prefer it. It also points out that expansions can use standard pipes |.

Let's add a new command (with description) to our file:



% strace

# Display system calls for a command
strace <CMD>

; 👇 is new
# Display system calls for a single PID
strace -p <PID>

$ CMD: echo 'firefox google-chrome' | tr ' ' '\n'

Now we can type out a PID we wish to attach strace to.

But there is a problem; since we didn't provide an expansion, it's up to us to type in a valid PID. For example, typing foo just blindly makes our command line:



$ strace -p foo

Which will error when run.

Using a static expansion can help with this somewhat:

NOTE: on 64bit systems PID_MAX_LIMIT is around 4 million...but for simplicity and demo sake we'll make it the 32bit limit since we'll be changing it shortly anyways.



$ PID: echo {0..32768} | tr ' ' '\n'

Which is better, we'll get a huge list of numbers (that we can fuzzy match through). And we'll be dropped on the command line with a PID:



$ strace -p 28267

But we still have the problem of someone can just type foo and again we're in invalid territory.

navi provides a delimiter we can use on the variable expansions --- to provide additional context navi either about the selection itself, or presentation of the options. One such option is --prevent-extra which limits the user to list of options you provided, and nothing more.

NOTE:
--prevent-extra is listed as experimental. And when user provides something outside the presented list, navi exits with an error. I'm on the fence about if that's better or not. But it's an option no less.

Changing our expansion to the following solves the invalid issue we had before:



;                                    👇 is new
$ PID: echo {0..32768} | tr ' ' '\n' --- --prevent-extra

Dynamic Expansions

That crazy list of PIDs with no context isn't a ton of help either. Luckily navi can come up with options dynamically!

So instead of just providing 32,000 numbers in sequence, with no process names, or even knowing if its a valid PID of a current process we can just ask navi to look at the currently running processes via standard shell commands and
present those to the user.

But we have a slight problem, something like ps -aux prints out far more information than we need. We need a PID and a name, more or less.

Since navi expansions are just shell pipes, we can use standard utilities like grep and awk to fix that!



$ PID: ps -aux | grep -v PID | awk '{print $2"    "$11}' --- --prevent-extra

We leave the --prevent-extra because we're still limited on if the PID isn't present in this list, well its not valid.

Now we get a nice fuzzy match-able list of PIDs and the associated processes!

However, running this shows one more issue...we get the whole selection on our command line:



$ strace -p 3860    /usr/bin/firefox

NOTE:
the extra spaces in there come from our expansion option too!

navi options to the rescue again! There is a --column option which allows us to use only a single column of output as the final selection!

Changing our expansion to:



;                                                            👇 is new
$ PID: ps -aux | grep -v PID | awk '{print $2"    "$11}' --- --column 1 --prevent-extra

Gives us the correct output.



$ strace -p 3860

Much better!

Additive Options

Ok, so we can now build an strace command, fuzzy match through a valid list of processes running on our computer, but strace provides all kinds of additional options for us. For example, what we want to suppress attach/detach messages (i.e. -q), or it's variations (-qq or -qqq)?

Turns out, we can use expansions for that too!

Changing our command template slightly, to add a placeholder for the -q|-qq|-qqq:



# Display system calls for a single PID
strace <QUIET> -p <PID>

We can then add another expansion this variable:



$ QUIET: echo -e '\t\tNo suppression;-q\t\tSuppress attach/detach messages;-qq\t\tAlso suppress exit statuses;-qqq\t\tSuppress every suppressible message' | tr ';' '\n' --- --column 1

We're just building off what we already know. Splitting options into two columns with the flag we want to add on the left, and description on the right, separating all options with ; which we can replace with a newline via tr,
then telling navi to only use the first column as the actual selected output for the expansion.

Now we get a new screen asking us if we want to suppress messages or not, with the associated option on the left!

PRO TIP: It's a good idea to place the "none" or "no" option that represents no additional flags first, as the user can then just fly through pressing <enter> when they don't wish to add anything additional. This greatly helps when there are a bunch of optional flags.

One final nicety I like to add is the --header option, which allows one to give a short text describing what the user is actually selecting. This comes in handy sometimes when the choices aren't super explanatory.

So our final expansion version becomes:



$ QUIET: echo -e '\t\tNo suppression;-q\t\tSuppress attach/detach messages;-qq\t\tAlso suppress exit statuses;-qqq\t\tSuppress every suppressible message' | tr ';' '\n' --- --column 1 --header "Suppress any output?"

This shows a text message above the input box with description of what navi is
asking.

We can continue fleshing out the options repeating the above steps for the flags we want to present.

Here is more complete set of expansions:



# Display system calls for a command
strace <SUMMARY> <QUIET> <FOLLOW_FORKS> <SYSCALL_TIMES> <SUCCESS_FAILED> <DECODE_FDS> <CMD>
;      👆 these variables are new
# Display system calls for a single PID
strace <SUMMARY> <QUIET> <FOLLOW_FORKS> <SYSCALL_TIMES> <SUCCESS_FAILED> <DECODE_FDS> -p <PID>
;      👆 these variables are new

$ PID: ps aux | grep -v 'PID' | awk '{print $2"    "$11}' --- --column 1 --header "PID to attach to?"
$ QUIET: echo -e '\t\tNo suppression;-q\t\tSuppress attach/detach messages;-qq\t\tAlso suppress exit statuses;-qqq\t\tSuppress every suppressible message' | tr ';' '\n' --- --column 1 --header "Suppress any output?"

; 👇 These expansions are new
$ TIME_STAMPS: echo -e "-t\t\tabsolute;-tt\t\tabsolute usecs;-ttt\t\tabsolute UNIX time" | tr ';' '\n' --- --column 1 --header "Print Timestamps?"
$ SUMMARY: echo -e "\t\tnone;-c\t\tsummary;-C\t\tsummary only" | tr ';' '\n' --- --column 1 --header "Print Summary?"
$ FOLLOW_FORKS: echo -e "\t\tno;-f\t\tfollow forks;-ff\t\tfollow forks and split output" | tr ';' '\n' --- --column 1 --header "Follow Forks?"
$ SYSCALL_TIMES: echo -e "\t\tno;-T\t\ttime spent in syscalls" | tr ';' '\n' --- --column 1 --header "Display time spent in syscalls?"
$ SUCCESS_FAILED: echo -e "\t\tdon't filter;-z\t\tsuccess only;-Z\t\tfailed only" | tr ';' '\n' --- --column 1 --header "Filter success or failed exits?"
$ DECODE_FDS: echo -e "\t\tno;-yy\t\tdecode all fds" | tr ';' '\n' --- --column 1 --header "Decode File Descriptors?"

Now that we'll have more expansions, you'll notice that in the upper section of the navi window, there is a preview of what expansions have been expanded to thus var. This comes in handy at times.

Limitations

This brings us to one of the limitations of navi (thus far), in that it's difficult to see if a previous variable has been expanded into something or another. There is variable dependencies but those don't quite do what I want.

A good example is the next (and probably most important) option for strace and that is the filtering.

For example, so see all syscalls related to processes (such as execve(2) for when a process is created), you could use:



$ strace -e trace=process ...

The -e option takes expressions in the form of option=value. There are quite a few predefined options, as well as values, groups, etc.

The way navi works now, you can't for example ask, "Do you want to add an expression" and insert -e foo=bar or nothing if they answer no. So we end up with two different command templates, one with expressions and one without.

Additionally, navi is limited to a single expression, you can't for example loop and provide multiple expressions. navi does provide a --multi option which allows FZF to pick multiple options from the list (i.e. imagine multiple
firewall ports), but not multiple pairs in the syntax strace expects. This is normally fine, as you can always add additional expressions after navi is finished, and many times a single expression is all you want anyways.

To get a better sense of this, we'll add two more command templates, adding expressions, and navi provided lists for the options and values pairs:



# Display system calls for a command (with a filter)
strace -e <EXPR_OPT>=<EXPR_SET> <SUMMARY> <QUIET> <FOLLOW_FORKS> <SYSCALL_TIMES> <SUCCESS_FAILED> <DECODE_FDS> <CMD>
;      👆 the -e variables are new
# Display system calls for a single PID (with a filter)
strace -e <EXPR_OPT>=<EXPR_SET> <SUMMARY> <QUIET> <FOLLOW_FORKS> <SYSCALL_TIMES> <SUCCESS_FAILED> <DECODE_FDS> -p <PID>
;      👆 the -e variables are new

$ EXPR_OPT: echo 'trace abbrev verbose raw signal read write fault inject status quiet kvm decode-fds' | tr ' ' '\n' --- --header "Expression Option (or type custom)?"
$ EXPR_SET: echo 'clock creds desc file fstat fstatfs ipc llstat memory net process pure signal stat statfs all none' | tr ' ' '\n' --- --header "Expression Set (or type custom)?"

Now we can interactively build a full strace command!

We want to allow the user to type something outside our list, because strace supports a bunch more options, and regex, or ! not expressions, etc. However, navi doesn't have a full way to express those options so we're limited.

But this is still leaps and bounds better than nothing!

Conclusion

Hopefully, this has shown how having a command builder can be a huge help. Yes...it is work to make the cheat sheets, however there are plenty out there ripe for copying. navi also has built in functionality to add public git
repos as cheat repositories, and have them automatically updated. I prefer to use my own and have more control over them, but that option exists for those that want to bootstrap making their own cheats.

There is quite a bit more navi can do, but this post is long enough!

Happy cheating.

My Shell Setup

Kevin K. — Sun, 27 Nov 2022 02:49:59 +0000

Details a few steps I take to make my terminal experience a little morepleasant.

I spend most of my day in the terminal in one form or another. There are a few steps I've collected over the years to make my experience more pleasant and most importantly more efficient.

Most of the steps require some level of software to be installed or configuration to be changed, so I can't perform these steps on all production servers I work in. However, many of the steps are only applicable to my local machine, and the short period of time I'm working in a remote server is minimal, or aided by changes I made locally.

Configuration

Before I talk about the actual steps, I should mention where I store my custom configuration files. Manually entering these on each machine I wish to configure would be huge pain.

Like many people, I keep all of my configuration files stored in a public dotfiles repository on GitHub. It's a bit of a mess as I only use particular parts on particular machines. So it's not
meant to be publicly read or analyzed. However, my point is that the configuration files are stored in a central location where I can simply clone the repository down to my machine, symlink the files or directories I care about and want to use and go about my day.

I don't use anything like Ansible to set up my machine because there are too many variables that control how I use a particular machine and which steps I want to actually utilize.

Aside: I actually have a baseline
set of scripts that do all the copying/symlinking for me based on a TUI dialog I fill out. To be clear, this is in no way meant to be publicly consumed as many of the steps are hard-coded for my specific machines.

Notes on Distribution

I use Fedora on all my workstations, but I've also run variants of Ubuntu at times for work, so all the steps work on either distro. The install commands listed in this post are for Fedora. The package names may be different in Ubuntu
and I try to call those out where I'm aware of any differences.

Improvements

These steps change some way in which I interact with the terminal, whether they be standalone tools, or work-flows I use. All these steps are used to improve the speed at which I work, or to better the experience.

Most of the changes build on each-other, or other skills I've built up along the years so they may not be widely applicable unless you're in a similar boat. For example, using the plugin vi-mode in my shell combined with fzf searching
through my command history is lightning fast (especially when I need to edit a previous command).

ZSH

While the default Bash shell is fine for many cases (and ubiquitous) it definitely has it's limitations. Many people are familiar with and prefer the more modern Fish shell. However, I find Fish breaks too many of my existing scripts (well, almost all of them) as well as my muscle memory, and causes me to have to change commands when Googling around for fixes because everyone assumes Bash (or Sh at the least).

Add to that, the thing most people ooooh and ahhhhh over with Fish, at least initially, is the auto-suggestions and syntax highlighting, which my preferred shell (ZSH) can do as well as you'll see soon.

ZSH is nearly fully compatible with Bash, in fact I've had to make zero changes to any existing scripts or commands to adapt. All my Bash knowledge works in ZSH.

Aside: The one area where I've personally hit differences is when writing completion scripts for programs (i.e. the menus that pop up when you hit <tab><tab>). ZSH completion scripts are pretty different (and far more functional), but I'd also wager that 99.7% of people out there are not writing
completion scripts regularly, or at all, so this should not affect them.

Since I mentioned completion scripts in the aside, programs that support ZSH completions are so much nicer, than the Bash completion variants.

Here's an example of completions of git in Bash:

Here's an example of completions of git in ZSH:

There are a bunch of ZSH features that I love, but the few that sound the smallest yet I probably use the most are case-insensitive paths, command-fix-suggestions, and not using cd to change directories.

Those sound tiny. But they add up quickly when using the terminal constantly.

In ZSH, you can type cat foo, when the file is really Foo and ZSH will auto-correct the path for you.

If you typo the command, and it's close enough to a real command, or file/directory ZSH will also suggest a fix that you can accept or ignore. For example:

$ gti commit -m "foo"
zsh: correct 'gti' to 'git' [nyae]?

Finally, instead of typing cd ../blah/ you can just type ../blah/.

Even though there is so much more ZSH can do, let's just assume you're sold, how do we install and use it? Here's my setup:

$ sudo dnf install zsh
  (install ZSH)

$ sudo chsh -s $(which zsh) $USER
  (change my default shell to ZSH)

Note: chsh is no longer available by default in some distributions. In Fedora, it's included in the package util-linux-user.

Then I use the following ~/.zshrc configuration:

ENABLE_CORRECTION="true"
COMPLETION_WAITING_DOTS="true"

export PATH=/usr/local/bin:$HOME/.local/bin:$PATH
export SSH_KEY_PATH="~/.ssh/rsa_id"
export TERM="xterm-256color"
fpath+=~/.zfunc
compinit -U

Oh-My-ZSH

ZSH by itself is pretty plain, even if functional. There is a ZSH framework that
adds a ton of useful functionality called oh-my-zsh. It also supports things
like themeing your prompt and whatnot but I don't really use that part as I rely
on other prompt software.

To install, you clone the repository and add a few lines to your ~/.zshrc:

$ git clone https://github.com/ohmyzsh/ohmyzsh ~/.oh-my-zsh

Then add a few lines to the top your ZSH configuration file:

export ZSH=$HOME/.oh-my-zsh
ZSH_THEME="robbyrussell"
plugins=(git systemd)
source $ZSH/oh-my-zsh.sh

That ZSH_THEME variable controls the theme, and there are a ton. However, in the following steps I disable it in favor of other options.

The plugins is the next most important line. There are a ton of plugins, and the ones listed above just a few that apply to almost any box. In the following sections we'll add a few more. Many plugins simply add the shell completions for particular commands, while others add entirely new functionality. It's
definitely worth a look through the list of plugins to see if any catch your eye for software you use frequently.

NOTE
There is a small startup cost associated with plugins, so it's good to only add the ones you'll use frequently.

Auto-Suggestions

This is the feature that most people love from Fish. It uses your command history to suggest commands live as you type, and frankly I love it too.

To add this functionality to ZSH you clone a repo, and add a plugin.

$ git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions

Then add zsh-autosuggestions to your plugins array in your ~/.zshrc

Syntax Highlighting

Another feature people love from Fish. This one colors the command you're typing green if the command is found and valid, or red if not. This again sounds so small, but can be such a huge help when typing commands you don't use often or
when copy/pasting from other medium. It can even catch typos before running the command. I love it.

Again, a simple clone and add a plugin.

$ git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting

Then add zsh-syntax-highlighting to the plugins array.

Abbrev-Alias

This plugin expands aliases automatically. For example, I have an alias of gl='git log --graph --all --oneline --decorate'. Using this plugin, when I type gl followed by <space> (assuming I want to add on to the command) or
<enter> to run the command the gl is replaced by the real underlying command.

Why would you want this?! Because all too often people memorize the alias, but not the command it represents. This means when you SSH into a server, or on a machine without all your custom aliases you can feel at a disadvantage. Using
abbrev-alias keeps these real commands front and center so that one at least sees them as frequently as they're run.

To install, you clone a repo and source the script in your ~/.zshrc, and use abbrev-alias instead of the normal alias command.

$ git clone https://github.com/momo-lab/zsh-abbrev-alias ~/.config/zsh-abbrev-alias/
$ echo 'source $HOME/.config/zsh-abbrev-alias/abbrev-alias.plugin.zsh' >> ~/.zshrc

And then for example, to add the gl alias I mentioned earlier

abbrev-alias gl='git log --graph --all --oneline --decorate'

Liberal use of aliases is a key aspect of quickly using the terminal. This plugin removes the biggest detriment of why some people dislike aliases!

NOTE:
The only time I still use normal aliases is if I'm replacing a command with another, i.e. alias ls=exa and the flags/options on the two commands are compatible (or nearly so).

Sudo

Another tiny quality of life improvement is the ability to quickly add sudo to the beginning of any command. If I'm half-way through typing a long command and realize I forgot to add sudo, in Bash I may have to erase the command and
start over, or move the cursor to the beginning of the line to type sudo and back to the end, or if the command is safe to run as an unprivileged user just run it and then quickly run sudo !! to replay the command. However, all
of those are not great and can be slow.

This plugin allows me to simply type <esc><esc> which adds sudo and keeps my cursor wherever I had it. As a bonus, it doesn't even negatively affect vi-mode (which we'll see next!) which also uses <esc>.

To install simply add sudo to the plugins array as it's built in to oh-my-zsh.

Vi Mode

This plugin is only useful if you're used to Vi(m) keybindings. I use it constantly. When combined with starship below, this even better because you get a visual indicator on your prompt if you're in Normal mode or Insert mode.

To install, simply add vi-mode to your plugins array as it's built in to oh-my-zsh.

Custom Prompts

Custom prompts allow you to visually determine facts about you're environment without having to run additional commands. Most of the time, all these bits of information you can find via manual commands, but to efficiently use the terminal we try to minimize the manual commands we have to use.

Some people prefer this information in other locations like a tmux status line. However, I prefer to have most of the information about my ephemeral environment in my shell prompt.

'Ephemeral environment' means it can change from command to command, or directory to directory. Non-ephemeral information is items like machine resource information, tickers, clocks, etc. Non-ephemeral information I prefer in places like tmux or my desktop environment (DE), not my prompt.

To combat the problem where long paths, or lots of bits of information can start to take up a large portion of the current line, I prefer prompt solutions which allow multi-line prompts where all the "status info" is on the first line, and my cursor/command is on the second line.

There are two great solutions to prompts; Starship and Powerlevel10k (based on Powerlevel9k). I find Powerlevel10k to be faster, but Starship is easier to customize, has some great features, and is easier to install. I use Starship by
default, but have switched to Powerlevel10k from time to time.

Starship

The starship prompt is run with a single binary, and configured with a single file which is one reason I really like it.

The demo site has a great GIF showing all the features and the why. As well as how to configure all the various items. I have a customized prompt, but the default is perfectly acceptable as well.

To install:

$ curl -fsSL https://starship.rs/install.sh | bash

To use, simply comment out your ZSH_THEME variable, and add this line to your ~/.zshrc (which the install script may have already added):

eval "$(starship init zsh)"

Powerlevel10k

To install we clone a repo, set our ZSH_THEME and configure a ~/.p10k file. I find it less configurable when it comes to the actual items I want displayed, but I also find it faster than starship at times since it was designed with ZSH
in mind (whereas starship is designed to be used with any shell).

Clone the project:

$ git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/themes/powerlevel10k

Then we set our ZSH_THEME="powerlevel10k/powerlevel10k".

When we open the terminal again we'll be prompted to configure a .p10k file interactively.

Tools

Now that we've set up our shell and prompt, there are a few tools I use constantly while in the terminal. Some are related to the actual shell, while others are just general use tools I find well worth the install.

Terminator, Tilix, or `tmux`

To begin, I'll mention a bit a cheat. The actual terminal emulator itself. There is one area where the terminal emulator itself can greatly improve one's efficiency.

Tiling.

Both Terminator and Tilix provide "tiling" to allow one to quickly open up another terminal to the side or below the current terminal. Terminator also allows one to define "groups" and type in one terminal, but broadcast each
command to all terminals.

This can be a huge productivity boost over disjoint terminals that require one to constantly move the mouse back and forth between windows, etc.

The alternative to this is if you're using a true tiling window manager such as i3, or tmux to handle the tiling inside a single terminal session.

NOTE:
Of course you could use tmux inside Tilix, inside i3...I feel like there is an Xzibit joke there.

Many people find using Terminator or Tilix to be a good introduction to tiling that provides a good mix between usability and functionality without fully committing to something like tmux or a Tiling Window Manager.

I'd don't personally use Tilix or Terminator much, as i3 handles all the tiling for me. I do, however, use tmux when SSH'ed into various servers and want to tile panes without having to re-authenticate.

FZF

The fuzzy finder. This tool is so amazing that it deserves a full post itself.

It fuzzy matches through lists and allows one to interactively select one (or multiple) items. Sounds simple.

When it comes to ZSH and the like the two features I use constantly are ctrl-r to fuzzy search through my history and ctrl-t to search through file paths.

Most package managers include the fzf package, and place the appropriate configuration files. All you have to do is install it, and source the ZSH script in your ~/.zshrc

$ sudo dnf install fzf
$ echo '[ -f ~/.fzf.zsh ] && source ~/.fzf.zsh' >> ~/.zshrc

FZF can also be used anywhere a list is produced and you want to pick an (or multiple) elements.

ctrl-r

Pressing <ctrl>-r will bring up a list of commands which you can fuzzy match through, press <ctrl>-up/down (or <ctrl>-j/k for vim keybindings) and hit <enter> on a selection to have that command pasted into your command line.

It's S-O-O-O much better than pressing up two hundred times to find the command you want in your history.

ctrl-t

This works the same as <ctrl>-r, except its a list of paths. 'Nuf said.

Navi

navi is about fuzzy search-able command cheat sheets. Yes please.

With this tool, I can search through (and create my own) command cheat sheets. It's difficult to explain, but when I press <ctrl>+n a TUI pops up (also using fzf by the way) which I can fuzzy match through. Upon selecting a command,
that command can have templates where I fill in the required information. Once complete, I'm dropped back to a terminal with the filled out complete command. Amazing!

Here's a GIF that explains it better than words can. I'm adding a new port the firewall via firewalld:

I tend to make more generic commands, and then will tweak the invocation manually at the end (which with vi-mode is super fast).

Installation is done by either building from source, or using bash:

$ bash <(curl -sL https://raw.githubusercontent.com/denisidoro/navi/master/scripts/install)

Making your own cheat sheets is beyond the scope of this post, but highly worth it. It's also possible to use other's repositories and have them auto updated. You can find my own sheets in a GitHub repository

By default, navi can be run from the command line normally. Setting up the <ctrl>-n keybinding can be accomplished by eval'ing the built-in widget command in your ~/.zshrc:

eval "$(navi widget zsh)"

By default the keybinding is <ctrl>-g. I prefer to just run navi widget zsh and paste the output into my ~/.zshrc and change the keybinding from g to n, and also point it to my own cheat repositories locally.

Conclusion

That's all for now. I could write more (and perhaps will later) about some of the tools I use regularly in the command line, such as exa, Ripgrep, fd-find, or bat. There's also the potential to write about the aesthetic changes I make
to improve my enjoyment of the terminal, such as font selection or which terminal emulator I actually use.

Network Coordinates

Kevin K. — Sun, 27 Nov 2022 00:22:30 +0000

Intro

Imagine you have a distributed system and need to answer something like, "Which node is closest to Node X?" or perhaps "Which N nodes are closest to Node Y?"

It turns out answering this question naively is wildly wasteful.

We'll dig in to a popular Network Coordinates system and how to use it correctly below.

As an example of the kinds of differences we're talking about, a naive solution to this question for a system of 1,000 nodes (where a "node" could be a host, a VM, or even a container) could require between 1GB to 240GB+ network bandwidth per day depending on how accurate (or the frequency of) one would like the results to be. Whereas the solution we'll be discussing here would require 272KB to 65MB for the same "accuracy" levels.

This more efficient system is known as Network Coordinates. But these coordinates also need to be used properly or you effectively have a fancy version of the naive solution.

This problem has come up rather frequently for me, and it's been surprising to me how often the naive solution gets picked.

When initially presented with this problem many people I've talked to decide to simply conduct a few latency checks (such as a ping or ICMP message) and store a map of the results at each node. Done. Move on the next problem.

We've already shown how wasteful this could be, but let me expand on it just to prove my case.

Lets Math

To demonstrate how this naive solution adds up, let's do some quick back-of-napkin-math for those 1,000 nodes. First I'll mention 1,000 nodes is a decent sized network yet still entirely possible. If we were to say a "node" could be a container then the 1,000 number becomes not only feasible, but entirely likely even for small to medium distributed systems.

Network Requirements

A ping (ICMP message) is an 8 byte header plus some payload size. On Linux the packet size is 64 bytes by default, although we also need to include the IP and Ethernet header size as well which adds another 40 bytes (assuming IPv4).
Keep in mind many real world scenarios use additional encapsulation such as tunnels or application layer messages that add more overhead. But let's assume the simple scenario of ~100 bytes per message, or 200 bytes network traffic round trip.

At this rate, a single node conducting a latency check against all other 999 nodes would require about 200KB network traffic round trip. This doesn't sound too bad yet, is probably why this solution gets picked so often.

However, all other nodes still need to do the same check.

Lets further assume symmetrical results, meaning if A does a check with B then B doesn't need to do a check with A and just reuse A's result.

Note
Symmetrical results is close, but not actually that realistic for real world
network latencies as packets can take different routes.

With this change our total isn't 2KB * 1,000^2 but instead 2KB * (1,000 + 999 + 998 ... 1) which is slightly better. Although, even this equals just under 1GB of traffic round trip for a single check.

Because we're talking network latencies which are very unstable and subject to change over time as well as jitter a single latency check isn't very representative of actual network latencies.

It's common to do several latency checks and take an average, say five checks (on the command line something like ping -c5). And then one would repeat this test over time, such as every 30 minutes.

At this rate (five ICMP messages every 30 minutes), we're in the range of 5GB of network traffic every 30 minutes, or 240GB a day just to be able to answer the question, "Which nodes are the closest."

Storage Requirements

There's also storage considerations as well, which are admittedly less concerning, however let's briefly address it so we can compare to the Network Coordinate solution later.

If we assume the "results" of this latency check are an integer, say four bytes, representing the average number of milliseconds round trip latency and a public IPv4 address as the node ID (another 4 bytes) it's around 8KB to store the entire map. This map is duplicated per node so around 8MB in total storage.

Granted, a 4 byte result and 4 byte ID is about as small as it could be and real world requirements are probably much, much higher doing things such as keeping a history of results, or at least a sliding window, and using longer
node IDs.

Network Coordinates

So how would these Network Coordinates work, and why are they so much better?

Vivaldi

The system we're discussing here is based on the Vivaldi Network Coordinates (PDF). Network coordinates function much like physical coordinates on a map. Two locations have some coordinate that represents their location.

In physical space it's relatively simple to take two coordinates and determine how "close" the points are. Or back to our original problem, trivial to take a list of coordinates and determine which of those are the closest to any other given coordinate.

Network coordinates are the same, except that they're trying to model a "location" not in physical space but within the network. Unlike physical space, the distance between two "locations" can fluctuate in network space because packets can take different routes, links could go down or become degraded, and we must account for jitter and spikes.

Vivaldi takes the approach of giving each node a random euclidean coordinate using an N-dimensional coordinate. In physical space we may use a 2D or 3D coordinate, but in Vivaldi you're not limited physical dimensions. In fact,
some research has gone in to picking an optimal number of dimensions for these coordinates. The more dimensions the more network traffic and storage requirements, but theoretically the more accurate as well.

That "optimal" number appears to be around 8D, although in my anecdotal testing even 6D or 4D coordinates seem to be accurate enough for many scenarios.

Let me briefly describe how Vivaldi works at a high level and what the code is doing under the hood.

How Does it Work?

I said that each node starts with a randomly assigned coordinate. Each node will then conduct round trip latency checks and adjust their coordinate based off observed latency. The goal is for each node to adjust (or move) to a coordinate that accurately reflects it's position.

Note
"Each node conducts latency checks..." This sounds just as bad our naive
solution! It's not I promise, this will become clear shortly.

The way I think about this is for example in a 2D map, with two nodes, the goal is for each node to enter any coordinate where the calculated latency between the two nodes matches (or is within a margin of error) of the observed latency.

Let's assume nodes A and B are physically 5ms round trip latency from one another. We start by giving them random coordinates (remember we're using 2D just to make this easy to visualize). I've added a dashed circle to represent any coordinate that give a correct distance estimate when comparing the two coordinates.

NOTE: I've deliberately left out the actual coordinate numbers (i.e. (2,4), etc.) because the coordinates themselves are irrelevant and meaningless. The only meaning comes from comparing two sets within the same system.

Now A and B do a latency check, observe real world latency and adjust their coordinates.

Node B then does the same thing

At this point the coordinates have been updated enough that a distance estimate between them is accurate with observed real world latency.

Like I said in the note above, this doesn't look much better than the naive solution. And with only two nodes it's not.

If we were to add a third node, let's see what happens. We'll repeat the above, except let's assume this new Node C is also doing checks with A, but has a real world latency of 10ms to A.

Notice all nodes are now accurate with observed latency.

But wait! What is the distance between B and C? For the sake of this post, and my diagramming skill, lets assume it's also 10ms.

But then that means the coordinates aren't yet accurate! We need to do some more checking!

In the diagram a single check made everything accurate again. However, a real world update to C could have well made it accurate for B but now A is not accurate so it needs to adjust, etc. etc.

NOTE: Additionally, the direction of movement is calculated from the position of the remote, which the diagram doesn't accurately reflect because it would have made the sequence more confusing as stabilization can take many rounds.

Error Estimates

Additionally, Vivaldi gives a notion of an "error estimate" or how sure a node is that it's coordinates are accurate. At first, because the coordinates are random the error rate will be very high.

When doing coordinate updates, if the remote node's error rate is high (it's not sure of it's accuracy) and the local rate is low (you're very sure of your own accuracy) you won't move much. But the inverse (you're not sure at all of your accuracy, but the remote is very sure) will result in your local coordinate updating a lot.

This allows the coordinates to converge quickly on known good points, but then slowly once the results become more accurate.

Performance

At this point it's looking like all nodes just need to do actual latency checks with all other nodes to have any shot at being accurate. This seems just as bad, if not worse than our naive solution! As we're doing latency checks anyways, why not just use those real latencies!

This is what I alluded to at the beginning of the post about having a fancy version of the naive solution. Because all nodes start with random coordinates they need to conduct updates with more "stable" nodes (low error estimates) to
converge on accurate results. Additionally, in large networks if the coordinate updates aren't conducted against shared stable nodes, you end up with "islands."

For example, assume a network of six nodes (A..F) where A and B are stable accurate coordinates. If C and D only update against A and E and F only update against B you'll end up with two islands of coordinates.

This can happen accidentally by relying on "random updates" where all nodes start random, and conduct updates against other nodes randomly or naturally as a side channel of regular communication. Some nodes will stabilize, while other
nodes will not.

To make matters worse, even if one of the nodes happens to conduct an observed check against another stable island (say C against B in our example, or even C against F once it's stabilized), that does not actually help D, or E at all unless they also start updating against other stable islands/nodes.

As it turns out, the solution to this problem and the solution to orders of magnitude better performance is the same. Use a single Origin Node, which all nodes conduct their observed latencies checks with. For example, if A became the only origin node.

This does not preclude a node from checking against another node, but it's not required.

Lets Math Again

We'll assume 8D coordinates, which is effectively a [f64; 8] in Rust plus a few f64's of metadata like error estimates and jitter smoothing, in total 88 bytes per Vivaldi message as a payload, and we're free to use whichever protocol best serves our application, say UDP which like ICMP is 8 bytes. We also have our original overhead of 40 bytes for the IPv4 and Ethernet header.

We add an origin node, so 1,001 nodes, and a single coordinate update for all nodes is about 272KB of traffic. Let's be fair and say we did five updates per node, so total network traffic for all nodes and those five updates is 1.36MB. Even if we did this every 30 minutes, that's only 65.28MB per day (versus our 240GB+ in the naive solution).

Real Power

However, it's not just more efficient by network traffic standards. The real power comes from any two nodes being able to estimate accurately their latency without ever having observed any latency check between one-another!

In the naive solution the only way to know what the latency between E and C was, was to check it (over and over and over). However, in this new system E and C need not ever have sent a single packet to each-other! We can just look
at their two coordinates and perform a quick calculation.

If the Origin stores all current coordinates (or even gossips them around to various local caches) we can just ask, "what are the N closest nodes to coordinate X?" at which point the origin can just zip through it's list of coordinates doing these calculations and give back a sorted list if desired!

But how fast are these calculations?

Speed

In my Rust Vivaldi implementation (called violin), a single calculation of two 8D coordinates using stack memory on my 8 core AMD Ryzen 7 5850U laptop with 16GB RAM takes ~16.5ns. A benchmark of 1,000,000 calculations takes 16.5ms.

Summary

I hope this post has given you some insight into how to efficiently get answers to questions like "network closeness." Using a Vivaldi based solution (correctly!) can vastly improve distributed systems performance.

If you're into Rust, check out violin! As a quick sample, this is what it looks like to use:

use std::time::Duration;
use violin::{heapless::VecD, Coord, Node};

// Create two nodes and an "origin" coordinate, all using a 4-Dimensional
// coordinate. `VecD` is a dimensional vector.
let origin = Coord::<VecD<4>>::rand();
let mut a = Node::<VecD<4>>::rand();
let mut b = Node::<VecD<4>>::rand();

// **conduct some latency measurement from a to origin**
// let's assume we observed a value of `0.2` seconds...
//
// **conduct some latency measurement from b to origin**
// let's assume we observed a value of `0.03` seconds...

a.update(Duration::from_secs_f64(0.2), &origin);
b.update(Duration::from_secs_f64(0.03), &origin);

// Estimate from a to b even though we never measured them directly
println!("a's estimate to b: {:.2}ms", a.distance_to(&b.coordinate()).as_millis());

As a bonus violin also works in no_std and no alloc environments (but with some performance penalties due to lack of platform intrinsics).

CLI Shell Completions in Rust

Kevin K. — Fri, 08 Jan 2021 19:09:35 +0000

In this post we add shell completions to the XKCD CLI utility we made earlier. We'll show how easy it is to support multiple shells, and generate the completions both at compile time and/or run time.

Why Completion Scripts

Perhaps my biggest daily, "Ugh.." moment is working on the command line and typing <TAB><TAB> and getting no shell completion help. Even when I'm using a tool that I'm extremely familiar with, I still end up using shell completions (also called tab completions) to discover new options or explore the flags further.

Sometimes I simply forgot the exact syntax..."is it --remove or --delete...Oh right, on this tool it's -rm"

-- (눈_눈)

I think we can agree shell completions are useful. So why don't all utilities provide shell completions? Probably because they're a giant pain to create, they must be kept in sync with the actual CLI flags/options, and there are a ton of shells to support!

Imagine typing foo --<tab><tab> and seeing --directory was a valid option, only to hit <return> and have the CLI tell you, error: --directory option not found or similar. Turns out the CLI updated and is now using --dir. Out of
sync completions are terrible.

`clap` to the rescue!

I've got great news. Adding/supporting shell completions with your Rust based CLI is very easy! Using the clap crate, we can generate shell completion scripts either at compile time, or provide a
special option to allow users to generate shell completion scripts at run time on the fly.

Or both.

-- (¬,‿,¬)

Using generated completion scripts we get the following benefits:

Our completion scripts will always be in sync with our actual CLI options
We can support Bash, Zsh, Fish, PowerShell, and Elvish using about five lines of code
Using both compile time and run time gives our users options for how/where to use these scripts

Whats so hard?

Before I said that creating shell completion scripts was hard, but how hard? A popular command line tool ripgrep which has a moderately large CLI space (not huge, but not trivial either) has a Bash completion script that is 213 lines lone (as of v12.1.1)! That's just Bash. Other shells have similarly sized scripts. rustup which has a fairly large CLI space has a 1,110 line Bash script.

Enough preamble, get to the code!

Overview

We'll be updating our grab-xkcd program from a previous
post with the new code, but in order to clearly demonstrate the different methods and not conflict with the original article I will use two new branches
completions-ct for compile time completions, and
completions-rt for run time completions. If the reader is feeling froggy, you can combine the two!

Let's start with compile time.

Compile Time Completions

I tend prefer compile time completions because those completions scripts can be checked into version control, or even tweaked further from the auto-generated script. Originally, the auto-generated scripts were meant to be a starting point, but they turned out to work so well that almost no one takes the time to actually tweak them further.

The downsides to the compile time method are that it requires two new "build dependencies" (meaning compile time), and can slow down compile times. Generally, this isn't an issue in practice as generating completion scripts is crazy fast. But, if you're very cautious with build times, or are trying to
avoid compile time dependencies for some reason, the run time method may be better.

Also, these scripts will still need to be consumed/installed by the user somehow. Normally this is done in a packaging format (deb, rpm, msi, exe, dmg, etc.). But if the CLI tool you're building does not have any packaging, the user will need to install these scripts manually based on the shell they're using.

Build Scripts

The process of telling clap to generate a completion script at compile time happens in a cargo build
script.

The way completion scripts are generated changed between clap v2 and the upcoming v3. In v3 (which we used in the previous article) moved the completion generating code to a separate crate to avoid code bloat. Since we're still using v3 we need to add that extra crate.

Build Dependencies

We also need to tell cargo that clap_generate (and also clap) is a build dependency. clap proper will also remain a regular dependency.

$ cargo add clap clap_generate --allow-prerelease --build
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding clap_generate v3.0.0-beta.2 to build-dependencies
      Adding clap v3.0.0-beta.2 to build-dependencies

If we were to look at our Cargo.toml we'd now see two dependencies tables, with the new one being [build-dependencies] listing the two crates above.
[[dependencies]] is unchanged, and still includes just normal clap since that's still required to parse run time arguments.

Next, since clap_generate needs a
clap::App instance in order to be able to walk our CLI and generate everything, we'll need to use
the into_app() method that was #[derive]d on our Args struct in the previous article.

Stubbing

Let's first, stub out the build.rs file in our project root directory:

// in build.rs

use clap::Clap;

include!("src/cli.rs");

fn main() {
    let mut app = Args::into_app();

    todo!("generate the completion scripts!");
}

We built grab-xkcd as a binary, and not a library. So we need some way to include our Args struct in this build script. We could have re-factored out the core logic into a library, and then consumed that library in our main.rs
binary...but that's overkill for this example. So we simply include!() the source file directly, which is like a copy/paste.

Generate Bash

Now that we have an App struct, we can pass that to clap_generate and see the magic!

Let's add a Bash completion script:

// in build.rs
use clap_generate::{generators::Bash, generate};
use clap::Clap;

include!("src/cli.rs");

fn main() {
    let mut app = Args::into_app();
    app.set_bin_name("grab-xkcd");

    let outdir = env!("CARGO_MANIFEST_DIR");
    generate_to::<Bash, _, _>(&mut app, "grab-xkcd", outdir);
}

There is a paper-cut that we have to call set_bin_name() due to some complexities about how the derive magic works, and how it interacts with what generate_to expects. The short version is generate_to is written to handle a large swath of CLI types, some of which may have different binary names, than program names (i.e. ripgrep-rg). Completion scripts are normally based off the binary name (which is how shells find which scripts to call). The derive magic in clap has no way of knowing the binary name, as our binary is certainly not called Args. We could have set the binary name as part of our Args declaration, which is probably the right way to do it in the long run, but for this demo it's overkill. So we just set it manually on the instance of the App struct and move on.

Walking through the rest of the above:

We import a single Generator, Bash which will walk our App struct and generate all options.
We get the value of cargo's environmental variable CARGO_MANIFEST_DIR which is our source root and where we will be saving the script to.
In generate_to we must provide some generic arguments. The first is the most important, and is the Generator that will be used to make the completion script. The other two are related to the other two final function arguments and just convenience conversion traits. Rust can infer them, hence the _.
The arguments are:
- The App struct to walk over
- The binary name to use throughout the script (which may differ from the real binary name in some special circumstances)
- A path directory to save the generated file to

That's it! If we look in our project root, you should see a grab-xkcd.bash! Even our super small CLI has generated a 65 line script:

$ wc -l grab-xkcd.bash
65 grab-xkcd.bash

Looking at the head you can see some of the boilerplate already:

$ head grab-xkcd.bash
_grab-xkcd() {
    local i cur prev opts cmds
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    cmd=""
    opts=""

    for i in ${COMP_WORDS[@]}
    do

...

Testing

So let's try it out! We can test it by just sourceing it, if you're using bash as your shell:

It works!

Project Cleanup

If we wanted to add additional shells it's just more calls to generate_to. We'll actually, place all the scripts in a new completions/ dir in our project root, so as not to get unwieldy.

$ mkdir completions

And we update our build script:

// in build.rs
// Same as before...
use clap_generate::{generators::*, generate_to};

fn main() {
    let outdir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("completions/");
    generate_to::<Bash, _, _>(&mut app, "grab-xkcd", &outdir);
    generate_to::<Fish, _, _>(&mut app, "grab-xkcd", &outdir);
    generate_to::<Zsh, _, _>(&mut app, "grab-xkcd", &outdir);
    generate_to::<PowerShell, _, _>(&mut app, "grab-xkcd", &outdir);
    generate_to::<Elvish, _, _>(&mut app, "grab-xkcd", &outdir);
}

We had to create a new Path and join it to our new directory, but that's not too bad. We also had to take a reference to outdir in each call to generate_to since we're not passing a PathBuf and not a &'static str any
longer. But hey, notice we didn't need to do anything else special and generate_to could handle those two totally different types with ease (thanks to the second _ generic parameter which is actually a T: Into<OsString> for those that care).

If we look in our completions/ dir, yup, we see a bunch of new completion scripts!

Run time

Ok, so compile time was pretty easy. Maybe a little fuss around adding build-dependencies and making a build script, but overall not too bad.

But let's say you're just making a small CLI, and don't want to have to worry about packaging or installing completions scripts with your program, etc. Luckily, we can have our binary just spit out completion scripts on demand to
stdout! Then the user can source the output if they want, or redirect it to a file if they want to "install" it.

A traditional way of doing this may have included inserting the completion scripts themselves into your binary file statically...which would probably be terrible (even though they compress well). This still runs the risk of getting
out of sync with your CLI, etc.

Let's use what we learned with the compile time completion scripts, but just do it at run time instead.

Extend the CLI

In order to do this, we'll actually need to extend our CLI a little bit to include a new option/flag for generating these completions. I normally advocate for using subcommands instead of options/flags, but since our CLI is so small and we don't have any other subcommands we'll forgo that idea. Instead we'll add a -c/--completions option which accepts any of the supported shells and spits out the completion script to stdout.

/// A utility to grab XKCD comics
#[derive(Clap)]
pub struct Args {
    // .. Same as before

    /// Generate a SHELL completion script and print to stdout
    #[clap(long, short, arg_enum, value_name="SHELL")]
    pub completions: Option<Shell>,
}

#[derive(Clap, Copy, Clone)]
pub enum Shell {
    Bash,
    Zsh,
    Fish,
    PowerShell,
    Elvish
}

This should look pretty familiar to you from the previous article, however let's step through it really quickly:

With short, long we tell clap to generate a short -c and long --completions automatically from the name of the field
arg_enum tells clap the enum we are using should be used as the only allowed value variants
value_name="SHELL" tells clap to replace the default placeholder in the --help message from --completions <completions> which is just derived from the field name, to --completions <SHELL>. It's not required, but I think it helps readability.
We create our enum with the variants we wish to support.

Using `--completions`

With that out of the way we can now look at using this new argument. In our main() function we will check if that argument was used, and if so generate the script and print to stdout and exit.

Since we'll need to know which shell to generate our completions for, we can push that logic to the enum itself and keep our main() function clean.

fn main() -> Result<()> {
    let args = cli::Args::parse();

    if let Some(shell) = args.completions {
        shell.generate();
        std::process::exit(0);
    }

    let client = client::XkcdClient::new(args);
    client.run()
}

This could be condensed into a single args.completions.map(..), but for this demo we'll leave it as is since the function is so short and concise anyways. I
also like that the std::process::exit function is visible so we know this could end execution of our program just from looking at main. If we were to reduce this to a map we'd probably also rename the enum's generate method to something like generate_and_exit to make it more clear as well. But I digress.

`clap_generate::generate`

Instead of the clap_generate::generate_to that was used in compile time, we'll be using clap_generate::generate inside our Shell::generate method. First let's add clap_generate to our normal run time dependencies:

$ cargo add clap_generate --allow-prerelease
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding clap_generate v3.0.0-beta.2 to dependencies

Now we can actually implement Shell::generate:

// in src/cli.rs
use std::io::stdout;

use clap::{Clap, IntoApp};
use clap_generate::{generators::*, generate};

impl Shell {
    fn generate(&self) {
        let mut app = Args::into_app();
        let mut fd = std::io::stdout();
        match self {
            Shell::Bash => generate::<Bash, _>(&mut app, "grab-xkcd", &mut fd),
            Shell::Zsh => generate::<Zsh, _>(&mut app, "grab-xkcd", &mut fd),
            Shell::Fish => generate::<Fish, _>(&mut app, "grab-xkcd", &mut fd),
            Shell::PowerShell => generate::<PowerShell, _>(&mut app, "grab-xkcd", &mut fd),
            Shell::Elvish => generate::<Elvish, _>(&mut app, "grab-xkcd", &mut fd),
        }
    }
}

A mouthful of a match but on closer inspection isn't too bad.

We create the clap::App struct from our CLI just like in the compile time version
We get a reference to the stdout buffer which implements the std::io::Write trait that clap_generate::generate is expecting.
We match on our shell type, and then pass on to the real generate function

Testing them out

With that all wired up, it's time for a test!

$ grab-xkcd --completions bash | head
_grab-xkcd() {
    local i cur prev opts cmds
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    cmd=""
    opts=""

    for i in ${COMP_WORDS[@]}
    do

Looks the same as before! I bet we can source/eval it.

Yay!

Wrap Up

In this article we've seen how we can add shell completion scripts for various shells with relative ease. We've seen both completion scripts generated at compile time, and run time.

It's totally possible to implement both strategies as well.

In all honesty, one could mock existing CLIs not written in Rust, and simply generate completion scripts for them using this method.

Again, the complete code from this article can be found on the two branches:

Compile time: completions-ct
Run time: completions-rt

Rust RedBPF Networking (Part 4)

Kevin K. — Tue, 05 Jan 2021 17:55:44 +0000

Current lay of the land

NOTE: Before we start I need to disclaim that this post/series is NOT//NOT a critique of redbpf-*! This is simply my observations for my own use case, with ideas for improvement. The developers of RedBPF are extremely professional, and my going back and forth with them over chat I can assure you they are also extremely proficient and running laps around me at low level code (especially when it comes to things like LLVM code generation and other topics that make the hair on the back of my neck stand up!).

First, we need to acknowledge what our state of affairs is, and what our goals should be before we can address whether or not our goals are being met and if we can improve on methods to achieve them.

Let's look at how it currently works under the hood. We will be spending 99% of our time in redbpf-probes because that's where the majority of the networking code lives. We may hop over to the companion crate redbpf-macros for a brief time since although a companion it's basically essential to using the networking modules in redbpf-probes. Once we get a feel for where we are we'll have a better idea of any shortcomings, if any. If we do find areas we'd like to modify I won't mention any specific changes I'd make just yet, and will wait until I'm done walking through the current implementation so as not to get side tracked down any rabbit holes with possible details on implementation ideas.

XDP Entry Point

Like all good programs, we must start somewhere. Unlike a traditional binary that we'd execute on the host machine, BPF programs don't usually have a main() function. Instead, we have various code sections of the ELF binary that the kernel will run directly, which is typically a function (or multiple
functions that have been inlined into a single function) of some form. This means we only need to define a function with a correct signature, and tell the kernel where it's located in the final ELF binary.

In redbpf-probes we define our entry point function by tagging it with the attribute #[xdp]. This attribute is a procedural macro that performs a few useful tasks.

The #[xdp] macro is defined in redbpf-macros. Of the tasks it performs for us, perhaps most importantly it wraps our user function in an outer function with the correct kernel BPF API.

It then places this outer function in a particular ELF section (the section name xdp/foo [where foo is our inner function] is used arbitrarily). Next this macro takes the kernel context pointer which is a raw pointer (*mut) to an
xdp_md struct and wraps it in a more friendly redbpf-probes type before handing that to our inner function. The new type is called XdpContext which is the actual value that gets passed to our inner function. This new type allows
redbpf-probes to implement additional methods and functionality on top of the raw pointer handed out by the kernel.

The outer XDP function also takes the return value we (inner function) finish with and performs a match on it, turning this value into something BPF can understand. It's worth noting that if we return an Err variant, it will get
translated into XdpAction::Pass (accept packet and continue up the networking stack).

If you're not into procedural macros, no worries the simplified expansion of our #[xdp] marked function foo() is the following:

#[no_mangle]
#[link_section = "xdp/foo"]
fn outer_xdp(ctx: *mut xdp_md) -> XdpAction {
    let ctx = XdpContext { ctx };
    return match foo(ctx) {
        Ok(action) => action,
        Err(_) => XdpAction::Pass
    };

    fn foo(ctx: XdpContext) -> XdpResult {
        // our code here ...
    }
}

In this post we'll dive into all the pieces that make up the above expansion, as well as the components that make up our example BPF program from Part 3 as well.

First up, let's start with that new type XdpContext given to us by the outer function.

The `XdpContext` struct

At the beginning of our inner function, we're given an instance of XdpContext where the inner value is pointer to a binding of a C struct xdp_md.
The xdp_md struct contains a couple of values that'll we'll be interested in, but only two are used in redbpf-probes, the data and data_end fields which are addresses in memory (not pointers) for the beginning and end of the packet
memory.

This means the XdpContext struct wraps a pointer, which itself contains "pointers" directly to packet memory. Unlike other contexts we'll see later on, the memory addresses within the range pointed to by these fields is directly
mutable.

The XdpContext struct doesn't provide a whole lot on top of the xdp_md struct, just a method to retrieve the inner raw pointer to the xdp_md struct. It does however implement a trait that lets us do a lot more with that memory.

The NetworkBuffer trait.

The `NetworkBuffer` trait

The current design of the networking portions ofredbpf-probes in a way revolves around the trait NetworkBuffer. Its purpose is to represent abstract functionality for any buffer of network memory as described by two memory addresses (for the beginning and end of the buffer).

This trait allows a type to implement accessing raw pointers and memory of the buffer (with some basic bounds checking math), and also some convenience methods for retrieving pointers to specific header structs that may be contained
in the buffer itself. This is all done via pointer arithmetic to increase or decrease the addresses of either the beginning or end of buffer.

Wait, what is a network packet?

Let's take quick step back and look at what a network packet is made of first.

You can think of a network packet as a "header" followed by a "body/payload".

I use the terms body and payload interchangeably. As well as the terms header and prefix.

-- ¯\(ツ)/¯

The header lays out a very strict set of packed fields/values along with their byte offsets. In this way we can pack as much information into as few bytes as possible without having to waste space (bandwidth) on sending "useless" data such as things like padding. Directly after the header starts that particular packet type's body.

The body, is many times an encapsulated header of another type, along with another inner body. These header/body encapsulations can continue for an arbitrary depth until you reach the final body/payload which is the application
data being transmitted.

Headers can be various sizes depending on the protocol they're representing, and almost all of them have varying fields. The total packet size (so all the headers + the final body) is limited by the Maximum Transmission Unit (MTU) for
a given network medium. For example with Ethernet and it's family that is 1500 bytes. So we can't really just recursively encapsulate forever, or we'd not have any room left for the actual application data.

If we have application data that is too large for the space left (after all headers have been encapsulated) that data can be broken up into multiple network packets at either the application level (which is preferred), or the network level (known as packet fragmenting).

There are also minimum packet sizes, such as 64 bytes for Ethernet, so there are instances were we could end up adding padding of some sort to the final body if the application data was extremely small.

All this talk of "levels" means I should bring up the OSI networking model; which defines nice delineations of responsibilities for a typical network packet. When creating a network packet
(i.e. the bytes are being written to a buffer by the application or OS) it's standard to first have a "layer 2" header, followed by a "layer 3" header, and finally a "layer 4" header whose body contains the application data.

If you're not familiar with that model, it's OK, it's just to say for example an Ethernet packet will encapsulate a IPv4 packet, which will encapsulate a TCP packet, who will encapsulate the application data. The layers just refer to how the model stacks the protocols and where they fit in the abstraction stack.

In the above image "packet" can be confusing because the term is overloaded, but it only refers to "layer 3" protocols. Also note that layer 1, "physical" is the actual bits on the wire and not something we are concerned with while parsing
network packets. We're primarily concerned with layers 2 through 4, and final body/payload which makes up layers 5 through 7 which the application is responsible for.

Accessing Raw Headers in a `NetworkBuffer`

When we have a raw network buffer, we'll most likely want to find out what the first header is, so we can start to do some further parsing.

At least for the first header, we can make an educated guess at type of header based off the network medium the packet is coming from. For example, if we're attaching this XDP program to an ethernet NIC, chances are good (if not certain) the packet's first header is an Ethernet (802.3) header. Likewise, if we're attaching this XDP program to a wireless NIC, chances are
good (if not certain) the first header will be an 802.11 header for Wireless LAN

redbpf-probes currently only supports Ethernet (802.3) out of the box, one would have to parse the bytes as another header type manually.

For example, the trait provides NetworkBuffer::eth which returns Option<*const ethhdr> by trying to interpret the first few bytes of the buffer as an Ethernet (802.3) header and gives back a raw constant pointer on success or an Err otherwise.

But what constitutes success? Currently, the only way to fail this is not if the packet does not contain an Ethernet header, but instead if the packet buffer is too small to contain an Ethernet header. So long as the packet contains
enough bytes to hold an Ethernet header, the bytes will be interpreted as an Ethernet header (*const ethhdr) regardless of what the actual bytes were supposed to represent.

This means validation of any kind is left to the caller.

Other methods on the trait do similar, such as NetworkBuffer::ip which assumes the first few bytes of the buffer are an ethernet header, then tries to parse the next few bytes as an IPv4 header giving out a raw constant pointer on success.

This method does a little bit of validation, because as part of the Ethernet header, there is a field that lists the encapsulated packet type. If the field is anything other than IPv4, an Err is
returned. Like the NetworkBuffer::eth method, having a packet that is too small to contain both an Ethernet and IPv4 header will be considered invalid and return None as well.

Further down the list we see two interesting methods, as they don't return raw pointers (at least on first glance);
NetworkBuffer::transport and NetworkBuffer::data.

Transport Headers

We'll first look at NetworkBuffer::transport which we're using in the example code in Part 3. It returns an enum where each variant is a tuple struct with an inner value of a raw constant pointer to some type of transport protocol header (such as TCP or UDP).

This enum provides access methods to both source and destination ports since both TCP and UDP utilize such.

Like the NetworkBuffer::ip method, it also uses the built in IPv4 header fields to determine which type of packet it is encapsulating, and returns the appropriate variant, as well as the standard size checking like all previous methods.

Payload Data

Finally, NetworkBuffer::data returns a new struct we haven't seen yet called Data<T> which
represents a packet payload after all the known headers. An interesting note about Data<T> is that it's generic over some type T that implements NetworkBuffer, which is essentially XdpContext or SkBuff (which we haven't
discussed quite yet, but will shortly).

Also, the way a Data<T> is created is by first matching on the NetworkBuffer::transport return, and then calculating a base address of the payload after the headers. If you'll remember from just a moment ago, to get the Transport enum, we first make the assumption that the packet contains an
Ethernet and IPv4 header.

The type Data<T> itself provides a few basic methods for getting the current offset from the base of the packet memory, viewing this data as a Rust slice, and perhaps most interestingly reading raw bytes into a NetworkBufferArray. As it turns out NetworkBufferArray is just a marker trait for byte arrays of 512 bytes or smaller (and is our second/final tiny dip into redbpf-macros).

Recursive Packets

It's important to note that Data<T> requires that T implement NetworkBuffer. In theory this should allow us to parse recursive packets which is something is somewhat common in network traffic. A recursive packet is instead of the traditional Layer 2 containing Layer 3 which contains Layer 4 (as I'll abbreviate L2->L3->L4, where -> means "contains" or "encapsulates), a layer contains either another header of the same layer, or potentially even a layer above itself. For example, where a TCP payload contains another TCP header, or an IPv6 payload contains an IPv4 header (which then contains
something like a UDP header, etc. etc.).

However, while parsing down the headers past the first layer 4 header (i.e. TCP or UDP), there doesn't seem to be an easy way to get a new NetworkBuffer of just the payload region where all the pointer addresses are pointing to the new
"base" (i.e. what was the payload) so that we can continue to parse headers. And even if there were, because the current implementation assumes the headers are strictly L2->L3->L4, there is not easy way (i.e. not manual) to parse a packet
that is something like L2->L3->L4->L4 or even L2->L3->L3->L4->L4.

For example, the L2->L3->L4->L4 mentioned above, may look like this if the packet was a "TCP in TCP" packet (Note, this is generally a bad idea...but sometimes things like this are required or unavoidable):

SkBuff

The final piece to talk about is SkBuff which is the sibling to XdpContext except for Socket or TC BPF programs. Instead of a *mut xdp_md, it wraps a *const __sk_buff. If you'll glance back
at part 2, __sk_buff is the struct context that is created once the kernel has done some basic parsing of the packet data, and allocated some memory to hold a sk_buff struct (Socket Buffer). BPF programs aren't given direct access to the packet memory anymore, and instead are given the __sk_buff which mirrors the kernel's in memory sk_buff struct. In order to mutate the contents of the actual packet, one must call the bpf_skb_write_bytes helper function.

The SkBuff struct provides one method, SkBuff::load for reading raw bytes out of the internal buffer by offset.

Because the bytes pointed to in either the XdpContext or SkBuff are networking bytes, they are stored in "network byte order" (Big Endian) and must be converted to "host byte order" (which is often Little Endian, but determined by platform). SkBuff::load takes care of the conversion for us, but XdpContext, NetworkBuffer, or Data<T> do not, and leave that to the calling code.

NOTE: A full discussion on Endianess is beyond the scope of this series (see here for a quick intro), but essentially it's the order in which bytes are stored (least significant bits first is Little Endian, and most significant bits first is Big Endian).

Wrap Up

Now that we've done a good quick deep dive of what's currently implemented and available in the RedBPF networking modules, we can begin to look at improvements we could make in the next post!

DEV Community: Kevin K.

Configuration Tetris

Play Field

Rules

Falling Blocks

Using a Trait in Derive CLIs

Previously On...

Builder Trait Revisited

Back to present day

CLI Value Structs

Single Source of Truth in run

Using Box<dyn Cmd> will now allocate...a lot

Context

Our First Hard Decision

Child Contexts

Enums and Subcommands

The Rest of the Owl

Summary

Appendix A: Short Lived CLI Structs

Removing run

Renaming run to runner

Derive Based CLIs

Previously On...

Going Back in Time

Running the Program

Naive Matching and no Ctx

A second talk about Ctx

Next Time

Appenix A: Removing More Layers

Using a Trait in Builder CLIs

Previously On...

Traits have entered the chat

Cmd::update_ctx

Cmd::run

Cmd::next_cmd

Traversing the call-stack with Cmd

Our new main

CLI Structs

Intermediate Subcommands

The Rest of the Owl

One Final Change

Summary for Builder based CLIs

Next Time!

Builder Based CLIs

Previously On...

Let's gooooo

Running the Program

Naive Matching and no Ctx

Adding Ctx

Next Time

CLI Contexts

Intro

Type of CLI

The CLI - bad rustup...bustup

Parsing Library

bustup Command Structure

Context Structs

Meeting Ctx

Single Source of Truth

Initializing Ctx

Next Time!

Generically Bloated

Prelude

Basics

Library Authors Beware

Preconditions

Immediate Dispatch to the Rescue

In Action

Counting Functions

aside: cargo-llvm-lines

cargo-bloat

Generic Bloat

aside: macros

Trying Immediate Dispatch

A Note on Inlining and Release Builds

Why not just use a String as the parameter instead of the generic?

An Even More Contrived Example

Conclusion

Using navi for CLI Cheats

Enter strace

Single Source of Truth in `run`

Using `Box<dyn Cmd>` will now allocate...a lot

Removing `run`

Renaming `run` to `runner`

Naive Matching and no `Ctx`

A second talk about `Ctx`

`Cmd::update_ctx`

`Cmd::run`

`Cmd::next_cmd`

Traversing the call-stack with `Cmd`

Our new `main`

Naive Matching and no `Ctx`

Adding `Ctx`

`bustup` Command Structure

Meeting `Ctx`

Initializing `Ctx`

aside: `cargo-llvm-lines`

`cargo-bloat`

Why not just use a `String` as the parameter instead of the generic?

Enter `strace`

`navi` Overview

What is a `.cheat` file?

Terminator, Tilix, or `tmux`

`clap` to the rescue!

Using `--completions`

`clap_generate::generate`

The `XdpContext` struct

The `NetworkBuffer` trait

Accessing Raw Headers in a `NetworkBuffer`