DEV Community: Stephen Ball

From awk to a Dockerized Ruby Script

Stephen Ball — Sun, 08 Aug 2021 04:00:00 +0000

For a programming project I wanted to easily get a list of the printable ASCII characters such that I could easily loop through them and do some work with each individual ASCII character.

That seems simple enough! But I didn’t quite find anything that did what I wanted.

I found the very cool ascii program available via homebrew but as awesome as it is it does not have an output that is simply a list of the printable ASCII.

I found printable ASCII lists online of course, but I didn’t want to simply cut and paste into a text file.

So I turned to scripting out something and AWK is a great one for simple scripts.

$ awk 'BEGIN { for(i=32;i<=126;i++) printf "%c\n", i; }'

!
"
#
$
%
&
'
(
)
*
+
,
-
.
/
0
1
-- etc --

That’s nice and all but I didn’t want to keep making myself go to ctrl-r to use that data output.

So throw that into $HOME/bin/printable-ascii and done?

Well no. As I wrote that into a script I figured, since I’m making this an actual script I might as well take the rare opportunity these days to write some Ruby!

$ ruby -e "32.upto(126) { |n| puts n.chr }"

!
"
#
$
%
&
'
(
)
*
+
,
-
.
/
0
1
-- etc --

There, nice and expressive as we all expect Ruby to be.

A Real Ruby Script

But hold on. Since I’m bothering to put this into a script via Ruby I could output other interesting data as well. Like it would be coold to be able to run a script a see not only the ASCII but the decimal, octal, hexadecimal, and binary representations of the characters.

It’s all been a minute since I wrote a nice command line tool in Ruby and OptionParser is such a fantastic library in the Ruby standard library. It’d be a really refreshing change of programming pace compared to my daily programming work to be able to write something small and useful using only a good language with a good standard library.

A bit of scripting later and tada! printable-ascii 0.0.1. That version didn’t even hit the repo because I was still writing it in my dotfiles ~/bin directory.

But it was starting to be really cool. ASCII is such a neat slice of data and I’d never really parsed through it directly that much.

When I added the JSON output I just had to take this cool little script to its own repo! Fiat sdball/printable-ascii and printable-ascii v1.0.0

Did I stop there? I did not.

Homebrew

I thought it would be fun to see if I could get this script into homebrew. Can a Ruby script even be added to homebrew? Turns out yes and it’s pretty easy thanks to GitHub providing and hosting tar.gz files. ❤️GitHub!

class PrintableAscii < Formula
  desc "Output all printable ASCII characters in various representations and formats"
  homepage "https://github.com/sdball/printable-ascii"
  url "https://github.com/sdball/printable-ascii/archive/refs/tags/v2.1.0.tar.gz"
  sha256 "cf0b2dfa7c1e0eb851be312c3e53d4f67ad68d46c58d8983f61afeb56588b061"
  license "MIT"

  def install
    bin.install "bin/printable-ascii"
  end

  test do
    ascii_json = [
      { "character" => " " },
      { "character" => "!" },
      { "character" => "\"" },
      { "character" => "#" },
      { "character" => "$" },
      { "character" => "%" },
      { "character" => "&" },
      { "character" => "'" },
      { "character" => "(" },
      { "character" => ")" },
      { "character" => "*" },
      { "character" => "+" },
      { "character" => "," },
      { "character" => "-" },
      { "character" => "." },
      { "character" => "/" },
      { "character" => "0" },
      { "character" => "1" },
      { "character" => "2" },
      { "character" => "3" },
      { "character" => "4" },
      { "character" => "5" },
      { "character" => "6" },
      { "character" => "7" },
      { "character" => "8" },
      { "character" => "9" },
      { "character" => ":" },
      { "character" => ";" },
      { "character" => "<" },
      { "character" => "=" },
      { "character" => ">" },
      { "character" => "?" },
      { "character" => "@" },
      { "character" => "A" },
      { "character" => "B" },
      { "character" => "C" },
      { "character" => "D" },
      { "character" => "E" },
      { "character" => "F" },
      { "character" => "G" },
      { "character" => "H" },
      { "character" => "I" },
      { "character" => "J" },
      { "character" => "K" },
      { "character" => "L" },
      { "character" => "M" },
      { "character" => "N" },
      { "character" => "O" },
      { "character" => "P" },
      { "character" => "Q" },
      { "character" => "R" },
      { "character" => "S" },
      { "character" => "T" },
      { "character" => "U" },
      { "character" => "V" },
      { "character" => "W" },
      { "character" => "X" },
      { "character" => "Y" },
      { "character" => "Z" },
      { "character" => "[" },
      { "character" => "\\" },
      { "character" => "]" },
      { "character" => "^" },
      { "character" => "_" },
      { "character" => "`" },
      { "character" => "a" },
      { "character" => "b" },
      { "character" => "c" },
      { "character" => "d" },
      { "character" => "e" },
      { "character" => "f" },
      { "character" => "g" },
      { "character" => "h" },
      { "character" => "i" },
      { "character" => "j" },
      { "character" => "k" },
      { "character" => "l" },
      { "character" => "m" },
      { "character" => "n" },
      { "character" => "o" },
      { "character" => "p" },
      { "character" => "q" },
      { "character" => "r" },
      { "character" => "s" },
      { "character" => "t" },
      { "character" => "u" },
      { "character" => "v" },
      { "character" => "w" },
      { "character" => "x" },
      { "character" => "y" },
      { "character" => "z" },
      { "character" => "{" },
      { "character" => "|" },
      { "character" => "}" },
      { "character" => "~" },
    ]

    assert_equal ascii_json, JSON.parse(shell_output("#{bin}/printable-ascii --json"))
  end
end

The real core of the magic is bin.install "bin/printable-ascii". That installs the printable-ascii script into homebrew’s bin as an executable.

$ brew install --formula ./Formula/printable-ascii.rb
==> Downloading https://github.com/sdball/printable-ascii/archive/refs/tags/v2.1.0.tar.gz
Already downloaded: /Users/sdball/Library/Caches/Homebrew/downloads/1f5ded4652929fb1c8ca5ffdb1a733cdfa3e65e6bf447893ef59803f7f6919b9--printable-ascii-2.1.0.tar.gz
🍺 /opt/homebrew/Cellar/printable-ascii/2.1.0: 5 files, 22KB, built in 1 second
Removing: /Users/sdball/Library/Caches/Homebrew/printable-ascii--2.0.0.tar.gz... (6.3KB)

Right on! Maybe someday it’ll really be in Homebrew but for now it’s easy enough to install directly with the formula.

Docker

Since Homebrew would require cloning the repo and then manually installing from the formula file I should be able to pretty easily wrap up this script into a Docker image! Then anyone could easily run this silly script as long as they have Docker installed. Who doesn’t love running arbitrary scripts from the Internet via Docker?

Since I’ve been helping out as part of the DevOps team at work lately I had some hot loaded Docker knowledge ready to roll. I just need an image with Ruby, copy in the script somewhere, and set the script as the ENTRYPOINT.

Then with docker run any arguments passed to the docker run command will be passed to the script itself. ✨Docker!

I quickly found the official Ruby Docker image and just grabbed the first image I saw referenced in their README

FROM ruby:2.5

# throw errors if Gemfile has been modified since Gemfile.lock
RUN bundle config --global frozen 1

WORKDIR /usr/src/app

COPY Gemfile Gemfile.lock ./
RUN bundle install

COPY . .

CMD ["./your-daemon-or-script.rb"]

I slimmed that down a bit since I don’t have any need for bundler or Gemfiles. Here’s what produced the 1.0.0 Docker image for printable-ascii

FROM ruby:2.5
WORKDIR /usr/src/app
COPY bin/printable-ascii ./
ENTRYPOINT ["./printable-ascii"]

Easy peasy and it worked great!

GitHub Actions

Build a Docker image from my own laptop and publish to Docker Hub? That’s so archaic! What kind of DevOps newbie am I if I let myself live with that kind of publishing story?

GitHub Actions to the rescue! I whipped up a GitHub Actions workflow to publish to both Docker Hub and GitHub’s own container registry whenever I publish a new version of the script. It even checks that the script’s declared version matches the release that’s going to be published.

It’s a bit weird in that it means that the Docker image OF the script has the same version as the script itself. If I rev the script itself then it all makes sense because a new Docker image will contain the new script. But if I only update the Docker image then I don’t have a reasonable way to only change its version.

Thankfully this script and Dockerfile are very simple so I can simply keep them in sync and only update the Docker image when there’s a new version of the script to release.

But in practice for more complex relationships between the utility being provided by the Docker image and the Docker image itself it seems like there’s an opportunity for more metadata. Like a dual versioning of the image and its contents represented in a good, consistent way on Docker Hub.

That Docker image is way too huge

After I setup the GitHub Action I went back to look at my Docker Hub stats and WOW over 300MB for this script’s image! That’s no good at all!

I figured there had to be a slimmer Ruby image to work with. The default I was using probably has all kinda of superfluous (for me) utilities and libraries to support all kinds of projects.

After a bit of reading in Ruby’s official docker image I found there’s both a -slim and an -alpine version of the Ruby image to use. Since my script is really really Ruby and its standard library alone I was certain the -alpine image would work great.

Alpine is a special Linux distribution designed to create small Docker images.

Success! Using the Alpine version brought the image down to ~30MB! Roughly 10% of the original size!

FROM ruby:3.0-alpine
WORKDIR /usr/src/app
COPY bin/printable-ascii ./
ENTRYPOINT ["./printable-ascii"]

It works wonderfully! Anyone with Docker can get printable ASCII anytime

$ docker run sdball/printable-ascii --json --decimal --binary --hexadecimal --octal --range A-C | jq '.'

[
  {
    "character": "A",
    "decimal": "65",
    "binary": "1000001",
    "hexadecimal": "41",
    "octal": "101"
  },
  {
    "character": "B",
    "decimal": "66",
    "binary": "1000010",
    "hexadecimal": "42",
    "octal": "102"
  },
  {
    "character": "C",
    "decimal": "67",
    "binary": "1000011",
    "hexadecimal": "43",
    "octal": "103"
  }
]

Features on features

Working on this simple toy script has been an absolute joy so I’ve continued to add more features that no one needs. Not even me! I just think they’re neat!

A --range option to allow supplying one or more ranges of printable ASCII characters

A --random NUMBER option to pick NUMBER of random printable ASCII characters

And I’ve got plans for more silly options ha ha! Printable ASCII for all!

Maybe someday I’ll actually get back to the project where I needed a list of printable ASCII in the first place.

Finding leap years with the cal command

Stephen Ball — Sun, 11 Jul 2021 14:05:13 +0000

Did you know that there’s a calendar in the macOS and Linux command line? There is!

Introduction to cal

$ cal

     July 2021
Su Mo Tu We Th Fr Sa
             1 2 3
 4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31

A quick check of tldr shows how useful this utility can be

$ tldr cal

cal

Prints calendar information.

- Display a calendar for the current month:
    cal

- Display previous, current and next month:
    cal -3

- Display a calendar for a specific month (1-12 or name):
    cal -m month

- Display a calendar for the current year:
    cal -y

- Display a calendar for a specific year (4 digits):
    cal year

- Display a calendar for a specific month and year:
    cal month year

- Display date of Easter (Western Christian churches) in a given year:
    ncal -e year

But there’s one thing it doesn’t have. Leap years!

Cal knows about leap years of course, it wouldn’t be much of a calendar otherwise. But it doesn’t have a way to list them.

Recognizing leap years, hackishly

Let’s build it ourselves! Sure we could use a real programming language with a calendar library but building things by assembling command line components is fun and it’s not like we’re going to put this calculation into production.

First off, let’s agree that February is the indicator for a leap year.

$ cal 02 2021

   February 2021
Su Mo Tu We Th Fr Sa
    1 2 3 4 5 6
 7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28

$ cal 02 2020

   February 2020
Su Mo Tu We Th Fr Sa
                   1
 2 3 4 5 6 7 8
 9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29

Could we search for “29” and have it recognize leap years? Maybe!

$ cal 02 2020 | grep -q 29 && echo "LEAP" || echo "normal"

LEAP

$ cal 02 2021 | grep -q 29 && echo "LEAP" || echo "normal"

normal

So far so good? But I bet you see the problem.

$ for year in {2020..2030}; do
  printf "$year ";
  cal 02 "$year" | grep -q 29 && echo "LEAP" || echo "year";
done

2020 LEAP
2021 year
2022 year
2023 year
2024 LEAP
2025 year
2026 year
2027 year
2028 LEAP
2029 LEAP
2030 year

2029 is NOT a leap year. It’s only matching because of the 29 in 2029.

$ cal 02 2029

   February 2029
Su Mo Tu We Th Fr Sa
             1 2 3
 4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28

We could remove that troublesome result by removing the entire line with “February” from consideration.

$ cal 02 2029 | grep -v Feb

Su Mo Tu We Th Fr Sa
             1 2 3
 4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28

Now I hold that if we find a “29” in the result then it REALLY is a leap year.

$ for year in {2020..2030}; do
  printf "$year ";
  cal 02 "$year" | grep -v Feb | grep -q 29 && echo "LEAP" || echo "year";
done

2020 LEAP
2021 year
2022 year
2023 year
2024 LEAP
2025 year
2026 year
2027 year
2028 LEAP
2029 year

Does that match up with actual leap years?

Wolfram Research’s Leap Year page has this list as the leap years in the first half of the 21st century.

Let’s see how we do

$ for year in {2000..2051}; do cal 02 "$year" | grep -v Feb | grep -q 29 && echo $year; done
2000
2004
2008
2012
2016
2020
2024
2028
2032
2036
2040
2044
2048

Done!

But this grepping for 29 is too easy. Let’s get weirder!

Recognizing leap years, even more hackishly

The difference between a leap year and a normal year is that extra day of the 29th. Could we check the number of characters in the output from cal to recognize leap years?

Yes we can. But there’s a gotcha to get through first. Let me show you what I mean.

We know 2020 is a leap year and 2021 is not a leap year. Let’s check the difference between the number of characters output for February of each of those years. We’ll do that with the handy wc utility, specifically wc -c to count the byte characters.

$ printf "hello" | wc -c

5

5 characters in “hello” (using printf instead of echo because otherwise we’d have a newline character as well). Great!

$ cal 02 2020 | wc -c

184

$ cal 02 2021 | wc -c

184

There’s the stumbling block. Year each month has different output (2020 has a 29) but they both have the same number of characters because cal adds spaces to ensure consistent formatting. Spaces are characters just as much as numbers.

What can we do? We can remove all the spaces. A great utility for doing that is the tr command. As we saw in Let’s write a shell script, the tr command TRanslates string data. It can, say, change all “a” characters into “A” characters. It can also delete all specified characters which is exactly what we want.

In our case we want to delete all whitespace characters, no matter what they are (spaces, tabs, etc). There’s a useful grouping called a character class for that. [:space:] will target all whitespace characters.

$ cal 02 2020 | tr -d '[:space:]'

February2020SuMoTuWeThFrSa1234567891011121314151617181920212223242526272829

$ cal 02 2021 | tr -d '[:space:]'

February2021SuMoTuWeThFrSa12345678910111213141516171819202122232425262728

Great! Can we count the number of characters to find leap years now? We sure can!

$ cal 02 2020 | tr -d '[:space:]' | wc -c

75

$ cal 02 2021 | tr -d '[:space:]' | wc -c

73

Now let’s output some leap years!

$ for year in {2000..2051}; do
  cal 02 "$year" |
    tr -d '[:space:]' |
    wc -c |
    grep -q 75 && echo "$year";
done

2000
2004
2008
2012
2016
2020
2024
2028
2032
2036
2040
2044
2048

Hacks on hacks and it works!

Which do you think is faster? Yeah I have no idea either. The “remove February” path is maybe faster? But I’d be surprised if there’s a huge difference.

Let’s not be surprised. Let’s find out!

There’s a great utility called hyperfine that allows evaluating the performance of multiple command line calls

Which is faster? Hyperfine will tell us

Here’s the flags I’m giving to hyperfine along with the two commands.

--style basic Plain output styling
--export-markdown hyperfine.md Export results as markdown
--warmup 5 Do five runs before benchmarking
--ignore-failure Ignore non-zero exits

We’re not really concerned with benchmarking the loop of years itself, but the calculation of a year.

$ hyperfine --style basic --export-markdown hyperfine.md --warmup 5 --ignore-failure "cal 02 2050 | grep -v Feb | grep -q 29" "cal 02 2050 | tr -d '[:space:]' | wc -c | grep -q 75"

Benchmark #1: cal 02 2050 | grep -v Feb | grep -q 29
  Time (mean ± σ): 1.8 ms ± 0.3 ms [User: 0.7 ms, System: 1.9 ms]
  Range (min … max): 1.0 ms … 3.6 ms 493 runs

  Warning: Command took less than 5 ms to complete. Results might be inaccurate.
  Warning: Ignoring non-zero exit code.

Benchmark #2: cal 02 2050 | tr -d '[:space:]' | wc -c | grep -q 75
  Time (mean ± σ): 1.9 ms ± 0.4 ms [User: 0.9 ms, System: 2.7 ms]
  Range (min … max): 0.7 ms … 5.9 ms 811 runs

  Warning: Command took less than 5 ms to complete. Results might be inaccurate.
  Warning: Ignoring non-zero exit code.
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  'cal 02 2050 | grep -v Feb | grep -q 29' ran
    1.07 ± 0.28 times faster than 'cal 02 2050 | tr -d '[:space:]' | wc -c | grep -q 75'

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`cal 02 2050 \	grep -v Feb \	grep -q 29`	1.8 ± 0.3	1.0
`cal 02 2050 \	tr -d '[:space:]' \	wc -c \	grep -q 75`	1.9 ± 0.4

As expected, practically no difference between them as far as wall clock time, BUT the “remove February” path may be slightly faster.

That brings this command line tool assembly journey to an end.

Wrap up

You can build new command line features by assembling existing tools into a data pipeline
A bit of creativity can get surprisingly complex results from simple pieces
Hyperfine is a useful tool for benchmarking command line commands

Next steps

Mess around the the cal command.
Can you build a pipeline that counts all the non-space characters for a given year? e.g. cal 2020
Try running that pipeline against 1700–1799: notice anything odd?

The Problem of State

Stephen Ball — Sat, 01 May 2021 00:00:00 +0000

Among all the problems we create for ourselves when programming systems state is perhaps the most troublesome to deal with. Decisions made about the state of the system and its lifecycle with the application have far reaching consequences that can be extrememly difficult to unravel.

Not making deliberate decisions about the lifecycle of state within an application guarantees an application that will inevitably difficult to work on and changes will become slower and slower. At that point you must either live with the slow/difficult to change application, heavily refactor the system to organize the state, or throw out the whole thing and write an entirely new application.

Pure Functions

First let’s talk pure functions. These functions are living the dream! The darlings of unit testing tutorials everywhere.

What’s a pure function?

A pure function always produces the same output from the same input. A pure function has no side effects. No HTTP requests, no database queries, no looking up the time from the system clock, not even printing text or logging to a file.

def add(a, b)
  a + b
end

const add = (a, b) => a + b;

Input comes in, output goes out. When you’re dealing with pure functions you can definitively, absolutely know everything that function is taking in and that the same arguments will always produce the same output.

You can see how unit testing tutorials absolutely love pure functions. Testing pure functions is almost so easy that it might seem almost unnecessary if you’re only concerned with proving correctness.

Of course pure functions can complex but they always take some arguments and return the same output from those arguments. Pure functions can’t sneak in extra data from a database query or web request or referring to some object in memory.

Calculating bowling scores is a relatively famous pure data problem that turns out to be surprisingly complex. You can make the score calculation a large and complex pure function. To make it understandable you’d probably want to break it down into smaller functions. As long as all the functions are pure then the top level coordinating function can be considered pure as well (same output, no side effects).

But the point is that testing pure functions is a dream because you only need to determine the shape of the input data and then describe how it relates to the output data and confirm that the expectations are met.

When I give the “add” function two numbers the output is the sum of those two numbers. Every time.

When I give the “calculate bowling score” function a game’s score card the output is the score of that game. Every time.

Adding State

When we add state to the mix we suddenly have a lot more power. We can write to the database! We can log to a file! And, critically for this discussion, we can hold data in memory.

We can hold the ongoing bowling game in memory and modify the score based on the latest frame e.g. “bowlingGame.recordFrame(3, 7)”. We can add more and more numbers to our growing overall number system “numberTotal.plus(2)”.

That data in memory is state. And while it does give us flexibility and power it costs more than we may expect.

When we modify a function to have side effects such as writing data elsewhere we call it an impure function.

The Problem of State

State isn’t the exclusive source of bugs by any means, but I argue it’s the source of the most complex bugs to track down and fix. When you’re dealing with state suddenly your function’s inputs are not absolutely knowable. And that’s a real problem!

We can call something like bowlingGame.recordFrame(3, 7) and not know how many frames have already been recorded. That’s not good because a bowling game has a fixed number of frames by definition.

What happens to our system if we accidentally record too many frames? Will the extra frames be rejected? Will the score be added beyond the defined limits of a bowling game? Or, insidiously, will the system seem to work for a while until we call the bowlingGame.printResult() function and it explodes with exceptions due to the unexpected extra data?

That separation of an input from its bad effects makes debugging applications with state so intrinsically difficult. We have to hold not only the function we think we’re working on but all of the functions that preceded that function and all of the effects that they had on the state of the system.

Even when we’ve figured it all out and we know the problematic state if we want to test the problem then we have to further figure out how to stage the bad state in testing. For that “bowlingGame” example we’d need to build a testing setup that adds too many frames and then triggers the buggy behavior of “printResult” and ensure that printResult doesn’t explode if there are too many frames.

But later we, or some other ill-fated programmer, followup on that work and realize that printResult wasn’t the problem. The actual problem was recordFrame allowing too many frames! Fix the recordFrame code and job done! We could even write tests around the recordFrame code to assert the correct behavior.

But then running the entire test suite suddenly has failing tests for “printResult”? How could that be related? We have to dig into those tests (the production code is fine!) and hopefully recognize that it was the testing setup that intentionally adds too many frames in order to test that printResult doesn’t explode with exceptions in that state.

Even in this simple contrived example can easily cause those “our test suite is buggy and unreliable” symptoms! We can thank state for that kind of behavior.

If you’ve ever had to track down a memory leak well that’s state that isn’t cleaning up after itself. State that’s lingering and accumulating after its request has long since completed. More and more requests build up more and more bits of lingering state and… memory utilization chart goes up and to the right.

State gives our systems a LOT of flexibility and power. But it’s not without cost. When you don’t carefully, intentionally manage state it will become a nightmare for future development work. In fact I claim that overgrown state is one of the major anchors that drags down engineering work on an application over time.

State is powerful and any reasonably complex application will need at least some state. But carefully handle it. Be explicit about when and how you use it as much as possible.

Elixir and state

As a quick final note I’ll say that Elixir has my favorite approach for handling state. There is no ambient state. When you need to have state within an Elixir application then you have to write a “server” to hold the state and respond to calls to update or return that state. The implementation functions of that server take the state as an argument. I cannot overstate that point enough: that means you can test those functions using the state as pure functions.

But wait, you may say, all the callers have to hold and pass the state to the server? No! The server holds the state. Callers interact with the server using the server API and the server API then calls the implementation functions with the arguments from the API and the current state.

Even better? The implementation functions return the state as part of their returned data! The server API holds the state for itself (to pass to the next call to the server) and passes the explicitly declared return data back to the original caller. That’s all assuming the function call was even synchronous in the first place because the server API has explicitly named functions for making asynchronous (“cast”) vs synchronous (“call”) requests to the server.

Clojure Functions in Four Ways

Stephen Ball — Sun, 31 Jan 2021 00:00:00 +0000

Perl may have coined TMTOWTDI (there’s more than one way to do it) but Clojure takes that idea and runs very far with it.

Today I’d like to quickly show you four ways to create a function in Clojure.

Each block of code will define a doubler function that simply doubles its given integer argument.

defn

This is essentially the standard way to define a function in Clojure: name, args, function body.

(defn doubler [n] (* 2 n))

def and fn

(def doubler (fn [n] (* 2 n)))

This is a verbose variant of defn and is essentially what defn is doing behind the scenes. Here we have an anonymous function (the fn bound to a name with def).

Function literals

Clojure also allows defining functions with an even shorter syntax than fn. The # character can define function literals. With that syntax you can’t have named arguments but instead get positional references e.g. %1 %2 %3

(def doubler3 #(* 2 %1))

Using partial

Now it gets interesting. Clojure provides the most wonderful and excellent partial function. The partial function accepts a function and some arguments and returns a new function that calls the original function with those originally given arguments plus any remaining required arguments.

Ah it’s hard to explain so I’ll show it: this is doubler using partial.

(def doubler (partial * 2))

What’s happening there?

The def doubler is simply assigning the name so let’s ignore that.

The real interesting bit

(partial * 2)

That call to partial accepts a function * and an argument 2 and returns a new function that will accept more arguments to supply to the (* 2) call.

The new function takes any arguments it’s given and supplies them to (* 2) as though we had called it directly.

(def doubler (partial * 2))

(doubler 10) ; => 20 == (* 2 10)
(doubler 3 4) ; => 24 == (* 2 3 4)

You can see it’s not quite the same as the first two doubler functions in that it takes any number of arguments. Such is the power and flexibility of partial.

See Some Clojure

Stephen Ball — Tue, 19 Jan 2021 00:00:00 +0000

Have you seen any Clojure code?

Yes? You’ll almost certainly not see anything new or interesting here.

No? Let’s change that! I’m certainly not an expert: but I’ve been reading “Getting Clojure” by Russ Olsen so I at least know enough to give a quick tour.

Basics

Hello!

(println "Hello World")

Not too weird yet. There’s parentheses around the line but it’s not too far from something like print "Hello World".

But here, the following line adds up the integers 1 through 4.

(+ 1 2 3 4) ; 10

If you’ve never seen a Lisp language then this probably looks weird. Clojure is my first Lisp dialect and sure looked strange to me at first. But I was surprised to find how quickly I came to appreciate the clear structure of the parentheses.

xkcd Lisp Cycles comic

As a Lisp dialect, the syntax of Clojure is structured data in parentheses. To make anything happen in Clojure it’s gotta be in parentheses.

Comments in Clojure are prefixed with a semicolon (a double semicolon conventionally for lines that are entirely comments).

The structure of every (as far I have seen) Clojure call

(function arg0 arg1 arg2 arg3 … argN)

With that knowledge in mind we can make sense of that first (+ 1 2 3 4) line. The + function is being called with the arguments 1 2 3 4. The equivalent line In a non-Lisp language would be 1 + 2 + 3 + 4.

One of the major foundations of Clojure is that everything is done via functions. Arithmetic, variable assignment, logic, conditionals, and even key lookups in a map are done by treating the key as a function.

;; Arithmetic
(+ 1 2 3) ; 6
(/ 6 2) ; 3
(/ 24 2 3) ; 4

;; Variable Assignment
(def greeting "Hello")

;; Logic
(= 2 3) ; false
(= 2 2 2) ; true
(> 9 7 3) ; true
(> 9 4 5) ; false

;; Conditionals
(def a 9)
(def b 3)
(if (> a b) a b) ; 9

;; Map lookup
(def zork1 {:released 1980 :genre :text-adventure})
(:released zork1) ; 1980
(:genre zork1) ; :text-adventure

You’ll notice that a bunch of those functions take more arguments than you may expect. In many languages equality is a two argument expression e.g. a == b but in Clojure you can compare an arbitrary number of arguments quite expressively.

Conditionals

In Clojure conditionals are also functions. The if function is called with 2 (optionally 3) arguments.

If the first argument evaluates to true then the second argument is evaluated and its result becomes the result of the entire statement.
If the first argument evalulates to false then the result is nil OR the optional third argument is evaluated.

user=> (if (= 2 2) "yes 2 equals 2")
"yes 2 equals 2"

user=> (if (= 2 3) "yes 2 equals 2")
nil

Now the if statement from before can make more sense.

(if (> a b) a b)
; ⌃⌃⌃⌃⌃⌃⌃ ⌃ ⌃
; │ │ │
; └────── first arg: the logic to evaluate
; │ │
; └──── second arg: evaluated if first arg is true
; │
; │
; └── third arg: evaluated if first arg is false

Variables

Defining variables in Clojure is a function as well. The def function accepts a name and anything that evaluates to a value.

user=> (def number 123)
#'user/number

user=> number
123

user=> (+ number 4)
127

Functions

As a functional language, functions are naturally at the heart of Clojure. They can be defined with the fn function which accepts a vector (basically an array) of arguments followed by statements to evaluate. The last statement’s result will be the returned result of calling the function.

(fn [n] (* 2 n))

We can assign that function to a name with def

user=> (def doubler (fn [n] (* 2 n)))
#'user/doubler

user=> (doubler 3)
6

user=> (def printy-doubler (fn [n] (println "Doubling" n) (* 2 n)))
#'user/printy-doubler

user=> (printy-doubler 3)
Doubling 3
6

Using def and fn works but is not great experience that Clojure wants its programmers to have. Clojure provides the defn function to easily name and define a function in one call.

user=> (defn tripler [n] (* 3 n))
#'user/tripler

user=> (tripler 3)
9

The theme of providing useful functions for common actions is a major feature of Clojure. It provides many functions simply to make its programming more concise, expressive, and joyful.

Common Data Structures

Clojure has the commonly used data structures you should expect from a modern programming language.

As a functional language Clojure has immutable data structures. If you aren’t familiar with their use the quick gist is that whenever you manipulate data such as adding a new item to a list then you don’t mutate that list: a new list is created with the additional item.

Vectors

An ordered collection of items in a continuous block of memory (i.e. an array) that are defined with brackets.

[1 2 3 "four" ["any data even another vector"]]

(def v [1 2 3 4])
(count v) ; 4
(first v) ; 1
(rest v) ; (2 3 4)

Lists

Lists are also an ordered collection of items, but rather than being stored in a continuous block of memory they are implemented as linked lists. That has ramifications for when it’s appropriate to use either data structure: e.g. it’s trivial to add a new item to the beginning of a list but relatively expensive to add a new item to the beginning of a vector.

In practice vectors are far more commonly used.

Lists are defined with parentheses which means they need a slight tweak over what we’d expect so Clojure doesn’t confuse them with expressions: we need to prepend them with '

'(1 2 3 "four" ("a nested list"))

(def l '(1 2 3 4))
(count l) ; 4
(first l) ; 1
(rest l) ; (2 3 4)

Commas?

You may have noticed above that neither vectors nor lists used commas between the values. Clojure neatly does away with any comma arguments by treating them as whitespace. You can use them if you must but the common approach is to simply omit them.

;; these vectors are all the same data
(= [1 2 3 4] [1,2,3,4] [1,,,,,2,3,4] [1,2,3 4]) ; true

;; commas are whitespace even for expressions
(+,1,2,3,4) ; 10
(+,1,2,3,4,) ; 10

Maps

Data structures that associate keys and values (i.e. dictionaries or hashes). Note commas aren’t required between sets of key/values (because commas are whitespace).

Any value can be a key but Clojure programs usually use keywords as keys e.g. :keyword. Keywords can be used as functions themselves to retrieve values from a map.

;; a bare map
{:key "value" :another-key "another value"}

;; assigned to a variable
(def ps5 {:name "Playstation 5" :publisher "Sony" :released 2020})

(:publisher ps5) ; "Sony"
(keys ps5) ; (:name :publisher :released)

(conj ps5 {:output :hdmi})
;; {:name "Playstation 5", :publisher "Sony", :released 2020, :output :hdmi}

ps5
;; {:name "Playstation 5", :publisher "Sony", :released 2020}

Note that the conj (conjoin) function above does not modify the ps5 variable. It returns a new map with the additional data.

Documentation

Clojure has inline documentation you can access in its repl.

user=> (doc conj)
------------------------------
clojure.core/conj
([coll x] [coll x & xs])
  conj[oin]. Returns a new collection with the xs
    'added'. (conj nil item) returns (item). The 'addition' may
    happen at different 'places' depending on the concrete type.

user=> (doc +)
------------------------------
clojure.core/+
([] [x] [x y] [x y & more])
  Returns the sum of nums. (+) returns 0. Does not auto-promote
  longs, will throw on overflow. See also: +'

user=> (doc doc)
------------------------------
clojure.repl/doc
([name])
Macro
  Prints documentation for a var or special form given its name,
   or for a spec if given a keyword

Wrapping Up

You’ve now seen some Clojure! This isn’t even close to a comprehensive overview of this powerful and expressive language. Clojure has many more constructs to help write powerful programs simply.

If you’re interesting in learning more I highly recommend Russ Olsen’s Getting Clojure from the Pragmatic Programmers.

A simple language spec isn’t a feature when you’re building applications

Stephen Ball — Sun, 11 Oct 2020 13:35:00 +0000

Go famously gushes about its simple and readable language specification. One one hand kudos to Go for having a written specification and kudos for rigorously keeping the language simple.

But on the other, more pragmatic, hand a simple language spec means the language itself has fewer affordances out of the box. Programmers will be left to spin up their own designs for everything else. And, as we see in JavaScript, that’s a problem if you want to encourage a shared language community.

Everyone comes up with their own patterns, patterns which may not be obvious even as they are made up of the common components from the language spec.

A little copying

Go actively encourages this way of programming with the idiom “a little copying is better than a little dependency.” Here they are actively avoiding one of the major pitfalls of NodeJS. But at what cost?

A lot of places

I say the cost is higher than the blithe quote indicates. A little copying from one place leads to an accumulation of a lot of copying from many places. Because the copying isn’t named, versioned, or isolated it is peppered throughout the resulting program. And that means if you find a flaw you have many places to fix. And, worse, if someone else finds a flaw you have no idea unless you come across and recognize the improved version of that little copy.

Go is for systems not applications

Now, to be clear, I don’t think this is a flaw for Go’s supposed primary purpose: a systems programming language. Systems programming should be short, simple, sharp tools to perform high quality work with a strong performance story. You don’t want to burden your systems with huge, complex, dull, and ill-performing tools. Excel is wonderful, Excel is not a systems tool. Excel is an application.

But many programmers out there are building, or trying to build, the equivalent to Excel in Go. To write not sharp tools but large applications in Go because Go has become their hammer and everything is a nail even if it’s a skyscraper.

The Fastest Possible Tests

Stephen Ball — Sat, 03 Oct 2020 15:40:00 +0000

I love tests as a fast programming feedback loop! In the last twenty or so years I’ve seen them go from an occasional critical verification to a foundational component of modern systems design.

Why care about fast tests?

When I’m writing code, especially when doing test driven development, I want the fastest feedback possible. That means fast tests.

I find 100ms is my upper limit for considering any individual test to be fast enough. That is still painfully slow but only just.

Any slower than 100ms and I find the delay between calling for the test and noting its result is just enough of a stuttering friction that I — even unconsciously — start to shy away from running the tests. I’ll spend more time writing the code I think is right vs simply allowing the tests to give me that sweet green confirmation. I’ll go longer between writing new tests and start to spike off code without any guardrails or guidance.

If I’m in a codebase whose unit test suite take seconds or (gasp) minutes to complete I soon advocate for deleting such painfully useless tests.

To be clear I am talking about unit tests and not integration tests. Integration tests can require an actual database to actually take the data and actually store it by actually writing it to some structure either in memory, on disk, or (worse of all) on a remote machine. Those kind of tests have real limits on their speed depending on how much you are willing to isolate them and still consider them useful.

But unit tests: no network, no services. Only the purity of potentially production code and testing code calling the production code and making assertions about the results.

How fast can they go?

How fast should they go?

Ideally unit tests should take less than 10ms to run and at most 100ms.

If your language or testing framework does not allow running individual unit tests then the entire unit test suite should run in less than 1000ms (1 second).

If you’re using any widespread general programming language this is entirely possible even for test suites with thousands of tests. Computers are FAST!

How is that possible?

That’s a topic for a future blog post! But start with A Set of Unit Testing Rules by Michael Feathers.

A test is not a unit test if:

It talks to the database

It communicates across the network

It touches the file system

It can’t run at the same time as any of your other unit tests

You have to do special things to your environment (such as editing config files) to run it.

How fast is the fastest possible test? Theoretically?

There is an absolutely fastest possible testing speed. If we free ourselves from all the bounds of hardware and software there is a fundamental speed limit on how fast our fastest tests can possibly be.

That speed limit is — of course — the speed of light.

No information can travel faster than the speed of light. No matter how much we optimize, improve, tweak, and streamline we cannot break free of the limitation that we have a roundtrip of information from human to machine and back. Information fundamentally has a speed limit.

We need to tell our system to run a test
Our system needs to transmit the results of that test

How fast is that allowed to be by the speed of light? Obviously VERY fast unless our computer is very far away.

Let’s assume our computer is about two feet away. How fast could our test feedback be?

Two feet of distance means our signal and response needs to travel four feet. (Of course light has to travel within the machine and our own heads but let’s set those distances aside.)

To WolframAlpha! Lightspeed over four feet

lightspeed / 4 feet = 4.067ns (nanoseconds)

It’s a fun mathematical quirk that it takes light just about one nanosecond to travel one foot which makes gauging the speed of light to travel human scale distances using antiquated imperial units relatively easy.

How fast is 4ns? Let’s dive down the units of time!

1 second = 1 second (1 s)
1 second / 1000 = 1 millisecond (1 ms)
1 millisecond / 1000 = 1 microsecond (1 µs)
1 microsecond / 1000 = 1 nanosecond (1 ns)

Only a few orders of magnitude, no problem!

Compared to our reasonable case of 10ms for one unit test? In the time it would take to serially run a thousand of our 10ms unit tests we’d be able to run 2.5 million of our fastest possible 4ns tests!

It’s fair to say we’ll never reach that speed and have any meaningful computation for an individual test. But it’s good to have goals.

Shrink your data into bitfields (and out again)

Stephen Ball — Wed, 29 Apr 2020 16:15:00 +0000

Some applications want to save every byte. Every last byte! Maybe they have memory constraints, maybe they’re sending data over the network, maybe they just like saving bytes.

One way to help save bytes is to use efficient data structures. One popular memory efficient data structure is a bitfield. A bitfield allows complex, predefined data to be stored in a compact series of bits.

In this post I’m going to cheat a bit and use size when serialized to JSON as a proxy for the actual savings. High level programming languages do really neat things behind the scenes to store data efficiently. We’re going to sidestep all of that: JSON is JSON.

Let’s say we have some data

field	value
priority	urgent
location	Room 71
status	empty
candy	peppermint patties

That’s some good data. Now let’s say we need to represent that data in code. A hash should do nicely.

data = {
  "priority" => "urgent",
  "location" => "Room 71",
  "status" => "empty",
  "candy" => "peppermint patties"
}

That’s some nice data! But if we serialize it to JSON we have 99 bytes!

data.to_json.size # => 99

What can we do to use less bytes?

Give up the keys

If we agree on the keys ahead of time we can stop sending those around. That means we need to fix the data into an order and coordinate that among all systems using that data. If we’re ok with that then that’s big step!

smaller_data = [
  "urgent",
  "Room 71",
  "empty",
  "peppermint patties",
]
smaller_data.to_json.size # => 49

Wooo, 49 bytes! We’ve already gotten it down to 49.4% of the original size. We gave up some flexibility because we now we have a fixed set of keys in a fixed order but we saved a lot of space!

We can go even further if we’re willing to give up still more flexibility.

Limit the possible values

If we can give up having arbitrary values then we can much more efficently represent those values.

Take the “candy” for example. Right now it could be any string, any string at all!

Candy	Bytes to represent
peppermint patties	18
m&ms	4
reese’s pieces	14
butterfingers	13
cookies	7

But if we know we have a limited set of possible candies then we don’t need the ability to have any string.

Candy	Number	Bytes
peppermint patties	0	8
m&ms	1	8
reese’s pieces	2	8
butterfingers	3	8
cookies	4	8

That means we if replace “peppermint patties” in our data with 0 (our arbitrary agreed upon value to represent that candy) then we can cut our byte usage down a bit.

even_smaller_data = [
  "urgent",
  "Room 71",
  "empty",
  0
]

even_smaller_data.to_json.size # => 30

Now we’re down to 30.3% of the original size with that optimization alone!

Let’s squish down the other data by limiting their options.

priority = %w(low medium high urgent)

location = (1..100) # i.e. Room 1 to Room 100

status = ["not empty", "empty"]

candy = [
  "peppermint patties",
  "m&ms",
  "reese's pieces",
  "butterfingers",
  "cookies"
]

With those contraints our original data becomes much smaller still.

even_smaller_still_data = [3, 71, 1, 0]
even_smaller_still_data.to_json.size # => 10

10 bytes! We’re down to around 10.1% of the size of the original data!

At this point we need code to serialize the data down and parse it back out.

module IntegerArray
  PRIORITY = %w(low medium high urgent)

  LOCATION = (1..100)

  STATUS = ["not empty", "empty"]

  CANDY = [
    "peppermint patties",
    "m&ms",
    "reese's pieces",
    "butterfingers",
    "cookies"
  ]

  def self.serialize(data)
    Serialize.new(data).to_a
  end

  def self.parse(array)
    Parse.new(array).to_hash
  end

  class Serialize
    def initialize(data)
      @data = data
    end

    def priority
      PRIORITY.find_index(@data["priority"])
    end

    def location
      @data["location"].scan(/\d+$/).first.to_i
    end

    def status
      STATUS.find_index(@data["status"])
    end

    def candy
      CANDY.find_index(@data["candy"])
    end

    def to_a
      [priority, location, status, candy]
    end
  end

  class Parse
    def initialize(array)
      @array = array
    end

    def priority
      PRIORITY[@array[0]]
    end

    def location
      room_number = @array[1]
      "Room #{room_number}"
    end

    def status
      STATUS[@array[2]]
    end

    def candy
      CANDY[@array[3]]
    end

    def to_hash
      {
        "priority" => priority,
        "location" => location,
        "status" => status,
        "candy" => candy,
      }
    end
  end
end

data = {
  "priority"=>"urgent",
  "location"=>"Room 71",
  "status"=>"empty",
  "candy"=>"peppermint patties"
}

serialized = IntegerArray.serialize(data)
# => [3, 71, 1, 0]

parsed = IntegerArray.parse(serialized)
# => {
# => "priority"=>"urgent",
# => "location"=>"Room 71",
# => "status"=>"empty",
# => "candy"=>"peppermint patties"
# => }

other_data = {
  "priority"=>"low",
  "location"=>"Room 23",
  "status"=>"not empty",
  "candy"=>"m&ms"
}

serialized = IntegerArray.serialize(data)
# => [0, 23, 0, 1]

parsed = IntegerArray.parse(serialized)
# => {
# => "priority"=>"low",
# => "location"=>"Room 23",
# => "status"=>"not empty",
# => "candy"=>"m&ms"
# => }

That’s pretty efficient, but since we’re already paying the cost to have custom code for serializing and parsing the data we can go even further! Our data still has some lurking flexibility that we could trade for a decrease in size. The data values themselves!

Push all the values into a bitfield

Consider that status field. We know it can only have two possible values: empty or not empty. But we’re using a data value that could hold many many more numbers than a fixed set of two options. If we know there can be only two then we can limit ourselves to only using 0 or 1 and then we have information that could be stored entirely in a single bit. But how can we store the priorities or all the types of candy or all the possible room locations as bits?

The answer is to use the fewest number of bits that can hold the range of possible options.

Let’s illuminate that with the “priority” field. The priority field has four options.

Priorities
low
medium
high
urgent

We can’t represent those options with a single 0 or 1 bit. But we can represent them all with TWO bits.

Priority	Integer	Bits
low	0	00
medium	1	01
high	2	10
urgent	3	11

It’s the same data, but restricted down to the absolutely smallest and least flexible space.

Now you may be thinking to yourself, “Self, serializing something like 10 to JSON would be twice the data needed to serialize the single digit 2. How can that be a savings?” And you are right on! If we were to represent our bits with one digit per bit in JSON that would be much too large! Instead we’ll push all of our data together into a bitfield and represent that as JSON.

The number of bits you need for a field is enough to represent the power of two large enough to count all the values.

Let’s say we want a bitfield to hold days of the month. There are 31 possible values from 1 to 31. Let’s see how many bits we’d need to hold that data.

Number of Bits	Available Values	Range
1	2	0-1
2	4	0-3
3	8	0-7
4	16	0-15
5	32	0-31
6	64	0-63
7	128	0-127
8	256	0-255

Hey there we go! Using 5 bits gives us a bucket of 32 spaces which is just a little more than we need to hold any of the 31 possible days in a month.

Let’s take a look at the bits the rest of our data fields would require.

Data Field	Possible Values	Bits Required
priority	4	2
location	100	7
status	2	1
candy	5	3

Great! What would it look like to represent our sample data in these bits?

module BitsArray
  def self.serialize(data)
    IntegerArray.serialize(data).map do |value|
      value.to_s(2)
    end
  end

  def self.parse(array)
    integer_array = array.map { |v| v.to_i(2) }
    IntegerArray.parse(integer_array)
  end
end

data = {
  "priority"=>"urgent",
  "location"=>"Room 71",
  "status"=>"empty",
  "candy"=>"peppermint patties"
}

serialized = BitsArray.serialize(data)
# => ["11", "1000111", "1", "0"]

parsed = BitsArray.parse(serialized)
# => {
# => "priority"=>"urgent",
# => "location"=>"Room 71",
# => "status"=>"empty",
# => "candy"=>"peppermint patties"
# => }

Hey it works! In our example code we can see that Ruby has really great support for changing number bases. That’s a fundamental part of what we’re doing, but we’re not done yet. Currently we’ve gotten our data to an array of base-2 numbers instead of array of base-10 numbers. But that’s not a size reduction! Not at all!

serialized_base10 = [3, 71, 1, 0]
serialized_base2 = ["11", "1000111", "1", "0"]

serialized_base10.to_json.size # => 10
serialized_base2.to_json.size # => 24

So how is this going to save us any space? We’re gonna squish those bits! Instead of keeping all of those bits apart we’ll push them right next to each other and treat that as a single number.

Ultimately for this specific example we want to get this binary number:

1110001111000

Made up of these components

Priority	Location	Status	Candy
111111	100011110001111000111	111	000000000

Would that save us space? Yes! Because we can represent that long string of bits as an integer of whatever base we like!

0b1110001111000.to_s(2) # => "1110001111000"
0b1110001111000.to_s(8) # => "16170"
0b1110001111000.to_s(10) # => "7288"
0b1110001111000.to_s(12) # => "4274"
0b1110001111000.to_s(16) # => "1c78"

Sticking with base-10 for simplicity and all of our original data would be represented as 7288. That’s just 4 bytes as JSON! 4.04% of the original 99 bytes!

Constructing a bitfield

It might seem like we could simply take our individual integer values we’ve derived, convert them each to base-2, and then join them together into a string that will be our base-2 bitfield.

As a quick review we’ve gotten our data values into base-10 integers:

data = {
  "priority"=>"urgent",
  "location"=>"Room 71",
  "status"=>"empty",
  "candy"=>"peppermint patties"
}

serialized = IntegerArray.serialize(data)
# => [3, 71, 1, 0]

Let’s convert them individually to base-2

Field	Base-10	Base-2
Priority	333	111111
Location	717171	100011110001111000111
Status	111	111
Candy	000	000

Here you might spot the problem with a simple conversion of each integer to base-2 and joining them into a string. But if you don’t you’re in good company because the problem is not obvious nor intuitive.

Check out that last 0. Our candy field is supposed to have three bits of data. It’s only shown as a single 0 in our results so far because 0b0 is the same as 0b00 and 0b000 just like 000 is 00 and 0: they’re all simply zero! That was fine when our bits were separated into an array, but it’s not going to work at all when our bits are all going to be part of the same number.

Let’s see that problem directly:

oops = ["11", "1000111", "1", "0"].join
p oops # => "11100011110"

# convert to base-10
p oops.to_i(2) # => 1822

bitfield = ["11", "1000111", "1", "000"].join
p bitfield # => "1110001111000"

# convert to base-10
p bitfield.to_i(2) # => 7288

7288 is NOT 1822! Let’s compare what happens when we convert those back to base-2

1822.to_s(2) # => "11100011110"
7288.to_s(2) # => "1110001111000"

That might look like no big deal, it’s just the right side showing up as 0 instead of 000 right? Well yes, but that’s a huge deal! Like the number 23 is very different than the number 2300: those zeros on the right are very important!

# the bitfield is supposed to be 13 bits
# 2 for priority
# + 7 for location
# + 1 for status
# + 3 for candy
1822.to_s(2).ljust(13, '0') # => "1110001111000"
7288.to_s(2).ljust(13, '0') # => "1110001111000"

We can’t simply solve the issue by padding the entire string. Padding the numer with zeros only works in this case because the 0 value happens in the last field. If the 0 were in multi-bit field in the middle of the bitfield then no amount of left or right padding would fix the problem.

We could pad each individual integer to its known bit length and then join the results into a string that we could convert into base-2. But that would be hacking our way around the problem.

We must properly construct our bitfield. That means putting in a little more thought than simply converting each individual field to base-2 and joining them.

Fair warning, I’m going to ramble around the following explanations in various ways. If you feel like you’ve got it, skim along. If you don’t then hopefully one of the explanations helps get you there.

What is a bitfield?

A bitfield is, ultimately, a number. It’s not special, not magical. We make it special by assigning significance to specific divisors that make up the number. In a bitfield’s case we assign significance to the powers of 2 that make up the number.

Why powers of 2? Because that’s what computers are good at. Ultimately every bit is represented as either a 1 or 0. A bit is the smallest component of memory we can manipulate. So if we want our data to be as small as possible that means using the smallest number of bits that could represent the individual pieces of data we want to reference.

Since we want to represent more data than a single bit can hold and since we want to keep all of our data together we have to place all of our bits into a single number. That means every single bit and its position in the number is very important. As a series of numbers 101110 might look similar to 1011100 but its literally the difference between 46 and 92.

As a quick level set on number “places” in base-10 vs base-2.

| Base-10 Place | 1000s | 100s | 10s | 1s |
| Number | 1 | 0 | 0 | 0 |
| Base-2 Place | 8s | 4s | 2s | 1s |

NNN	N10N^{10}N10	N2N^{2}N2
0	1	1
1	10	2
2	100	4
3	1000	8
4	10000	16
……	……	……

Constructing a bitfield by hand

Ok, so let’s construct OUR bitfield.

Field	Bits
priority	2
location	7
status	1
candy	3

We have four fields: priority, location, status, candy. And we know the number of bits we need for each. Using that we can figure out how to turn each of our field values into their position in the bitfield.

To convert each value to its part of our bitfield number we need to determine what units of two it represents. That is: what’s the smallest “place” of its part of the bitfield? It’s a weird concept to explain so let’s do it instead and then circle back to try and put it all together.

We’ll build our bitfield from smallest to largest numbers. That is we’ll create it right to left: starting with the smallest number places of the bitfield and finishing with the largest number places.

First we have candy. Since it’s the first field its unit is 1. But our value (peppermint patties) is also 0 so we end up with a value of 0 going into the bitfield. It’s three bits of zero, but that’s not information we can encode in the zero itself.

Bitfield: (0∗1)(0 * 1)(0∗1).

Next we have status. Our value (empty) is represented as 1 in our design. BUT since candy takes three bits to hold its data our status value of 1 needs to use 8 for its units. The 1s, 2s, and 4s places are all reserved for candy data.

Bitfield: (0∗1)+(1∗8)(0 * 1) + (1 * 8)(0∗1)+(1∗8)

After status is location (Room 71). Its data starts off right next to status since status only takes one bit for its data. That means all of the location data will be in units of 16.

Bitfield: (0∗1)+(1∗8)+(71∗16)(0 * 1) + (1 * 8) + (71 * 16)(0∗1)+(1∗8)+(71∗16)

Finally we have priority. Since location took a whopping seven bits of data priority is pushed way up into units of 2048. Our data is “urgent” which we’ve represented as 3.

Bitfield: (0∗1)+(1∗8)+(71∗16)+(2048∗3)(0 * 1) + (1 * 8) + (71 * 16) + (2048 * 3)(0∗1)+(1∗8)+(71∗16)+(2048∗3)

What does that give us? 7288! Is that what we wanted? Well let’s see!

7288.to_s(2) # => 1110001111000

That’s a 11 for priority, a 1000111 for location, a 1 for status, and a 000 for candy. In base-10 they’re respectively 3, 71, 1, and 0. That checks out! (Don’t worry we’ll parse bitfields for real later on.)

Constructing a bitfield by adding powers of two

That worked, but we need a generalized programmable solution to putting our data into a bitfield. We need an algorithm! The result of our calculation should be the integer number 7288 in whatever base we find convenient to work with: probably base-10.

Our algorithm will need two sets of data:

The bit lengths of each field
The actual values for each field

Using the bit lengths of each field we’ll calculate the unit power of two for each field going from smallest to largest.

Candy, the first field, starts things off by having the unit 20=12^{0} = 120=1.

Next is the status field. Since the candy field took three bits status gets the unit 20+3=23=82^{0 + 3} = 2^{3} = 820+3=23=8.

Then the location field. Just one bit was taken for status so we have 20+3+1=24=162^{0 + 3 + 1} = 2^{4} = 1620+3+1=24=16

Finally the priority field coming in after the seven bits for location: 20+3+1+7=211=20482^{0 + 3 + 1 + 7} = 2^{11} = 204820+3+1+7=211=2048.

Given an ordered set of fields and their bit lengths we could arbitrarily map that data into the bit unit for each field. Great! That means we can construct that data in advance and it’s a simple matter of multiplication and addition to arrive at our bitfield integer.

The process would be:

Precalculate the base-2 units for each field
Receive incoming data to place into the bitfield
Map each data value to its integer representation
Multiply each of those values by its field’s base-2 unit
Add all of those calculated values to arrive at the bitfield integer

SIDE NOTE that’s jumping ahead: We could also use bitwise OR instead adding the values. Since each value is in its own space on the bitfield the end result is the same. If you don’t know what I’m talking about then don’t worry we talk about bitwise operations when we get into how we parse a bitfield.

Here’s a code example of a generic bitfield generator that has no internal knowledge of our fields. That means we have to give it the number values we want to go into the bitfield.

class BitfieldGenerator
  attr_reader :fields

  def initialize
    @fields = []
    @_current_exponent = 0
  end

  def add_field(name, bits)
    field_exponent = @_current_exponent
    @_current_exponent += bits
    @fields << {
      name: name,
      bits: bits,
      base: 2 ** field_exponent
    }
  end

  # operations kept distinct to aide readability
  def serialize(data_values)
    calculated = field_values(data_values)
    # => [0, 8, 1136, 6144]

    calculated.sum
    # => 7288
  end

  def field_values(values)
    values.map.with_index do |value, index|
      value * @fields[index][:base]
    end
  end
end

bf = BitfieldGenerator.new
bf.add_field('candy', 3)
bf.add_field('status', 1)
bf.add_field('location', 7)
bf.add_field('priority', 2)

After adding the fields our @fields data represents all of our bitfield knowledge.

[
  {
    :name=>"candy",
    :bits=>3,
    :base=>1
  },
  {
    :name=>"status",
    :bits=>1,
    :base=>8
  },
  {
    :name=>"location",
    :bits=>7,
    :base=>16
  },
  {
    :name=>"priority",
    :bits=>2,
    :base=>2048
  }
]

We can then generate a bitfield by passing a data values array ordered from smallest to largest field.

bf.serialize([0, 1, 71, 3]) # => 7288

That’s a number!

Constructing a bitfield using bitwise left shift

We have another option available to us when we’re constructing our bitfield: bitwise left shift. That operator allows us to push a value an arbitrary number of bits to the left. Essentially a shorthand for multiplying it by a power of 2. It can make for bitfield generation code that might be easier to follow since we can directly abstract each value’s position in the bitfield.

Bitwise left shift	Base-2	Base-10
`1 << 0`	1	1
`1 << 1`	10	2
`1 << 2`	100	4
`1 << 3`	1000	8
`1 << 4`	10000	16
`1 << 5`	100000	32
`1 << 6`	1000000	64
`1 << 7`	10000000	128

That’s awesome! We can take the number we know we want and then directly say where we want it to be in the resulting bitfield number. Let’s build this bitfield up iteratively. Of course there are also many ways we could nicely wrap it up into a class. For example our earlier class keeping track of the base for each field could instead keep track of the position directly. It’s a bit more intuitive to think about the position of each value vs its starting power of two.

bitfield = 0

candy_value = 0
candy_bits = 3
candy_position = 0
bitfield += (candy_value << candy_position)

status_value = 1
status_bits = 1
status_position = candy_position + candy_bits
bitfield += (status_value << status_position)

location_value = 71
location_bits = 7
location_position = status_position + status_bits
bitfield += (room_value << room_position)

priority_value = 3
priority_bits = 2
priority_position = location_position + location_bits
bitfield += (priority_value << priority_position)

bitfield # => 7288

And, again, we could certainly use bitwise OR instead of adding and we’d get the same result because each value has a separate place in the number.

Let’s call it done for creating a bitfield. Hooray!

Bitfield creation: recap

Have some data
Put the data into a consistent order (e.g. our priority, location, status, candy)
Limit the data to a set of predefined values (e.g. our candy can only be one of a predefined list of candies)
Determine how many bits is required to hold each set of data

At that point you’re ready to make a bitfield!

Convert each piece of data into its predefined value (e.g. peppermint patties converts to 0)
Incorporate it into the bitfield using one of the transformation techniques (e.g. bitshifting each value to its correct position and adding up the results)
Represent the resulting bitfield number in whatever base works well. Base-10 is probably easiest.

Great! Now that we have a bitfield. How do we get the data back out?

Parsing a bitfield

Now we have the opposite problem. We want to take a squished down opaque data representation and expand it back out into consumable data.

Or do we? Next up I’ll show one way of extracting each piece of data out of the bitfield. But then I show how you can use the bitfield as the data source directly. If you don’t need to represent the entire set of data at once then you could do less work by getting the fields directly from the bitfield as needed.

Extracting fields from a bitfield using bitwise right shift

This is the opposite of the bitwise left shift we used to assemble the bitfield. A bitwise right shift slides the binary number to the right: essentially dividing the number by powers of two.

(55).to_s(2) # => "110111"

(55 >> 0).to_s(2) # => "110111"
(55 >> 1).to_s(2) # => "11011"
(55 >> 2).to_s(2) # => "1101"
(55 >> 3).to_s(2) # => "110"
(55 >> 4).to_s(2) #=> "11"
(55 >> 5).to_s(2) #=> "1"
(55 >> 6).to_s(2) #=> "0"

# from there on out there's no more to shift
# any further right shifts result in 0

We can use this to extract the fields from the bitfield.

bitfield = 7288

candy_value = bitfield[0,3] # first three bits
bitfield = bitfield >> 3 # discard the first three bits

status_value = bitfield[0,1] # next bit is status
bitfield = bitfield >> 1 # shift away the status bit

# get the location then discard the bits
location_value = bitfield[0,7]
bitfield = bitfield >> 7

# now all that's left in the bitfield is the priority
# but we'll complete the pattern
priority_value = bitfield[0,2]
bitfield = bitfield >> 2

p [
  candy_value,
  status_value,
  location_value,
  priority_value
] # => [0, 1, 71, 3]

There we go! Then it’s only a matter of mapping those integer fields back to their actual values.

That process essentially splits up the bitfield by field bit lengths:

priority	location	status	candy
11	1000111	1	000

Parsing a bitfield using bitwise AND

Extracting all the fields from a bitfield is fine, but that’s not the only way to use a bitfield. We can also “query” it for values directly! The operator that makes that easy is bitwise AND. Let’s look at how it works.

Sidetrack into bitwise AND

Let’s say we have a perfectly interesting binary number.

00011011

Now let’s say that number is a bitfield like so:

Field 1	Field 2	Field 3
0001	10	11

Very cool. And now let’s say we want to check on Field 2. We don’t need to parse out Field 1 or Field 3. We just need the bits from field 2.

    ∨∨ these two bits right here
00011011
∧∧∧∧ not these four bits
      ∧∧ or these two bits

We can extract those bits out using bitwise AND.

Check it out!

"00011011".to_i(2) & "1100".to_i(2)
=> 8

Wow!

Ok that’s maybe not as exciting as it seems. Let’s step through it. I’m going to ramble around this explanation in the hopes of rounding up everyone to the same page.

The first thing we need to do to use bitwise AND to get information from our bitfield is to represent the bits we want as a number. That is, we need to make a binary number that has 1 where we want information and 0 where we don’t want information.

00011011 - the bitfield
00001100 - the comparison with 1s where we want data

Ruby and most langauges want to work with base-10 numbers and want to show us base-10 numbers. So let’s look at this via base-10.

"00011011".to_i(2) # => 27
"00001100".to_i(2) # => 12

So our bitfield is 27 in base-10. And the position of the bits we’re interested in is 12.

The operation we perform is then 27 bitwise AND 12. In Ruby the bitwise AND operator is &

The result is represented in base-10.

27 & 12 # => 8
(27 & 12).to_s(2) # => "1000"

That means 8 is the base-10 number we’d get if we represented the extracted bits from our field in base-10.

Let’s look at that all in base-2 AND base-10 to make it more clear what’s happening.

| Operation | Base-2 | Base-10 |
|--------------|----------|---------|
| BITFIELD | 00011011 | 27 |
| BITWISE AND | 00001100 | 12 |
| RESULT | 00001000 | 8 |

Zooming in on the field we care about. We know in our bitfield that it’s 10. We hit that 10 with an extraction hammer of 11 to knock out whatever the bits are at that position in our bitfield which is 10. But Bitwise AND can’t simply return “10” because the value of the number is significiant. So we end up with a number according to the following rules.

Bitfield	Comparison	Result
1	1	1
0	1	0
1	0	0
0	0	0

We can think of the comparison number as a hammer of 1 bits that will knock out either the 1 or 0 in its corresponding position in the bitfield and then the result will have 0s everywhere else.

Let’s see this in action by viewing every permutation of our “Field 2”. I’ll leave the rest of the bitfield as it is in our example to make it clear those numbers don’t matter at all. The bitwise AND gives us only the bits we care about.

  00011111 bitfield
& 00001100 comparision
---------------
  00001100 result (12 in base-10)

  00011011 bitfield
& 00001100 comparision
---------------
  00001000 result (8 in base-10)

  00010111 bitfield
& 00001100 comparision
---------------
  00000100 result (4 in base-10)

  00010011 bitfield
& 00001100 comparision
---------------
  00000000 result (0 in base-10)

What does this mean? It means that since we, as the bitfield designers, know that Field 2 has two bits and four possible settings we can inspect them using bitwise AND. A result of 12 will mean that we have 11 in the field, 8 means we have 10, 4 means we have 01, and 0 means we have 00.

The position of the field relative to its neighbors IS significant. To demonstrate why let’s consider Field 3.

  00011011 bitfield
& 00000011 comparision
---------------
  00000011 result (3 in base-10)

  00011010 bitfield
& 00000011 comparision
---------------
  00000010 result (2 in base-10)

  00011001 bitfield
& 00000011 comparision
---------------
  00000001 result (1 in base-10)

  00011000 bitfield
& 00000011 comparision
---------------
  00000000 result (0 in base-10)

All different numbers except for the 0 case because of the position of the field within the larger bitfield.

The key is then that we have to be aware of the possible results per bitfield by both bit length and bit position.

Let’s see how this works in practice by going back to our status messages bitfield.

Back to our candy messages example

To extract individual fields from our bitfield we need four different comparison numbers. Determining each number is simple: fill in a 1 for the bits of the field and leave everything else 0.

value	purpose
111000111100011100011110001110001111000	the bitfield
110000000000011000000000001100000000000	priority
001111111000000111111100000011111110000	location
000000000100000000000010000000000001000	status
000000000011100000000001110000000000111	candy

Let’s look at status and see how its bitwise AND comparison value can extract the actual value of the field while ignoring anything else in the bitfield.

not_empty = 0b1010101011101
empty = 0b1010101010101
status_comparison = 0b0000000001000

empty & status_comparison
# => 8
(empty & status_comparison).to_s(2)
# => "1000"

not_empty & status_comparison
# => 0
(not_empty & status_comparison).to_s(2)
# => "0"

It might not be super obvious there, but those are good results! The 1000 means that there was a 1 in the status field position so it’s included and every other digit in the bitfield is returned as a zero. Any leading zeros aren’t represented just like 000100 is simply 100. The 0 result means the status field was a zero so there’s nothing left in the result other than zero so it’s all represented as zero.

0b0000000001000 == 0b1000
# => true

0b0000000000000 == 0b0
# => true

That’s success! We can precisely extract only the bits that we need to know about using bitwise AND!

To interpret the result we have two options:

Understand the bitwise AND results directly
Bitwise right shift the results to line up with the original values

Looking at that status result for bitwise AND. We could either have our interpreting code understand that the value 8 means the status is “empty” or we can right shift the result the appropriate number of places to extract the value directly.

(empty & status_comparison) >> 3
# => 1

(not_empty & status_comparison) >> 3
# => 0

Let’s see the four possible settings of priority as a second example. This time without converting to base-2.

urgent = 0b1110101010101 # => 7509
high = 0b1010101010101 # => 5461
medium = 0b0110101010101 # => 3143
low = 0b0010101010101 # => 1365
priority = 0b1100000000000 # => 6144

# use these values directly
urgent & priority # => 6144
high & priority # => 4096
medium & priority # => 2048
low & priority # => 0

# or right shift out of the bitfield
(urgent & priority) >> 11 # => 3
(high & priority) >> 11 # => 2
(medium & priority) >> 11 # => 1
(low & priority) >> 11 # => 0

Bitfield interpretation: recap

Have a bitfield
Know the bitfields data fields and their bit positions
Know the predefined values for each field and how they map to the possible bit values per field
Use bitwise AND to extract specific fields from the bitfield and then either work with those results directly or bitwise RIGHT SHIFT the results back to the original values based on the fields position in the bitfield

Yay bitfields!

Bitfields are a somewhat obscure data structure: but they’re out there! Famously the Reddit r/place April Fool’s event was created by using a Redis bitfield.

We used a bitfield of 1 million 4 bit integers. Each 4 bit integer was able to encode a 4 bit color, and the x,y coordinates were determined by the offset (offset = x + 1000y) within the bitfield. We could read the entire board state by reading the entire bitfield. We were able to update individual tiles by updating the value of the bitfield at a specific offset (no need for locking or read/modify/write).

— How we built r/place

If you ever come across an integer that seems to be weirdly treated as data: there’s a good chance that’s a bitfield! If you ever need to squish a predefined and rigorously scoped set of data into the smallest possible space: that could be a bitfield!

Every “if” statement is an object waiting to be extracted

Stephen Ball — Wed, 19 Feb 2020 00:58:01 +0000

The problem

Consider the following Ruby code.

class Beverage
  attr_reader :kind

  def initialize(kind)
    @kind = kind
  end

  def typical_ounces
    if :coffee
      6
    else
      8
    end
  end

  def calories_per_ounce
    if :coffee
      0
    else
      :unknown
    end
  end
end

So we’re modeling beverages. Great! Our class can accept a kind of beverage and then respond to some messages about it.

coffee = Beverage.new(:coffee)
coffee.container # => "mug"
coffee.typical_ounces # => 6
coffee.calories_per_ounce # => 0

oj = Beverage.new(:orange_juice)
oj.container # => "glass"
oj.typical_ounces # => 8
oj.calories_per_ounce # => :unknown

Wonderful. But those if blocks aren’t great right? If we drop more kinds of beverage in that model then they’re going to get annoying pretty quickly. But no problem we can use some case statements to make a nice pattern!

class Beverage
  attr_reader :kind

  def initialize(kind)
    @kind = kind
  end

  def typical_ounces
    case kind
    when :coffee
      4
    when :orange_juice
      8
    else
      8
    end
  end

  def calories_per_ounce
    case kind
    when :coffee
      0
    when :orange_juice
      14
    else
      :unknown
    end
  end
end

Lovely. But there’s a problem here. As we add beverage after beverage the repetition of the conditionals becomes arduous.

class Beverage
  attr_reader :kind

  def initialize(kind)
    @kind = kind
  end

  def typical_ounces
    case kind
    when :coffee
      4
    when :orange_juice
      8
    when :rum
      1.5
    when :whole_milk
      8
    when :skim_milk
      8
    else
      8
    end
  end

  def calories_per_ounce
    case kind
    when :coffee
      0
    when :orange_juice
      14
    when :rum
      64
    when :whole_milk
      19
    when :skim_milk
      12
    else
      :unknown
    end
  end
end

Sure we could allow a lot of the typical_ounces fall through to the default case but then our data structures don’t line up with the calories_per_ounce and future devs won’t know if they really are the default or if we forgot to declare the correct data.

The tests are similarly convoluted and each new beverage is a data slog to sort through and easy to get wrong.

This isn’t fun! Let’s make it fun!

What we have here is a failure to utilize the full power of object oriented design.

“As an OO practitioner, when you see a conditional, the hairs on your neck should stand up. Its very presence ought to offend your sensibilities. You should feel entitled to send messages to objects, and look for a way to write code that allows you to do so.”

Excerpt From: Sandi Metz, Katrina Owen. “99 Bottles of OOP.” Apple Books.

Sandi and Katrina go on to explain that of course not every conditional is bad. The problem is having conditionals controlling the individual pieces of behavior in our class. A better object oriented design would pull together all the like behaviors into their own classes. Then have singular top level conditional that decides which behavior to use.

It’s like having a car repair manual peppered with conditionals. “If you have an Outback then do X, but if you have a Forester then do Y.” Not a great experience! A much nicer alternative is only making a decision about which car model you have once and then using its dedicated manual.

Replacing low level conditionals with objects

We can make our beverage class nicer to work with by not requiring it to be the only class we have. We can make a class for each individual beverage that all answer the same messages that we want to send them.

class Coffee
  def typical_ounces
    6
  end

  def calories_per_ounce
    0
  end
end

class OrangeJuice
  def typical_ounces
    8
  end

  def calories_per_ounce
    14
  end
end

class Rum
  def typical_ounces
    1.5
  end

  def calories_per_ounce
    64
  end
end

class WholeMilk
  def typical_ounces
    8
  end

  def calories_per_ounce
    19
  end
end

class SkimMilk
  def typical_ounces
    8
  end

  def calories_per_ounce
    12
  end
end

Check that out! Those classes are completely focused and minimal. Testing them becomes a simple exercise that they respond to the right messages.

But what happened to Beverage and where are the default values? Let’s take those in reverse order.

The default case

Before the default values fell out of our huge case statements in the final else path where we give up and say “8” for typical ounces and :unknown for calories per ounce. Unknown is not an actual beverage that we directly modeled before, but it’s absolutely behavior that we can extract into a name.

class UnknownBeverage
  def typical_ounces
    8
  end

  def calories_per_ounce
    :unknown
  end
end

Choosing the right behavior

The Beverage concept is still useful. Before we’d initialize it with a kind and then expect the resulting instance to have the correct behavior.

oj = Beverage.new(:orange_juice)
oj.calories_per_ounce # => 14

Here we can either adjust our API or add a little complexity to the Beverage class.

Adjusting the Beverage API

First, let’s see how adjusting our API would look.

In Ruby it’s idiomatic to add a class method called for to choose between different object behaviors e.g. Beverage.for(:coffee). We can call the decision method anything we like. Its job will be to pick the right beverage class so we can use whatever makes sense.

oj = Beverage.for(:orange_juice)
oj = Beverage.given(:orange_juice)
oj = Beverage.from(:orange_juice)
oj = Beverage.build(:orange_juice)

In our example let’s stick with for. Here’s what that top level decision behavior could look like.

class Beverage
  def self.for(kind)
    case kind
    when :coffee
      Coffee.new
    when :orange_juice
      OrangeJuice.new
    when :rum
      Rum.new
    when :whole_milk
      WholeMilk.new
    when :skim_milk
      SkimMilk.new
    else
      UnknownBeverage.new
    end
  end
end

Keeping the original Beverage behavior

Second, let’s see how we could keep the original Beverage.new behavior by making Beverage a little more complex. It can hold the specific beverage class chosen from kind and delegate the calls to the appropriate data class.

class Beverage
  def initialize(kind)
    @dataObject = case kind
                  when :coffee
                    Coffee.new
                  when :orange_juice
                    OrangeJuice.new
                  when :rum
                    Rum.new
                  when :whole_milk
                    WholeMilk.new
                  when :skim_milk
                    SkimMilk.new
                  else
                    UnknownBeverage.new
                  end
  end

  def typical_ounces
    @dataObject.typical_ounces
  end

  def calories_per_ounce
    @dataObject.calories_per_ounce
  end
end

Even better! Ruby has a stdlib for that: SimpleDelegator! With SimpleDelegator we give it the class we want our object to delegate methods calls to. Nice!

require "delegate"

class Beverage < SimpleDelegator
  def initialize(kind)
    dataObject = case kind
                 when :coffee
                   Coffee.new
                 when :orange_juice
                   OrangeJuice.new
                 when :rum
                   Rum.new
                 when :whole_milk
                   WholeMilk.new
                 when :skim_milk
                   SkimMilk.new
                 else
                   UnknownBeverage.new
                 end
    super(dataObject)
  end
end

Conditionals? Managed

Yes in all of these cases there’s still a conditional. But the key difference is that now we have a single top level conditional choosing a behavior rather than multiple low level conditionals choosing raw data. We have little to no chance of introducing a bug in the data due to a mismatch of logic. Before we could have easily introduced bug while trying to keep those large, parallel case statements consistent.

Choose Generic Tools

Stephen Ball — Fri, 03 Jan 2020 01:39:44 +0000

Upstream your tooling instead of rolling your own. The more you push upstream to gems or Rails, the less logic you need in your application. Save your application code for what truly makes your company special (i.e. Pull Requests), instead of tools to make your application run smoothly (i.e. concurrent testing libraries)

— Eileen Uchitelle : Upgrading GitHub from Rails 3.2 to 5.2

Ok so you’re you: programmer person at company place.

You need to do a thing. A small, simple thing. It’s so small and simple that you don’t even look around the tech landscape to see if anyone else has already figured out a way to do the thing. Even if you did you know in your heart that anything you write would be better and faster and absolutely custom fit for your problem domain. Naturally you write the TOOL! It works! Life is great!

But it doesn’t stop there.

You need to do a variation of the thing. Well hey edge cases right? You make the change to let TOOL handle both variants. It works! Life is great! TOOL is no longer quite so simple but that’s the price of progress right?

But it doesn’t stop there.

Someone else at company place needs to do the thing. This is great! TOOL will be so useful for them!

TOOL doesn’t work on their system.

Whoops! Your mistake. You wrote TOOL for your system and its quirks. The quirks are awesome everyone should use them but whatever. You can make TOOL work on more pedestrian systems. You make that change. It works! Life is great!

TOOL gets noticed. More people need to do the thing.

TOOL needs further changes to account for different systems and different use cases. It’s far from the simple thing that it was. But still overall? Right on. Your work is helping everyone that needs to do the thing at company place. How valuable is that? This is definitely a chip you can use when you talk about your next compensation increase. TOOL is now a fundamental part of company place.

TOOL breaks.

Well no, of course TOOL didn’t break. TOOL is awesome. Someone broke it because they didn’t understand it. Who has time to document things? Especially such a simple and straightforward thing as TOOL?

TOOL breaks.

Ok this time TOOL still didn’t break. TOOL is awesome. But one of its dependencies had to be version locked to the (now) previous version to keep it working as TOOL expects. TOOL is still great. Life is great!

You move on. TOOL doesn’t.

You leveraged your TOOL experience and the go-getter reputation it into a better gig at company place. TOOL is now supported by the company developer community at large. That’s good progress!

The nature of the thing to do changes.

Progress indeed. Now company place has different engineering requirements. The thing to do is still essentially the same, but different enough that TOOL needs to be heavily rewritten as its foundational assumptions no longer apply.

It doesn’t work.

The new people working on TOOL can’t get it to twist to fit the new shape without completely rewriting it. But we can’t abandon TOOL! It’s a critical part of the company engineering stack. But it can be rewritten. Here comes TOOL2!

Months pass.

Now there are two stacks. Teams using the original TOOL and teams using the newness of TOOL2. Everyone knows that TOOL2 is the future, but switching from TOOL isn’t free and there’s real work to be done.

New developers continue to join company place.

Not having been slowly boiled in the story of TOOL they are agast at the mess of TOOL vs TOOL2 and the entire situation. Many companies need to do the thing and they’ve all solved it in various ways. Some intrepid teams have even produced generic solutions as either a revenue stream for paid solutions or as a marketing strategy for free solutions.

New developers question the TOOL ecosystem.

Some few developers survey the technical landscape to find generic tools that could fit the need that TOOL provides. A brave few publically propose replacing TOOL with a generic piece of software. They are quickly shot down by the collective understanding that TOOL has essential company domain knowledge that cannot be replaced by anything generic. Anyone proposing any generic tools is looked at askew. Are they really good company place programmers if they don’t see the value that only TOOL can bring?

New developers learn not to question the TOOL ecosystem.

Thats bad for innovation. Bad for team building. Bad for everyone.

There’s a problem with in-house tooling. Yes it fits the problem domain perfectly but on the downside it fits the problem domain perfectly. If anything about the problem domain changes then the tool needs to be abandoned or changed.

Choose generic tools. When you choose generic tools you get several benefits right away:

New developers can come in with experience using those tools already. Instead of being a distraction from the in-house tooling their experience becomes a valuable asset to enrich the entire experience.
The tools are searchable. This is a huge benefit! In house tools are unsearchable technology which are only fully documented in critical people’s minds. But generic tools have examples and documentation all over the Internet.
The tools are alive. As the world changes the tools change as well. Instead of devoting more and more time to working on the tools themselves developers can focus on the actual domain problems.
The tools have a community. Company developers will be able to answer questions about the tools and build the company’s engineering reputation. If the tools are open source then company developers will be able to contribute back to the generic tools and build the company’s engineering reputation.

There’s a timeline of cost for in-house tools vs generic tools.

In-house

Write a specific, simple tool for a specific, simple thing: cheap! If it’s really a simple and specific thing then it’s a one-time cost of some dev time.
Generalize that simple tool to meet new or additional needs: expensive! When you generalize the simple in-house tool you will now unwittingly make all the mistakes that version 0.x of a generic tool made. You also commit all developers to one or both of maintaining and learning the in-house tool.
Ongoing costs: high! Developer time needs to be devoted to the tool itself vs the actual company domain problems. Any problems with the tools must involve other company developers because there’s no one else in the world that can know anything about the in-house tool.

Generic

Pick a generic tool: cheap! Even paying for a tool is cheap when we’re talking about developer time.
Apply the generic tool to the specific problem: cheap! Developers will need to learn the tool and how to apply it. But you avoid the distraction and expense of writing and maintaining the tool.
Ongoing costs: low! Developer time is maximized to work on the actual company domain problems. Any problems with the tools can start with Internet searches and may not even need other company developers.

Running bundle install on rails master

Stephen Ball — Sun, 01 Dec 2019 03:24:11 +0000

“Ok! Let’s check out this Rails 6 goodness”, I thought.

Clone rails/rails — easy
Look around the latest commits — a great way to see what’s been going on
bundle install :-(

Error 1: Failed to install libxml-ruby 3.10 gem

Installing libxml-ruby 3.1.0 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/libxml-ruby-3.1.0/ext/libxml
/Users/sdball/.rbenv/versions/2.6.5/bin/ruby -I /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/2.6.0 -r
./siteconf20191130-15096-mz0ci9.rb extconf.rb
/Users/sdball/.rbenv/versions/2.6.5/bin/ruby: warning: shebang line ending with \r may cause problems
checking for libxml/xmlversion.h in
/opt/include/libxml2,/opt/local/include/libxml2,/usr/local/include/libxml2,/usr/include/libxml2... no
***extconf.rb failed***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --without-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/Users/sdball/.rbenv/versions/2.6.5/bin/$(RUBY_BASE_NAME)
    --with-xml2-config
    --without-xml2-config
    --with-xml2-dir
    --without-xml2-dir
    --with-xml2-include
    --without-xml2-include=${xml2-dir}/include
    --with-xml2-lib
    --without-xml2-lib=${xml2-dir}/lib
 extconf failure: need libxml2.

    Install the library or try one of the following options to extconf.rb:

      --with-xml2-config=/path/to/xml2-config
      --with-xml2-dir=/path/to/libxml2
      --with-xml2-lib=/path/to/libxml2/lib
      --with-xml2-include=/path/to/libxml2/include

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/extensions/x86_64-darwin-18/2.6.0/libxml-ruby-3.1.0/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/libxml-ruby-3.1.0 for inspection.
Results logged to
/Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/extensions/x86_64-darwin-18/2.6.0/libxml-ruby-3.1.0/gem_make.out

An error occurred while installing libxml-ruby (3.1.0), and Bundler cannot continue.
Make sure that `gem install libxml-ruby -v '3.1.0' --source 'https://rubygems.org/'` succeeds before bundling.

In Gemfile:
  libxml-ruby

Ok this isn’t too weird. There must be some libxml library that I need to install first so the rubygem has the files it needs. And check it out: extconf failure: need libxml2

So let’s fix that!

$ brew install libxml2
==> Downloading https://homebrew.bintray.com/bottles/libxml2-2.9.10.mojave.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/47/472ed1a73a91c49fd9f39bd8cc4a7472b09c691659b3b9305c9da42ed35e1475? __gda__ =exp=15
######################################################################## 100.0%
==> Pouring libxml2-2.9.10.mojave.bottle.tar.gz
==> Caveats
libxml2 is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have libxml2 first in your PATH run:
  echo 'export PATH="/usr/local/opt/libxml2/bin:$PATH"' >> ~/.zshrc

For compilers to find libxml2 you may need to set:
  export LDFLAGS="-L/usr/local/opt/libxml2/lib"
  export CPPFLAGS="-I/usr/local/opt/libxml2/include"

For pkg-config to find libxml2 you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/libxml2/lib/pkgconfig"

==> Summary
🍺 /usr/local/Cellar/libxml2/2.9.10: 280 files, 10.5MB

Great!

Fetching libxml-ruby 3.1.0
Installing libxml-ruby 3.1.0 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/libxml-ruby-3.1.0/ext/libxml
/Users/sdball/.rbenv/versions/2.6.5/bin/ruby -I /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/2.6.0 -r
./siteconf20191130-53906-d0hyry.rb extconf.rb
/Users/sdball/.rbenv/versions/2.6.5/bin/ruby: warning: shebang line ending with \r may cause problems
checking for libxml/xmlversion.h in
/opt/include/libxml2,/opt/local/include/libxml2,/usr/local/include/libxml2,/usr/include/libxml2... no
***extconf.rb failed***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --without-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/Users/sdball/.rbenv/versions/2.6.5/bin/$(RUBY_BASE_NAME)
    --with-xml2-config
    --without-xml2-config
    --with-xml2-dir
    --without-xml2-dir
    --with-xml2-include
    --without-xml2-include=${xml2-dir}/include
    --with-xml2-lib
    --without-xml2-lib=${xml2-dir}/lib
 extconf failure: need libxml2.

    Install the library or try one of the following options to extconf.rb:

      --with-xml2-config=/path/to/xml2-config
      --with-xml2-dir=/path/to/libxml2
      --with-xml2-lib=/path/to/libxml2/lib
      --with-xml2-include=/path/to/libxml2/include

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/extensions/x86_64-darwin-18/2.6.0/libxml-ruby-3.1.0/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/libxml-ruby-3.1.0 for inspection.
Results logged to
/Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/extensions/x86_64-darwin-18/2.6.0/libxml-ruby-3.1.0/gem_make.out

An error occurred while installing libxml-ruby (3.1.0), and Bundler cannot continue.
Make sure that `gem install libxml-ruby -v '3.1.0' --source 'https://rubygems.org/'` succeeds before bundling.

In Gemfile:
  libxml-ruby

Aww c’mon. I guess simply installing the library isn’t enough. I should actually read those homebrew install details.

If you need to have libxml2 first in your PATH run:
  echo 'export PATH="/usr/local/opt/libxml2/bin:$PATH"' >> ~/.zshrc

For compilers to find libxml2 you may need to set:
  export LDFLAGS="-L/usr/local/opt/libxml2/lib"
  export CPPFLAGS="-I/usr/local/opt/libxml2/include"

For pkg-config to find libxml2 you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/libxml2/lib/pkgconfig"

Ok, cool. Coooool. I guess I should set one of those flags.

$ gem install libxml-ruby -v '3.1.0' --source 'https://rubygems.org/' --with-xml-dir=/usr/local/opt/libxml2
ERROR: While executing gem ... (OptionParser::InvalidOption)
    invalid option: --with-xml-dir=/usr/local/opt/libxml2

Uh hm. I say I should set one of those flags.

gem install libxml-ruby -v '3.1.0' --source 'https://rubygems.org/' --with-xml-dir "/usr/local/opt/libxml2"
ERROR: While executing gem ... (OptionParser::InvalidOption)
    invalid option: --with-xml-dir

Uh.

$ gem install libxml-ruby -v '3.1.0' --source 'https://rubygems.org/' --with-xml-dir /usr/local/opt/libxml2
ERROR: While executing gem ... (OptionParser::InvalidOption)
    invalid option: --with-xml-dir

Ok. A quick DuckDuckGo and yeah, we need to separate our options with -- to indicate that we’re passing separate options.

$ gem install libxml-ruby -v '3.1.0' --source 'https://rubygems.org/' -- --with-xml-dir=/usr/local/opt/libxml2
Building native extensions with: '--with-xml-dir=/usr/local/opt/libxml2'
This could take a while...
ERROR: Error installing libxml-ruby:
    ERROR: Failed to build gem native extension.

    current directory: /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/libxml-ruby-3.1.0/ext/libxml
/Users/sdball/.rbenv/versions/2.6.5/bin/ruby -I /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/2.6.0 -r ./siteconf20191130-58777-73jhsk.rb extconf.rb --with-xml-dir\=/usr/local/opt/libxml2
/Users/sdball/.rbenv/versions/2.6.5/bin/ruby: warning: shebang line ending with \r may cause problems
checking for libxml/xmlversion.h in /opt/include/libxml2,/opt/local/include/libxml2,/usr/local/include/libxml2,/usr/include/libxml2... no
***extconf.rb failed***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.

Well that’s not what I was hoping for.

I tried various combinations of paths and options but to no avail.

Web searches aren’t working to provide the details for this weird install option declaration. Time for my secret weapon: GitHub code searches!

Searching GitHub code for --with-xml2-dir

GitHub code search results for –with-xml2-dir

There we go! Someone’s already figured some magical declarations for me! Specifically manuelvanrijn with his nicely organized dotfiles repo containing a bundler config with the declarations I want.

--------
BUNDLE_BUILD__NOKOGIRI: "--with-xml2-dir=/usr/local/opt/libxml2/include/libxml2"
BUNDLE_GEM__COC: "true"
BUNDLE_GEM__MIT: "true"
BUNDLE_GEM__TEST: "rspec"
BUNDLE_BUILD__EVENTMACHINE: "--with-cppflags=-I/usr/local/opt/openssl/include"
BUNDLE_JOBS: "11"
BUNDLE_IGNORE_MESSAGES__CAPISTRANO-HARROW: "true"
BUNDLE_IGNORE_MESSAGES__CAPISTRANO: "true"
BUNDLE_BUILD__MYSQL: "--with-mysql-config=/usr/local/Cellar/mysql@5.7/5.7.25/bin/mysql_config --with-ldflags=-L/usr/local/opt/openssl/lib --with-cppflags=-I/usr/local/opt/openssl/include"
BUNDLE_BUILD__LIBXML-RUBY: "--with-xml2-config=/usr/local/opt/libxml2/bin/xml2-config --with-xml2-dir=/usr/local/opt/libxml2 --with-xml2-lib=/usr/local/opt/libxml2/lib --with-xml2-include=/usr/local/opt/libxml2/include"

Trying out that first approach with a .bundle/config and woo!

$ cat .bundle/config
BUNDLE_BUILD__LIBXML-RUBY: "--with-xml2-config=/usr/local/opt/libxml2/bin/xml2-config --with-xml2-dir=/usr/local/opt/libxml2 --with-xml2-lib=/usr/local/opt/libxml2/lib --with-xml2-include=/usr/local/opt/libxml2/include"

$ gem install libxml-ruby -v '3.1.0' --source 'https://rubygems.org/' -- --with-xml2-config=/usr/local/opt/libxml2/bin/xml2-config --with-xml2-dir=/usr/local/opt/libxml2 --with-xml2-lib=/usr/local/opt/libxml2/lib --with-xml2-include=/usr/local/opt/libxml2/include
Building native extensions with: '--with-xml2-config=/usr/local/opt/libxml2/bin/xml2-config --with-xml2-dir=/usr/local/opt/libxml2 --with-xml2-lib=/usr/local/opt/libxml2/lib --with-xml2-include=/usr/local/opt/libxml2/include'
This could take a while...
Successfully installed libxml-ruby-3.1.0

Error 2: Failed to install mysql gem

An error occurred while installing mysql2 (0.5.2), and Bundler cannot continue.
Make sure that `gem install mysql2 -v '0.5.2' --source 'https://rubygems.org/'` succeeds before bundling.

In Gemfile:
  mysql2

So again we probably need some more header files to build a gem.

$ brew install mysql

$ gem install mysql2
Building native extensions. This could take a while...
ERROR: Error installing mysql2:
    ERROR: Failed to build gem native extension.

    current directory: /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/mysql2-0.5.3/ext/mysql2
/Users/sdball/.rbenv/versions/2.6.5/bin/ruby -I /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/2.6.0 -r ./siteconf20191130-87797-zmrpf5.rb extconf.rb
checking for rb_absint_size()... yes
checking for rb_absint_singlebit_p()... yes
checking for rb_wait_for_single_fd()... yes
----------
Using mysql_config at /usr/local/bin/mysql_config
----------
checking for mysql.h... yes
checking for errmsg.h... yes
checking for SSL_MODE_DISABLED in mysql.h... yes
checking for SSL_MODE_PREFERRED in mysql.h... yes
checking for SSL_MODE_REQUIRED in mysql.h... yes
checking for SSL_MODE_VERIFY_CA in mysql.h... yes
checking for SSL_MODE_VERIFY_IDENTITY in mysql.h... yes
checking for MYSQL.net.vio in mysql.h... yes
checking for MYSQL.net.pvio in mysql.h... no
checking for MYSQL_ENABLE_CLEARTEXT_PLUGIN in mysql.h... yes
checking for SERVER_QUERY_NO_GOOD_INDEX_USED in mysql.h... yes
checking for SERVER_QUERY_NO_INDEX_USED in mysql.h... yes
checking for SERVER_QUERY_WAS_SLOW in mysql.h... yes
checking for MYSQL_OPTION_MULTI_STATEMENTS_ON in mysql.h... yes
checking for MYSQL_OPTION_MULTI_STATEMENTS_OFF in mysql.h... yes
checking for my_bool in mysql.h... no
----------
Don't know how to set rpath on your system, if MySQL libraries are not in path mysql2 may not load
----------
----------
Setting libpath to /usr/local/Cellar/mysql/8.0.18_1/lib
----------
creating Makefile

current directory: /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/mysql2-0.5.3/ext/mysql2
make "DESTDIR=" clean

current directory: /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/mysql2-0.5.3/ext/mysql2
make "DESTDIR="
compiling client.c
compiling infile.c
compiling mysql2_ext.c
compiling result.c
compiling statement.c
linking shared-object mysql2/mysql2.bundle
ld: library not found for -lssl
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [mysql2.bundle] Error 1

make failed, exit code 2

Gem files will remain installed in /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/mysql2-0.5.3 for inspection.
Results logged to /Users/sdball/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/extensions/x86_64-darwin-18/2.6.0/mysql2-0.5.3/gem_make.out

Yeah that’s not SUPER great!

Luckily Manuel is the best and also has a mysql declaration I can try.

BUNDLE_BUILD__MYSQL: "--with-mysql-config=/usr/local/Cellar/mysql@5.7/5.7.25/bin/mysql_config --with-ldflags=-L/usr/local/opt/openssl/lib --with-cppflags=-I/usr/local/opt/openssl/include"

But no, we still run into the key error:

ld: library not found for -lssl

This time a web search found a potential answer: DuckDuckGo web search for “ld: library not found for -lssl mysql gem” [

ld: library not found for -lssl while installing mysql2 gem

Let’s try out the suggested fix. MORE declarations to ensure that the mysql2 gem picks up /usr/local/lib/openssl.

$ gem install mysql2 -v '0.5.2' -- --with-cflags=\"-I/usr/local/opt/openssl/include\" --with-ldflags=\"-L/usr/local/opt/openssl/lib\"
Building native extensions with: '--with-cflags="-I/usr/local/opt/openssl/include" --with-ldflags="-L/usr/local/opt/openssl/lib"'
This could take a while...
Successfully installed mysql2-0.5.2
Parsing documentation for mysql2-0.5.2
Installing ri documentation for mysql2-0.5.2
Done installing documentation for mysql2 after 0 seconds
1 gem installed

So neat, let’s fix that bundler config. (And belatedly realize the original config was for mysql and not mysql2.)

BUNDLE_BUILD__MYSQL2: "--with-cflags=\"-I/usr/local/opt/openssl/include\" --with-ldflags=\"-L/usr/local/opt/openssl/lib\""

But nooo! For whatever reason I simply could not get the bundler config to affect my gem install mysql2 outcome.

So ok fine. We can gem install and then bundler will see that the gem is already available.

$ gem install mysql2 -v '0.5.2' -- --with-cflags=\"-I/usr/local/opt/openssl/include\" --with-ldflags=\"-L/usr/local/opt/openssl/lib\"

Then one very long bundle install later and:

Bundle complete! 67 Gemfile dependencies, 192 gems now installed.

Hooray!

Use tldr for command line examples

Stephen Ball — Wed, 20 Nov 2019 01:06:00 +0000

manual pages are a reference

If you use the command line you’ve probably already seen manual pages, accessed via the man command.

$ man git-log

NAME
git-log - Show commit logs

SYNOPSIS
git log [<options>] [<revision range>] [[--] <path>...]

DESCRIPTION
Shows the commit logs.

The command takes options applicable to the git rev-list command to
control what is shown and how, and options applicable to the git diff-*
commands to control how the changes each commit introduces are shown.

OPTIONS
--follow
Continue listing the history of a file beyond renames (works only for a single file).

--no-decorate, --decorate[=short|full|auto|no]
.
.
.

man pages are great when you need a full reference for a command. “What’s the git-log flag to only show merge commits? (--merges)

But what if you want examples of common use? For that man pages aren’t so helpful. Sometimes you’ll get examples throughout the doc or maybe at the end. Or maybe you can piece together the command patterns from the synopsis.

SYNOPSIS
git commit [-a | --interactive | --patch] [-s] [-v] [-u<mode>] [--amend]
           [--dry-run] [(-c | -C | --fixup | --squash) <commit>]
           [-F <file> | -m <msg>] [--reset-author] [--allow-empty]
           [--allow-empty-message] [--no-verify] [-e] [--author=<author>]
           [--date=<date>] [--cleanup=<mode>] [--[no-]status]
           [-i | -o] [-S[<keyid>]] [--] [<file>...]

There’s a better way?

There IS a better way! TLDR pages!

TLDR pages are community driven reference examples accessed by a tldr command made available by a wide range of clients. From brew install tldr to npm install -g tldr and many others: you can probably find a tldr client implementation you can live with. There’s even a generated PDF of tldr pages if you must have it.

What do we get from `tldr`?

Examples!

$ tldr git log

git log

Show a history of commits.
More information: <https://git-scm.com/docs/git-log>.

- Show the sequence of commits starting from the current one, in reverse chronological order:
    git log

- Show the history of a particular file or directory, including differences:
    git log -p path/to/file_or_directory

- Show only the first line of each commit message:
    git log --oneline

- Show an overview of which file(s) changed in each commit:
    git log --stat

- Show a graph of commits in the current branch:
    git log --graph

- Show a graph of all commits, tags and branches in the entire repo:
    git log --oneline --decorate --all --graph

- Show only commits whose messages include a given string (case-insensitively):
    git log -i --grep search_string

Way awesome! Nice, clear examples of how to commonly use the git-log command.

$ tldr tar

tar

Archiving utility.
Often combined with a compression method, such as gzip or bzip.
More information: <https://www.gnu.org/software/tar>.

- Create an archive from files:
    tar cf target.tar file1 file2 file3

- Create a gzipped archive:
    tar czf target.tar.gz file1 file2 file3

- Extract a (compressed) archive into the current directory:
    tar xf source.tar[.gz|.bz2|.xz]

- Extract an archive into a target directory:
    tar xf source.tar -C directory

- Create a compressed archive, using archive suffix to determine the compression program:
    tar caf target.tar.xz file1 file2 file3

- List the contents of a tar file:
    tar tvf source.tar

- Extract files matching a pattern:
    tar xf source.tar --wildcards "*.html"

Yes!

But what about different platforms?

I hear you. There are lots of commands that are available to Linux and not MacOS and vice versa. Not to mention SunOS.

tldr is there for you. You can pass a -p or --platform flag to tldr to tell it the platform docs it should check instead of defaulting to the platform you’re currently using.

If I run tldr service on macOS then I only get back a “page doesn’t exist” message because service is not a command on macOS. But it is a command on Linux! So if I want to lookup the Linux service command from a macOS machine I pass -p linux to the tldr command.

$ tldr service
This page doesn't exist yet!
Submit new pages here: https://github.com/tldr-pages/tldr

$ tldr -p linux service

service

Manage services by running init scripts.
The full script path should be omitted (/etc/init.d/ is assumed).

- Start/Stop/Restart/Reload service (start/stop should always be available):
    service init_script start|stop|restart|reload

- Do a full restart (runs script twice with start and stop):
    service init_script --full-restart

- Show the current status of a service:
    service init_script status

- List the status of all services:
    service --status-all

Pages that don’t exist yet

If you find there’s a command you know about that doesn’t yet have a tldr page then you can submit a PR to the project with the examples. For example how that tldr service on macOS helpfully shared with me where I could submit a PR.

You can check out all the newly proposed tldr commands since they’re all nicely tagged with a new-command label.

DEV Community: Stephen Ball

From awk to a Dockerized Ruby Script

A Real Ruby Script

Homebrew

Docker

GitHub Actions

That Docker image is way too huge

Features on features

Finding leap years with the cal command

Introduction to cal

Recognizing leap years, hackishly

Recognizing leap years, even more hackishly

Which is faster? Hyperfine will tell us

Wrap up

Next steps

The Problem of State

Pure Functions

Adding State

The Problem of State

Elixir and state

More reading

Clojure Functions in Four Ways

defn

def and fn

Function literals

Using partial

See Some Clojure

Basics

Conditionals

Variables

Functions

Common Data Structures

Vectors

Lists

Commas?

Maps

Documentation

Wrapping Up

A simple language spec isn’t a feature when you’re building applications

A little copying

A lot of places

Go is for systems not applications

The Fastest Possible Tests

Why care about fast tests?

How fast should they go?

How is that possible?

How fast is the fastest possible test? Theoretically?

Shrink your data into bitfields (and out again)

Give up the keys

Limit the possible values

Push all the values into a bitfield

Constructing a bitfield

What is a bitfield?

Constructing a bitfield by hand

Constructing a bitfield by adding powers of two

Constructing a bitfield using bitwise left shift

Bitfield creation: recap

Parsing a bitfield

Extracting fields from a bitfield using bitwise right shift

Parsing a bitfield using bitwise AND

Sidetrack into bitwise AND

Back to our candy messages example

Bitfield interpretation: recap

Yay bitfields!

Further reading

Every “if” statement is an object waiting to be extracted

The problem

Replacing low level conditionals with objects

The default case

Choosing the right behavior

Adjusting the Beverage API

Keeping the original Beverage behavior

Conditionals? Managed

Choose Generic Tools

In-house

Generic

Running bundle install on rails master

Error 1: Failed to install libxml-ruby 3.10 gem

Error 2: Failed to install mysql gem

Use tldr for command line examples

What do we get from `tldr`?