DEV Community: Roberto Dip

Weekly Command: comparing files line by line with diff

Roberto Dip — Tue, 29 May 2018 18:33:26 +0000

The basics

diff [option] [files]

diff outputs the differences between two files, line by line. For files that are identical it produces no output, for binary files only reports if they are different or not.

The set of differences produced by diff is often called a diff or patch, and this output can be used later on by the patch command to change other files.

Play by play

Before digging into the examples, let's define two files that will help us to illustrate some concepts.

To keep the conceptual overhead as low as possible, the files are with-text.js and without-text.js to represent if the text variable is present or not.

File with-text.js:

function example (text) {
  console.log("====")
  console.log(text)
}

File without-text.js:

function something () {
  console.log("====")
}

Default output

By default, diff outputs a list of all the lines that are different between the files, one after the other, alongside with useful information about what changed. A change can be:

An addition, represented by the letter a.
A deletion, represented by the letter d.
A change, represented by the letter c.

For every line in which diff finds a change, it outputs:

What type of change is (a/c/d)
What is the line number affected in each file
The content of the lines affected

In a more concrete example, if you run:

$ diff with-text.js without-text.js

1c1
< function example (text) {
---
> function something () {
3d2
<   console.log(text)

You get:

1c1 indicating that there's a change in line 1 of both files.
< function example (text) { indicating how the line looks in with-text.js.
> function something () { indicating how the line looks in without-text.js.

Are you able to deduce what the rest of the lines are expressing in the output?

Side by side comparison

Apart from the default output format, you can specify the --side-by-side (short -y) to produce a side-by-side view of what changed.

This format is more visual and can be easier to understand at first sight, for example:

$ diff with-text.js without-text.js --side-by-side

function example (text) {             | function something () {
  console.log("====")                     console.log("====")
  console.log(text)                   <
}                                       }

Working with directories

A very powerful feature of diff is the ability to work with directories. It's important to note that actual contents of directories are never compared as if they were a file, instead diff uses the following rules:

If one file is a directory and the other is not, diff compares the file in the directory whose name is that of the non-directory.
If two file names are given and both are directories, diff compares corresponding files in both directories, in alphabetical order.

Colorised output

Since version 3.4 (launched around August of 2016), diff supports the --color flag print colorised output in the terminal.

Reading standard input

If you provide - as a file name, diff will use it for text read from the standard input. As a special case, diff - - compares a copy of standard input to itself.

Omitting differences

diff also provides ways to suppress differences between files that may not be important to you, common examples of this are changes in the amount of white space between words or lines.

Flag	Description
-i --ignore-case	Ignore case differences in file contents.
--ignore-file-name-case	Ignore case when comparing file names.
--no-ignore-file-name-case	Consider case when comparing file names.
-E --ignore-tab-expansion	Ignore changes due to tab expansion.
-b --ignore-space-change	Ignore changes in the amount of white space.
-w --ignore-all-space	Ignore all white space.
-B --ignore-blank-lines	Ignore changes whose lines are all blank.
-I RE --ignore-matching-lines=RE	Ignore changes whose lines all match RE.

Using the right tool for the job

While diff awesome, it's not always the right tool for the job, here are some better fits for different scenarios:

The diff3 command can be useful to show differences among three files.
For binary files, cmp reports the differences between two files byte by byte, instead of line by line.
If you are looking only for visual diffing without the patching, vim -d and git diff do a good job.

Weekly Command: inspecting network usage with iftop

Roberto Dip — Mon, 21 May 2018 19:42:55 +0000

If you need a tool to inspect your network usage, without having to wire a proxy or install a more complex tool like Wireshark, iftop may be what are you looking for.

The basics

iftop | [-nNpblBP] [-i interface] [-f filter] [-F net/mask] [-G net6/mask6]

When running, iftop uses the whole screen to display network usage. At the top there’s is a logarithmic scale which gives a visual indication of traffic, and at the bottom a summary with information of all packages captured.

By default, the program shows for each pair of hosts, the rate at which data has been sent and received over the preceding 2, 10 and 40-second intervals. The arrows (<= and =>) indicates the direction of data flow. For example:

foo.example.com => bar.example.com 1Kb 500b 100b
                 <= 2Mb 2Mb 2Mb

If invoked without arguments, iftop displays information for the first interface it can find which looks like an external interface, most of the time you may want to provide a specific interface with the -i flag.

note: to find interfaces, you can use ifconfig in *nix systems, or networksetup -listallhardwareports in macOS.

The system needs special privileges to execute iftop, which in most cases implies running it as a root.

The program is very versatile, you can filter, sort, and even use it in text-mode if you don’t need the interactive interface.

Examples

Screen filters

l allows you to enter a POSIX extended regular expression that will be used to filter hostnames shown in the display.

It’s important to note that this happens at a much later stage than filter codes and does not affect what is actually captured, therefore, display filters DO NOT affect the totals at the bottom of the screen.

Filter codes

Screen filters are good to do a quick, visual filter, but if you only need to track packets transmitted under certain conditions, you can use a filter code to select the packets to count.

iftop uses pcap formatted filters as input, and you can provide them with the -f flag, or in the app by pressing f. As a general rule, I will always recommend using the flag to prevent unexpected behavior.

note: because only IP packets are ever counted, the specified code is evaluated as “(filter code) and ip ”.

Sorting

You can also use a set of pre-defined sort methods to order the data presented in the view by pressing one of these keys:

1/2/3 - sort by 1st/2nd/3rd column (2, 10 and 40-second intervals)
< - sort by source name
> - sort by dest name
o - freeze current order

Text mode

You can enable text-mode with the -t flag, in text-mode the output is printed to STDOUT in intervals.

Weekly Command: managing processes with htop

Roberto Dip — Thu, 10 May 2018 13:08:43 +0000

The basics

htop [-dChustv]

htop is an interactive command-line process manager that allows you to visualize, sort and manage processes running in your system.

As with many other great command-line utilities, htop is built with ncurses, so the interface may look familiar to you.

You can run the command without any flags and start interacting with your running processes immediately.

fun fact: the community has a list of TV shows and films where htop has appeared.

Examples

htop is very well designed, and your great ally is the help bar at the bottom, which makes the program very intuitive to use, nonetheless here are some examples.

Sending different signals

While the bottom-bar menu presents an option to kill the selected process with F9, you can send any signal to them by pressing the k key and selecting the desired signal.

Tagging multiple processes

Multiple processes can be marked and then signaled at the same time by using the tagging functionality. You can tag a process with the spacebar ␣ or a process with all its children with c.

To untag all processes at once you can use U.

Searching and filtering

You can search and filter processes incrementally by name by using F3 or / to search and F4 or \ to filter.

Leveraging arguments

Besides all the goodness of the user interface, you can also manage some of the features of htop when launching it via custom flags, here are some of my favorites:

--no-color starts the program in monochrome mode, personally, I find it very pleasant to use.
--pid=PID1,PID2... only shows the given comma-separated list of PIDs.
--tree shows the processes in the tree view.

Configuring

htop provides a nice interface to configure the UI: from the meters at the top to the number of columns displayed for every process, almost everything can be changed. This interface lives in the setup menu, which can be opened with F2, once there you will be presented with several menus to configure.

Your configurations are stored in ~/.config/htop/htoprc by default, and this file is automatically managed by htop so you shouldn’t have to manually change it. The path to the config file can be set with the $HTOPRC environment variable, allowing you to have different config files for different scenarios.

Getting better at team communication

Roberto Dip — Mon, 30 Apr 2018 16:51:10 +0000

Communication is frequently mentioned as one of the fundamental skills of a good software developer, but finding simple and actionable ways to get better it’s hard.

A couple of months ago I came into the realization that I needed to improve my communication skills, which was exacerbated by the fact that I work in a remote team. This is my intent to write down what gave me good results over the last six months; the conclusions I came after making an active effort to change my communication habits. Please take them with a grain of salt, this a work in progress and nothing is backed with more data than my own—probably very biased—experience.

For the sake of organizing things, I’ve grouped the bullets into three categories: the other, the group and the self.

The other

The other alludes to 1:1 communication with other teammates, it’s partially different from group communications because the dynamics and the constraints are different, but almost everything in this section can be extrapolated to the group section.

It may sound cliché, but communication is a two-way street, don’t expect others to hear you if you don’t pay attention them.
Always make sure you understand 100% of what the other is trying to manifest, and ask until it’s cristal clear to you. Making “progress” based on wrong assumptions wastes way more time and adds frustration to both parts.
Try to find the balance between not asking questions and asking too much. It’s easier said than done, but at the end of the day is a muscle that has to be exercised.
If you are having a 1:1 conversation with somebody that needs your help, try to stay in the convo until the issue is sorted, async communication is great, but if you can unblock your teammate by giving exclusive attention to the chat for 10 minutes it’s worth it.

The group

The group alludes to 1:many communication, this includes chat groups, mails with several recipients, pull requests, issue trackers, etc.

If you are using pull requests or a similar way to deal with team collaboration, treat them as documentation, put as much info as you can so other members in the team can read and understand what is going on.
If you are not posting in groups of people because you don’t want to disturb the whole team, think about that twice: you don’t need to mention @everyone every time you post, if you are working on something and you think that could be beneficial for other people of the team to know about that go ahead and post.
If you find anything interesting, whether is a tool or something related to the product that you are building, don’t keep it for yourself.
Tools don’t matter that much: don’t be attached to them, communication is always possible when goodwill is present.

The self

While the previous two sections are clearly self-actionable, I think there’s room for a third group: the self, here are some points that make your personal communication skills that are worth paying attention:

Commit yourself to make an active effort to communicate better, communicating is hard.
Take the time to re-read a long message before sending it, sometimes a brain dump is not clear for people that are not in your head!
Try to be clean in regards to formatting, it helps to process the information.
Find ways to express yourself and your work better, an image, a GIF or a video can show in a few seconds what could take a lot of text.

Final words

Communication is a complex and difficult topic, I plan to expand and revisit this list as the time goes on. If you have any tips or resources, please ping me at @roperzh!

Weekly Command: Fuzzy finding everything with fzf

Roberto Dip — Thu, 26 Apr 2018 14:08:35 +0000

The Basics

fzf [--options]

fzf is an interactive fuzzy finder, it reads a list of items from STDIN, and writes the selected item to STDOUT.

If no input is provided via STDIN, fzf by default will use the find command to fetch the list of files excluding hidden ones.

fzf fits very well into to the *nix philosophy, which gives it a lot of flexibility, including a lot of built-in goodness for bash and zsh shells (head to the [examples][#examples] for more).

The program is portable, has no dependencies and can be installed easily following the instructions in the README of the project.

Examples

Without further introduction, here are some examples of fzf in action.

Bare

As stated earlier, fzf can be invoked on its own, and it will provide fuzzy completion for the items found in the current directory and subdirectories.

Previewing

$ fzf --preview="head -$LINES {}"

In addition, you can provide a --preview flag with a command to execute when a file is selected. The command string accepts placeholders to be replaced by fzf during file selection:

{} single-quoted string of the current line.
{+} space-separated list of the selected lines (or the current line if no selection was made) individually quoted.
{q} current query string.

Fuzzy completion for commands

# Files under current directory
$ cat **<TAB>
# Processes
$ kill -9 <TAB>
# Host names
$ ssh **<TAB>
# Environment variables / Aliases
$ export **<TAB>

A great feature which comes bundled-in for bash and zsh is fuzzy completion for different commands, in most cases just typing ** and pressing ↹ (TAB) will give you the fuzzy completion interface to feed the command with the selected items.

On bash, fuzzy completion is enabled only for a predefined set of commands via the complete built in, but you can enable it for other commands with:

complete -F _fzf_path_completion -o default -o bashdefault <COMMAND_NAME>

Searching history

Also by default in bash and zsh, there’s built-in support for the reverse incremental history search.

Further configuration

You can provide additional configuration options via environment variables:

FZF_COMPLETION_TRIGGER helps you to define a custom trigger sequence instead of the default **
FZF_COMPLETION_OPTS allows you to define and provide custom default flags
FZF_DEFAULT_COMMAND default command to use when the input is tty

Sources

Weekly Command: going over Git history with tig

Roberto Dip — Thu, 19 Apr 2018 02:41:39 +0000

There are many UI utilities that allow you to explore a Git repository, but for me, nothing beats tig.

The Basics

tig [view] [-options] [revs]

tig is a text-based user interface for Git that can be used directly from your terminal to explore the repository in the current working directory. It can be invoked directly or by piping Git output.

Views

The program exposes different views, each view displays different data from the repository. You can tell tig to use a view with the optional view argument or you can switch views once tig is open with the default keybindings:

View	Argument	Keybinding
main view	-	m
diff view	-	d
log view	log	l
show view	show	-
blame view	blame **	b *
grep view	grep **	g
refs view	refs	r
status view	status	s
help	--help	h

You need to have a file selected in the UI ** You neeed to supply an argument

If you are familiar with Git commands, view names are almost self-explanatory, I strongly encourage you to download tig and try the different views by yourself, they are very intuitive to navigate.

Pager mode

You can also pipe regular Git commands via stdin to tig, for example:

$ git show | tig

Resources

SCUMM internals and syntax for the sake of nostalgia

Roberto Dip — Tue, 17 Apr 2018 17:20:20 +0000

I spent an unbelievable amount of time playing games from the LucasArts franchise circa 1999 in an Intel 486. It was already an old computer at the time, but it was more than enough to play games like Maniac Mansion and Monkey Island.

As I grew up and learned more about computers, I found that “SCUMM” wasn’t just the name of a bar in Monkey Island, it was the platform on which all my favorite childhood games were programmed on.

This post explores the SCUMM scripting language based on a list of sources, from talks to interviews and various posts.

Brief History

SCUMM stands for “Script Creation Utility for Maniac Mansion”, and as the name implies, it was created as to serve as a toolset to develop Maniac Mansion.

The game had to work in a Commodore 64, so as it was done with other games at the time, the game was developed at first in Assembly, but in order to deal with the complexity of the game (it had seven different characters and a huge amount of combinations), Ron Gilbert decided to create a higher level syntax.

I started kind of hand-coding it all in assembly language, […] and it was just obvious this was going to be a very difficult task without some kind of a language to be able to abstract the gameplay.

— Ron Gilbert, interview for IGN

Once Maniac Mansion was finished, they realized that SCUMM could be used as a general purpose tool to create adventure games.

It’s worth noting that SCUMM wasn’t only a scripting language, it also included other utilities named after body fluids like SPIT (a font editor) and FLEM (a graphical interface.)

Concepts and Syntax

The syntax of SCUMM scripts is really high level and strongly related to the characteristic graphics interface of the adventure games that were built with it:

Screenshot of a scene of Maniac Mansion gameplay portraying a typical SCUMM interface: a character, objects to interact with and actions.

If you haven’t played any of the old school LucasArts adventure games (and I really encourage you to do it), they consist of a story/puzzle that you must solve: you are in control of a character, which is able to interact with objects via a set of predefined actions placed at the bottom of the screen.

The scripting language itself is a very well done abstraction: it has almost a 1:1 mapping regarding what you see in the screen to what you code, and is based upon simple concepts/keywords like:

room: each screen that the user sees is a room, which is no more than a group of object definitions.
actor: the characters in the game, actors can wear costumes and interact with objects.
object: elements in the room, objects can have verbs, which are related to actions you can apply to them.
costumes: definitions of a player costume.
sounds: definitions of sounds.

Doing a more detailed summary of the syntax would be pointless, but if you are interested, the SCUMM Tutorial is an in-depth resource. You can even write your own games using compilers like SCUMMC and SCUMM-8.

To give you a taste of what a bit of code may look like, let’s try to code the scene in the image above (Dave in the doorstep)

room "doorstep" doorstep {
  sounds {
    "SFX/doorclos" door-close
    "SFX/dooropen" door-open
  }

  costumes {
    "costumes\dave" dave-skin
  }

  enter {
    ; Code to set up the room lives here,
    ; you can trigger sounds, animations, etc.
  }

  exit {
    ; Things to do when exiting the room,
    ; cleaning states, stopping sounds, etc
  }

  object Key {
    name is "key"

    verb look-at {
      say-line "it is a key"
    }

    verb pick-up {
      pick-up-object key
    }
  }
}

Multitasking

Probably the most distinctive part of SCUMM was that it was multi-tasking. This meant that multiple scripts could effectively run simultaneously. You might create a clock on the wall in Zak McKracken’s office and have it animate. There would be a single, very simple script just for the clock that would tell the animation engine to change from one image of the clock to the next, tell the sound engine to play the “tick” sound, and then tell the script to “sleep-for 1 second” and then repeat.

— Aric Wilmunder, Gamasutra interview

Another of the outstanding points of SCUMM is how easy it made for the developer to add cooperative multitasking into the system. You could trigger a task with the do keyword and the task will keep running once per frame until it hits a break-here, then it runs the next task.

For example, in the next image, we can think that every ghost pirate and the shadow moving in the background is a separate script running on its own:

Here’s a script transcripted from Gilbert’s talk which details how you would make a clock tick:

script clock-tick {
  do {
    clock-state = not clock-state
    object living-room-clock state clock-state
    play-sound clock-tick
    break-here 60
  }
}

Other interesting bits

To wrap up, here are some bits that may be interesting:

The SCUMM compiler was written in Yacc and Lex.
SCUMM requires carriage returns at the end of lines but how things are tabbed does not matter. It is not possible to put open brace, an instruction and a closing brace on the same line.
The SCUMM system has global and local variables. A global variable can be used throughout the game. A local variable is specific to a script.

Resources

Weekly Command: jumping directories with z

Roberto Dip — Thu, 12 Apr 2018 21:58:14 +0000

This week we are going to review one of my favorite and most used commands. If you move around the command line, you may have experienced the pain of recalling directory structures and typing them over and over to access directories that are distant from each other.

z is specifically designed to fix this pain by allowing you to jump trough directories.

h/t to @luislavena for sharing the existence of z with me.

The basics

z [-chlrtx] [regex1 regex2 ... regexn]

As simple as it sounds, z will take you directly to a directory that matches the regexes you provide, no matter where you are.

It’s worth emphasizing that the argument is a regex, which is especially useful if you are lazy: I usually type the first two or three letters of the directory that I want to access and it’s enough.

z is very easy to install, if you are anxious to try it by yourself head to the install instructions in the README.

An important gotcha: z is only able to take you to directories that you already visited before: it needs a learning phase because of how it works.

How it works

z does its magic by storing the directories you visit in a flat file database and matching your query against this list. This file is located by default in $HOME/.z, but the directory can be changed via the $_Z_DATA variable.

z works so well because this list of directories is ordered based on what the author refers as “frecency” (a term apparently coined by Firefox developers, if you have more info please let me know!) which combines frequency: the amount of time you spend in a directory with recentness: how recently you were in a directory.

Session using z on the left, flat file database on the right: notice how it gets updated every time a directory is visited.

As you keep navigating directories, the rank maintained by z undergoes aging based on a simple formula:

The rank of each entry is incremented every time it is accessed. When the sum of ranks is over 9000, all ranks are multiplied by 0.99. Entries with a rank lower than 1 are forgotten.

Examples

Command	Functionality
`z foo`	cd to most frecent dir matching foo
`z foo bar`	cd to most frecent dir matching foo, then bar
`z -r foo`	cd to highest ranked dir matching foo
`z -t foo`	cd to most recently accessed dir matching foo
`z -l foo`	list all dirs matching foo (by frecency)

Sources

z Repository

Weekly Command: processing JSON with jq

Roberto Dip — Thu, 29 Mar 2018 23:30:46 +0000

The command of this week is jq, a flexible tool to process and manipulate JSON input.

Before we start, let me say that a blog post can’t do justice to jq without turning into a full manual. If you are eager to learn more, head to the resources section.

The basics

**jq** [_options_...] _filter_ [_files_...]

While jq can take a wide variety of options they mostly concern input and output formatting, the real magic happens in the filter argument.

The key point to understand how jq works is to think about it as a processor: receives an input, processes the input based on the defined filter and generates an output.

Examples

Most of the examples below will use the /feed endpoint of the NASA NeoWs API, which retrieves a list of asteroids near Earth in a date range.

For the sake of simplicity, let’s assume that in every example, the following curl command is used to retrieve and pipe JSON into jq:

$ curl "https://api.nasa.gov/neo/rest/v1/feed?\
start_date=$(date +%Y-%m-%d)\
&api_key=DEMO_KEY"

The endpoint retrieves a response that roughly looks like:

{
  "links" : { /* pagination info here */ },
  "element_count" : 70,
  "near_earth_objects" : {
    "2018-03-30" : [/* Asteroids around this date */],
    "2018-04-01" : [/* Asteroids around this date */]
  }
}

If you want to follow along and don’t have a terminal at hand, there’s an online playground that you can use.

Basic filter

$ jq '.'

The basic filter produces the input unchanged. You can think of the dot as the “current context” which in this case is the whole JSON input.

tip: since jq formats its output by default, you can use this to pretty print JSON in the console.

Accessing fields

$ jq '.element_count'

Fields can be accessed in the JSON data with a dot (.) followed by the field name. You can think of this as accessing a property from the current context.

In this case, we are retrieving the element_count field to know how many asteroids around the Earth are. At the time I ran it, there were 70!

Collections

$ jq '.[]'

The family of .[] filters is very versatile:

If the input is an array, .[] will iterate through the elements.
If the input is an object .[] will iterate through the object values.
If you provide an index, .[0] will access elements in arrays by their position.
If you provide a string, .["foo"] will access elements in objects by their keys.

Pipes

$ jq '.near_earth_objects | .[] | .[].name'

Filters can be chained with pipes as you would normally do with commands in the terminal. In this example, we are looping through the fields of the .near_earth_objects, which are arrays, and retrieving the names of all asteroids.

Result (click to expand)

"(2015 FN120)"
  "(1999 FP19)"
  "(2017 FW90)"
  "(2018 FH5)"
  "(2018 FM4)"
  "(2018 FF1)"
  "348306 (2005 AY28)"
  "(2008 GB110)"
  "(2012 QG42)"
  "(2016 CC194)"
  "(2017 GO4)"
  "(2018 CZ13)"
  "(2018 FV3)"
  "(2005 GR33)"
  "(2007 GS3)"
  "(2007 RX8)"
  "(2011 HN5)"
  "(2014 OE338)"
  "(2016 GA3)"
  "489486 (2007 GS3)"
  "(2018 FV1)"
  "(2018 EM4)"
  "(2007 DB61)"
  "17511 (1992 QN)"
  "(2001 OT)"
  "(2004 FG29)"
  "(2018 ER1)"
  "(2004 FG1)"
  "(2008 GH110)"
  "(2015 XE261)"
  "474574 (2004 FG1)"
  "(2017 RO17)"
  "498548 (2008 GH110)"
  "(2018 FW4)"
  "(2011 EC7)"
  "85953 (1999 FK21)"
  "204232 (2004 DG2)"
  "225586 (2000 WS67)"
  "(2009 FF)"
  "(2015 TC25)"
  "(2017 GX6)"
  "(2017 UD1)"
  "(2018 FW2)"
  "(2007 WB)"
  "(2008 GE)"
  "(2011 HJ)"
  "(2013 HM11)"
  "(2016 AG9)"
  "(2015 XT168)"
  "(2008 VR4)"
  "(2013 OW2)"
  "509520 (2007 WB)"
  "(2013 GZ7)"
  "(2011 UH20)"
  "441304 (2008 AU26)"
  "(2004 FJ29)"
  "(2015 EL7)"
  "(2016 FF1)"
  "(2016 WB10)"
  "(2018 EB)"

Object and Array construction

$ jq '{count: .element_count}'
$ jq '.near_earth_objects | [.[] | .[]]'

jq also provides granular control over the formatting of the output by allowing you to explicitly define how it will look like.

In the first example, we are returning an object with element_cont as count.

The second example is a bit more involved, we are arranging all near_earth_objects of all dates in a single array. This can be done in a nicer way with the help of operators and functions (see below).

Result 1 (click to expand)

{count: 70}

Result 2 (click to expand)

[
    {/* Asteroid data */},
    {/* Asteroid data */},
    {/* Asteroid data */},
    {/* Asteroid data */},
  ]

Operators and functions

$ jq '.near_earth_objects | map(.[])'

jq also comes with a group of built-in operators and functions: +, -, length, map, add, range, and the list goes on. In this case we are revisiting our previous example to arrange all near_earth_objects of all dates in a single array using map.

Result (click to expand)

[
    {/* Asteroid data */},
    {/* Asteroid data */},
    {/* Asteroid data */},
    {/* Asteroid data */},
  ]

All together now ♫

$ jq '.near_earth_objects | map(.[] | {name, is_potentially_hazardous_asteroid})'

To wrap up, let’s build a filter to freak out our family and friends. Every asteroid has a name and a param called is_potentially_hazardous_asteroid which, as the name implies, is currently defined based on parameters that measure the asteroid’s potential to make threatening close approaches to the Earth (source).

Let’s retrieve a list of all asteroids names indicating if they are hazardous or not:

Result (click to expand)

[
    {
      "name": "(2018 FF1)",
      "is_potentially_hazardous_asteroid": false
    },
    {
      "name": "348306 (2005 AY28)",
      "is_potentially_hazardous_asteroid": true
    },
    {
      "name": "(2008 GB110)",
      "is_potentially_hazardous_asteroid": false
    },
    {
      "name": "(2012 QG42)",
      "is_potentially_hazardous_asteroid": true
    }
  ]

Resources

Memories of writing a parser for man pages

Roberto Dip — Sat, 24 Mar 2018 18:30:13 +0000

I generally enjoy being bored, but sometimes enough is enough—that was the case a Sunday afternoon of 2015 when I decided to start an open source project to overcome my boredom.

In my quest for ideas, I stumbled upon a request to build a “Man page viewer built with web standards” by Mathias Bynens and without thinking too much, I started coding a man page parser in JavaScript, which after a lot of back and forths, ended up being Jroff.

Back then, I was familiar with manual pages as a concept and used them a fair amount of times, but that was all I knew, I had no idea how they were generated or if there was a standard in place. Two years later, here are some thoughts on the matter.

How man pages are written

The first thing that surprised me at the time, was the notion that manpages at their core are just plain text files stored somewhere in the system (you can check this directory using the manpath command).

This files not only contain the documentation, but also formatting information using a typesetting system from the 1970s called troff.

troff, and its GNU implementation groff, are programs that process a textual description of a document to produce typeset versions suitable for printing. It’s more ‘What you describe is what you get’ rather than WYSIWYG.

— extracted from troff.org

If you are totally unfamiliar with typesetting formats, you can think of them as Markdown on steroids, but in exchange for the flexibility you have a more complex syntax:

The groff file can be written manually, or generated from other formats such as Markdown, Latex, HTML, and so on with many different tools.

Why groff and man pages are tied together has to do with history, the format has mutated along time, and his lineage is composed of a chain of similarly-named programs: RUNOFF > roff > nroff > troff > groff.

But this doesn’t necessarily mean that groff is strictly related to man pages, it’s a general-purpose format that has been used to write books and even for phototypesetting.

Moreover, It’s worth noting that groff can also call a postprocessor to convert its intermediate output to a final format, which is not necessarily ascii for terminal display! some of the supported formats are: TeX DVI, HTML, Canon, HP LaserJet4 compatible, PostScript, utf8 and many more.

Macros

Other of the cool features of the format is its extensibility, you can write macros that enhance the basic functionalities.

With the vast history of *nix systems, there are several macro packages that group useful macros together for specific functionalities according to the output that you want to generate, examples of macro packages are man, mdoc, mom, ms, mm, and the list goes on.

Manual pages are conventionally written using man and mdoc.

You can easily distinguish native groff commands from macros by the way standard groff packages capitalize their macro names. For man, each macro’s name is uppercased, like .PP, .TH, .SH, etc. For mdoc, only the first letter is uppercased: .Pp, .Dt, .Sh.

Challenges

Whether you are considering to write your own groff parser, or just curious, these are some of the problems that I have found more challenging.

Context-sensitive grammar

Formally, groff has a context-free grammar, unfortunately, since macros describe opaque bodies of tokens, the set of macros in a package may not itself implement a context-free grammar.

This kept me away (for good or bad) from the parser generators that were available at the time.

Nested macros

Most of the macros in mdoc are callable, this roughly means that macros can be used as arguments of other macros, for example, consider this:

The macro Fl (Flag) adds a dash to its argument, so Fl s produces -s
The macro Ar (Argument) provides facilities to define arguments
The Op (Optional) macro wraps its argument in brackets, as this is the standard idiom to define something as optional.
The following combination .Op Fl s Ar file produces [-s file] because Op macros can be nested.

Lack of beginner-friendly resources

Something that really confused me was the lack of a canonical, well defined and clear source to look at, there’s a lot of information in the web which assumes a lot about the reader that it takes time to grasp.

Interesting macros

To wrap up, I will offer to you a very short list of macros that I found interesting while developing jroff:

man

TH: when writing manual pages with man macros, your first line that is not a comment must be this macro, it accepts five parameters: title section date source manual
BI: bold alternating with italics (especially useful for function specifications)
BR: bold alternating with Roman (especially useful for referring to other manual pages)

mdoc

.Dd, .Dt, .Os: similar to how man macros require the .TH the mdoc macros require these three macros, in that particular order. Their initials stand for: Document date, Document title and Operating system.
.Bl, .It, .El: these three macros are used to create list, their names are self-explanatory: Begin list, Item and End list.

Sources

Concurrency != Parallelism

Roberto Dip — Wed, 31 Jan 2018 02:11:56 +0000

This post was originally published on monades.roperzh.com

I truly enjoy listening to Carl Hewitt talk about computers, and something he repeats often is “concurrency is not parallelism”. For me, there was no real difference, and honestly, I’ve never bothered to dig into it.

Last week, I’ve stumbled upon Rob Pike’s talk on the subject, which leads me to finally do some research on the matter. Here are my findings.

beware: as with most things in life, a lot of people consider that there’s no real difference between the two.

Concurrency

Concurrency is all about doing tasks in the same interval of time. The important detail here is that the tasks don’t necessarily execute at the same time, but they can be divided into smaller tasks that can be interleaved.

A good place where concurrency takes place is in the kitchen, imagine a single chef cutting lettuce while is also checking from time to time something in the oven. He has to stop cutting, check the oven, stop checking the oven and then start cutting again, and repeat the process until is done.

As you can see, concurrency is mostly related to the logistics, without concurrency, the chef will have to wait until the meat in the oven is ready in order to cut the lettuce.

Parallelism

Parallelism, on the other hand, is about doing tasks literally at the same time, as the name implies they are executed in parallel.

Back in the kitchen again, now we have two chefs, one can take care of the oven, while the otter cuts the lettuce, we split the work at the expence of having another chef.

Parallelism is a subclass of concurrency: before you execute multiple tasks simultaneously, you first have to manage multiple tasks.

Sources

Brave Clojure: The Sacred Art of Concurrent and Parallel Programming
Haskell Wiki
Rob Pike’s talk

Bonus

If you have time, take a look at this humorous exchange between Carl Hewitt and a Wikipedia moderator about concurrency vs parallelism.

Rediscovering make: automatic variables

Roberto Dip — Tue, 23 Jan 2018 21:32:33 +0000

Despite its simplicity, automatic variables are very convenient when writing non-trivial makefiles. This article covers what they are and how to use them.

note: if you need a quick refresher on rules, check out the previous post on the make series.

Automatic variables

As patterns can be really useful to define targets and prerequisites, they are also dynamic: every time the recipe is run the wildcard can be matching different files, which is a problem if you need to reference them in your recipe.

You can deal with this using automatic variables. As the name implies, make defines a set of variables for you every time a rule is executed, based on the target and the prerequisites of the rule.

While automatic variables are most useful with patterns, the following examples use static file names in order to simplify the concepts, but all of what’s described above works for patterns too.

Suppose you have the following rule:

targetdir/targetfile.o : pre.c predir/pre.h | order-only.c
    # recipe

When the recipe is run, the following variables are automatically assigned:

Variable	Meaning	Example
$@	file name of the target	targetdir/targetfile.o
$<	name of the first prerequisite	pre.c
$?	names of all prerequisites newer than the target	depends on context
$^	names of all prerequisites	pre.c predir/pre.h
$		names of all the order-only prerequisites
$*	stem with which an implicit rule matches	targetdir/targetfile
$%	name of the archive member (see archives)	-

Modifiers

Sometimes, you may only need only the directory or the filename of the variable, for this scenario make offers two modifiers you can add to your variables:

Modifier	Meaning	Example
D	the directory part	`dir`
F	the file-within-directory part	`simple.c`

Modifiers are added right next to the variable and need to be wrapped in parenthesis, for example if $@ is targetdir/targetfile.o:

$(@D) is targetdir
$(@F) is targetfile.o

Helpers

As automatic variables can be difficult to remember, here are a couple of resources to help you:

Phony makefile

A good way to learn is by experimentation: create a directory, place mock files in there, and define your targets and prerequisites, then, instead of doing something useful in the recipe, just print the variables:

targetdir/targetfile.o : predir/pre.h | order-only.c
  @echo '@: $@'
  @echo '%: $%'
  @echo '<: $<'
  @echo '?: $?'
  @echo '^: $^'
  @echo '|: $|'
  @echo '*: $*'

Snippets

I published snippet packs for Visual Studio Code and Sublime Text that may come in handy: they give you a list of the available variables, followed by a short explanation of their purpouse:

note: If you are using another editor there are tools to convert between snippets formats.