DEV Community: Keith Bennett

AutoPilot, CoPilot, or AI Assistant?

Keith Bennett — Sat, 03 Feb 2024 00:47:00 +0000

AI Downsides

In New GitHub Copilot Research Finds 'Downward Pressure on Code Quality', Visual Studio magazine cites a GitClear study that points out that AI for coding results in less maintainable (and therefore costlier) code.

They point out, for example, that copying and pasting code in multiple locations where it is needed, rather than moving it to a single location and referring to it there, is prone to producing repetitive code, that is, code that is not DRY.

In addition, at the time of this writing (January 2024), AI still makes plenty of mistakes; that is, a) misunderstandings of intent, b) hallucinations, and c) downright errors.

Resisting Oversimplification

While the study describes in great detail the pitfalls it found in AI-assisted coding, we must not succumb to the oversimplification so prevalent in these times and generalize that "AI is 100% bad". We need to realize that the positive or negative value of AI depends on this crucial nuance: the extent to which the developer relies on and defers to AI, rather than supervising it with a critical eye.

The name "Copilot" is apt. It must not be "Autopilot". However, even "Copilot" may not sufficiently express the subservient position that AI should take; "AI Assistant" would probably be more accurate, and is exactly the name used by JetBrains, the company that produces outstanding IDE's.

The Optimum Balance

An organization that mandates excessive reliance on AI to quickly reduce labor costs will have a rude awakening when, over time, the poor code quality reveals itself in the form of reduced quality and increased cost.

On the other hand, an organization that completely ignores the immense productivity boost that coding AI offers will "leave money on the table" and fall behind the competition.

Organizational Responsibility

To quote my favorite paragraph in the study:

"If there isn't a CTO or VP of Engineering who actively schedules time to reduce tech debt, you can add "executive-driven time pressures" to the list of reasons that newly added copy/paste code will never be consolidated into the component libraries that underpin long-term development velocity."

In other words, resisting the temptation to build working code quickly in the short term at the expense of the medium and long term requires will power and effort, plus initiative or at minimum support from the highest levels of the organization. Those organizations that are not able or willing to delay gratification will suffer the consequences.

Investing in code quality pays off well, but does have an initial cost. In the case of AI, this will be especially important.

Conclusion

The bottom line is that we still need expert software developers to know how to use the tool, appreciating its knowledge and assistance, but taking the lead to verify and integrate it correctly. The AI landscape is changing at astonishing speed. Organizations and their developers need to reevaluate tools and policies frequently to ensure they are making optimal use of this game changing technology.

Installing International Language Keyboard Input Methods on Linux Mint 20 Cinnamon & Mate

Keith Bennett — Mon, 24 Aug 2020 16:39:31 +0000

I just started using a desktop system using Linux Mint 20 Cinnamon, and wanted to make sure that keyboard input methods worked for Thai and Korean. This took me down a long road, consulting several blog articles, questions, and answers, so I decided to document it to save others (and future me!) this effort.

I've also briefly tested that this procedure works with Arabic, French, Hebrew, Russian, and Swedish, so I'm fairly confident that it will work for any supported language and input method.

Below are the instructions for Linux Mint Cinnamon; I believe they will be identical for Mate.

Language Support

Press the {Super} key, start typing Languages, and select the "Languages" application.
Click "Install/Remove Languages" at the bottom right.
If requested, enter your login password to grant superuser rights to "Languages".

For each language you want to add:

Click "Add"
Start typing the name of the language, select the language you want, and then click the "Install" button. There may not be any confirmation that it has completed.
Find the newly installed language in the list (again, you can start to type it to find it), select it, and click the "Install Language Packs" button.
Confirm any confirmation dialogs as necessary.

If...

The remainder of work to be done involves the IBus input method framework.

If at any point during this procedure, you see a dialog saying "The IBus daemon is not running. Do you wish to start it?", click "Yes".

Also, if you encounter a dialog saying "IBus has been started! If you cannot use IBus, add the following lines...", just click OK. I have seen this happen but have never needed the information displayed in the dialog.

Install IBus

Install the IBus keyboard input management framework and the Hangul (native Korean alphabet) support:

sudo apt install -y ibus-m17n ibus-hangul

Add IBus to the Startup Applications List

Press the {Super} (Windows/Mac) key, type startup and select "Startup Applications".

Click the "+" button.
Select "Custom command".
For Name: IBus
For Command: ibus-daemon
Click "Add".

Configure IBus

Press the {Super} key and type ibus. Select "IBus Preferences".

Add a keyboard shortcut

Cinnamon uses {Super}{Space} to navigate panel entries, and I have not found a way to disable that, so I assign another shortcut for language switching, {Ctrl}{Super}{Alt}-k ("k" for keyboard). Here's how:

On the "General" tab:

click the button labelled "..." to the right of the "Next Input Method" text field
with "{Super}" selected, click the "Delete" button
Replace any text in the "Key code" input field with "k"
check "Control", "Alt", and "Super"
click the "Add" button
click the "OK" button

You may not need to do this if you are using Mate and the {Super}{Space} key combination is available for IBus to use.

Also, I found that {Ctrl}{Super}{Alt}-k did not work when pressed while certain input methods were active. I briefly researched how to disable Cinnamon's use of {Super}{Space} but could not find an answer. Anyone? Assigning multiple key combinations to this action is supported, so that is another approach.

Add input methods

On the "Input Method" tab, click "Add".

For each desired input method, click your desired language, or if it is not listed (as with Korean and Thai), click the three vertical dots entry, click the text field to give it focus, and type the language into the text field, then select the entry below the language name that you want. In the case of Korean, there is only one ("Hangul"). Click "Add".

Reboot and Test

Reboot the system. (If you want to save time, you could log out and then in again instead of rebooting, but that will not verify that IBus was started on system startup.)

Test switching input methods with {Ctrl}{Super}{Alt}-k, and typing text into an application that can accept it.

The Language Panel Applet

There are two methods for invoking input method selection:

the keyboard shortcut ({Ctrl}{Super}{Alt}-k, {Super}{Space}, etc.)
the panel applet

The panel applet is on the system panel, and will display the currently selected language. Clicking it will display a list of languages from which you can select a different one. This applet can be used as a fallback mechanism for changing language if the keyboard shortcut does not work.

Unfortunately, the text on this indicator is displayed in a dark blue text that is difficult to see against the black background, so it may take some effort to find. The Korean language setting deals with this by showing a colorful icon instead of text.

Cautions

Invoking Input Method Selection

As mentioned, if the keyboard shortcut for selecting an input method does not work, remember the panel applet and use that instead.

Know Your Input Methods

The Korean input method uses a {Shift}{Space} toggle key combination to toggle between Hangul and English character input. When you switch into Korean input mode from another language, it will initially be in English mode, so it may not be obvious that you have successfully switched to Korean. Press {Shift}{Space} to toggle to Hangul input mode.
If you're new to a language, there may be things to learn about that language's computer input methods that are not taught in school or otherwise obvious. If the input does not appear to work correctly, make sure you're not overlooking a feature of that input method.
Some languages have many input methods from which to choose, and some may be much better than others for your purposes. For example, some are Dvorak layouts of interest only to the hardiest of souls.

Language-Specific Configuration

There may be language specific requirements with other languages like the need to install the ibus-hangul package for Korean.

You're done!

Thanks for reading, and let me know in a comment on the dev.to article page if you have any corrections or improvements to offer.

Using Lambdas to Simplify Varying Behaviors in Your Code

Keith Bennett — Tue, 12 May 2020 18:37:42 +0000

Lambdas are anonymous (unnamed) functions. Unlike methods in object oriented programming languages, lambdas are not bound to any object. Although they are best known in the context of functional programming languages, object oriented programming languages that support lambdas, such as Ruby, can greatly benefit from their use. This is especially true when it comes to implementing varying behaviors(1).

By varying behaviors, I mean situations where the code that needs to run is not fixed, but rather specified, using parameters, subclassing, or configuration. Examples would include cases for which you would use the Command and Strategy design patterns.

Many developers are already familiar with using lambdas for event handling (implemented with the function keyword in JavaScript and also as arrow functions in ES6) , but there are many other cases in which they are helpful, a couple of which I will describe below.

The Problem with Non-Lambda Approaches

The procedural if-elsif-end or case clauses work when you have a small number of conditions and actions that are known in advance, but if you don't, they're pretty useless.

And although the object oriented approach of polymorphism by inheritance (2) can produce a correct result, in many cases it is unnecessarily verbose, ceremonial, and awkward.

To further embellish on this point, @kayess pointed me to a discussion on a Scheme forum where Anton van Straaten said about closures (lambdas are closures): "A closure's simplicity can be an asset: classes and interfaces can get in the way of simple parameterization of behavior. Anyone who's tried functional programming in Java or C++ has encountered this - it can be done, but it's more tedious." Anton goes on to provide an entertaining koan about the subject.

Furthermore, though we're accustomed to thinking about this problem in the context of a single customizable behavior, what if there are several?

Let's say we have a class that contains 3 varying behaviors. As an admittedly contrived example, let's say we have classes for each of hundreds of different species of animals, and they each have a move, sleep, and vocalize behavior. As a simplifying assumption, let's say that each of these behaviors has 7 possible variations, each of which is shared by many species. If we were to write a class to implement each possible set of the 3 behaviors, we would need the Cartesian product of classes, (7 * 7 * 7), or 343 classes! That would be a silly monstrosity for several reasons of course, one of which being we could simplify the design by providing a class hierarchy for each of the three kinds of behavior, and plug those into the larger class -- but then we would still need (7 + 7 + 7), or 21 classes! (Probably 24 really, as pure design would dictate an additional class as an abstract superclass for each set of 7 implementations).

If these behaviors are truly complex enough to justify a class of their own, this is not a problem. However, often they are not, and the solution is many times as verbose and complex as it needs to be.

A better solution is using callables such as lambdas.

Callables as a Superset of Lambdas

In traditional object oriented languages such as Java and C++, polymorphism is (in general) implemented by inheritance. Ruby does this also (3), but in addition, Ruby uses duck typing, meaning that any object that responds to the method name can be used, regardless of its position in the class hierarchy.

This means that in Ruby, since the method used to call a lambda is `call`, any object that responds to `call` can be used in place of a lambda.

It could be a lambda, an instance of a class, or even a class or module. This provides great flexibility in implementing varying behavior. You can choose what kind of object to use based on your situation. For complex behaviors you may want modules or classes, and for simpler behaviors a lambda will work just fine.

Since any object responding to call can be used in place of a lambda, I will use the term callable instead of lambda where applicable.

The Buffered Enumerable

I once worked on a project where I needed to implement buffering of multiple kinds of things received over network connections. I started writing the first one, and noticed how the code could be cleanly divided into two kinds of tasks: 1) knowing when to fetch objects into the buffer and other buffer management tasks, and 2) knowing how to fetch each block of objects and what else to do each time that fetch is performed (e.g. logging, displaying a message to the user, updating some external state).

Realizing that #1 would be common and identical to all cases, and only #2 would vary, I thought about how wasteful it would be to implement #1 separately in all cases. I thought about the admonition about high cohesion / low coupling, and the Unix axiom "do one thing well", and decided to separate the two.

The most natural way to design this functionality in Ruby is with an Enumerable, which will have access to all kinds of functional wizardry thanks to the methods it gets for free by including the Enumerable module. In addition, it can easily be used to generate an array by calling its to_a method.

This is the origin of the BufferedEnumerable class in my trick_bag gem. This class manages buffering but has no idea how to fetch chunks of data, nor what else to do at each such fetch; for that, the caller provides callables such as lambdas. (Upon user request, the ability to subclass it and override its methods was also added.) The result is a dramatic simplification, where the logic of buffering is defined in only one place, and the places it is used need not be concerned with its implementation (or its testing!).

To create an instance of this with callables, we call the BufferedEnumerable class method create_with_callables, which is defined as follows(4):

 def self.create_with_callables(chunk_size, fetcher, fetch_notifier = nil)
    instance = self.new(chunk_size)
    instance.fetcher = fetcher
    instance.fetch_notifier = fetch_notifier
    instance
  end

When a fetcher callable has been defined (as opposed to the use of the subclassing approach, where an overriding method is used), it is called with the empty data buffer and the number of objects requested as shown below whenever the buffer needs to be filled, as follows:

fetcher.(data, chunk_size)

(In Ruby, .( is an abbreviation for .call(.)

A trivial fetcher that merely fills the array of the requested chunk size with random numbers could look like this:

fetcher = ->(data, chunk_size) do
  chunk_size.times { |index| data[index] = Random.rand }
end

Below is a pry example that illustrates the call to that fetcher, and its effect on the passed array:

[7] pry("")> a = []; fetcher.(a, 2)
2
[8] pry("")> a
[
    [0] 0.4885287976297428,
    [1] 0.5493143769524284
]

After the buffer is filled, if a fetch notifier callable has been defined (unlike the fetcher, this is optional), it too is called, with the data buffer:

fetch_notifier.(data)

A trivial fetch notifier might look like this:

->(data) { puts "#{Time.now} Fetched #{data.size} objects" }

(data.size will not necessarily be equal to chunk_size, especially on the last fetch.)

This notifier might produce something looking like this:

2018-07-26 17:19:47 +0700 Fetched 1000 objects

After defining the fetcher and fetch_notifier lambdas, we could call the class method for creating a BufferedEnumerable shown above as follows:

buffered_enumerable = BufferedEnumerable.create_with_callables( \
    1000, fetcher, fetch_notifier)

This enumerable can be used to call each or any other of the rich set of methods available on the Enumerable module.

By parameterizing the behaviors with callables, we have increased the simplicity of the implementation by separating the two orthogonal tasks into separate code areas, and avoided the unnecessary overhead of the inheritance approach, which would have packaged these functions in classes.

Using Predicate Callables to Implement Filters

Predicates are functions that return a Boolean value, that is, either true or false. There are many uses for predicates in software: filters, boundaries, triggers, authentication results...again, anything that produces a true or false value.

Configurable predicates are another natural fit for using callables.

I once had to write a DNS mock server for network testing that could be configured to respond with specific behaviors based on the characteristics of the incoming request. In another situation more recently, I was writing some accounting software and wanted to be able to filter the working set of transactions based on date, category, etc.

Both cases were an excellent fit for using callables as filters.

In the case of the mock DNS server, there were multiple criteria for the DNS request filters, such as protocol (TCP vs. UDP), qtype (question type), qclass (question class), and qname (question name). I provided methods that return lambdas that filter for specific values for those attributes; for example, to create a filter that will return true only for the qname example.com, you would do the following:

MockDnsServer::PredicateFactory.new.qname('example.com')

The qname method (i.e. the method that returns a filter for exactly one qname value) is defined as (roughly):

def qname(qname)
  ->(message, _protocol = nil) do
    eq_case_insensitive(message.qname, qname)
  end
end

If you view the method body from left to right, you will notice the prominent -> and its corresponding end two lines beneath it, which tell you that the value returned by this method is a lambda. The leading underscore of the _protocol parameter is a convention that indicates that that parameter is unused by the lambda.

Notice that the qname parameter's value (e.g. "example.com") is effectively stored in the lambda that the method returns? This technique is called partial application, and is extremely useful when working with lambdas.

Does storing state in the lambda make it any less functional? Not really; the state is immutable and used only for comparison.

If we were to refine the filter by adding the requirement that the qtype be 'A' and the protocol be 'TCP', then we could call methods to return those filters as well, and combine all three using the all compound filter. (Other compound filters are any and none.) Here is what that would look like:

pf = MockDnsServer::PredicateFactory.new
filter = pf.all(
    pf.qtype('A'),
    pf.qname('example.com'),
    pf.from_tcp
)

The all compound filter is nothing more than a simple wrapper around Ruby's Enumerable's all? method:

  def all(*predicates)
    ->(message, protocol = nil) do
      predicates.all? { |p| p.call(message, protocol) }
    end
  end

Why does all this work? The filters are interchangeable because they all have uniform inputs and outputs. That is, they take the same parameter list ([message, protocol] in this example), and they all return a value usable by the caller (true or false in this example). We've already seen one implementation, the lambda returned by the qname method shown above. Here is another one; this one returns true if and only if the message was sent over TCP:

def from_tcp
  ->(_message, protocol) { protocol == :tcp }
end

In this case, we only care about the protocol so we can ignore the first (message) parameter.

Conclusion

I hope I have been successful in persuading you to consider using callables for implementing variable predicates and actions.

These are just two examples. I will stop here for the sake of brevity, but if you have any questions or suggestions for elaboration, please contact me.

You can find me on:

Also, for more information about lambdas, feel free to check out my Ruby lambdas presentation (Speakerdeck slideshow and YouTube video).

Footnotes

(1) One might ask what else is there about lambdas other than varying behavior, but there are other reasons to use them in Ruby. For example, they can be used:

as nested functions
as truly private functions (which, unlike private methods, cannot be accessed with send
for self invoking anonymous functions, to hide variables
for defining functions where methods cannot be defined (e.g. in some RSpec code)
for defining classes dynamically with the class...end notation which may be easier to understand than the Class.new approach.
for chaining operations, as in an array of lambdas

(2) Polymorphism by inheritance is a key characteristic of object oriented design whereby, by virtue of having a common ancestor in the class hierarchy that contains the method in question, objects of different classes can respond to the same message (typically identified by a method or function name) differently. This can be a nice design in some cases, but in others it is an overly heavy handed solution to a simple problem, as it forces the developer to create multiple classes in a class hierarchy.

(2 (cont'd)) @kayess points out that the more precise terms in software engineering are nominal typing for polymorphism by inheritance, and structural typing for duck typing.

(3) To be more precise, Ruby supports polymorphism by inheritance, but not by checking the class hierarchy like most OO languages do. Instead, it is using duck typing, and merely calls the method by name; since a subclass will be able to call its superclass' method by default, it works.

(4) The omission of the fetcher and fetch_notifier parameters from the constructor is intentional. The constructor is intended to be used directly by the caller only when the subclassing approach is used; for lambdas the static method create_with_callables should be used.

This article may be improved over time. To see its revisions you can go to its Github commit history.

Add Bootstrap to Rails 6 with Two Shell Commands

Keith Bennett — Thu, 07 May 2020 21:15:18 +0000

Introduction

In this article I will discuss a very simple approach to configuring your Rails 6 application to work with Bootstrap. An entire Rails repo is provided, including commit history, but this might be all you need:

curl https://raw.githubusercontent.com/keithrbennett/rails-bootstrap-example/master/0001-Add-Bootstrap-configuration.patch | git apply -
yarn add bootstrap jquery popper.js

I recently wanted to add Bootstrap to a new Rails 6 application, but even after reading the documentation and several blog articles, success eluded me.

What finally worked was to follow along with this article by Chris Oliver -- well, more precicely, the video linked to in the article. In it, he shows exactly what to do -- and doing what he said to do worked for me.

Even Simpler -- A Patch

In order to even further simplify the process for future developers, I generated a patch file to make the changes needed to provide the correct configuration. (These changes do assume a new Rails application, so it's possible that some modification to the changes would be required for existing projects.)

You can make the changes to your existing project or a fresh one generated with rails new. Here's how to do it:

Change directory to your project root.

I suggest you have a "clean working tree" (no git-relevant changes since the most recent commit) before you apply the patch; this will make it easier to revert the change or limit your next commit to only the Bootstrap configuration change.

You can download the patch in its raw format (i.e. only the unadorned text and not the HTML web page displaying it) to your local filesystem with the following command:

curl -o bootstrap.patch https://raw.githubusercontent.com/keithrbennett/rails-bootstrap-example/master/0001-Add-Bootstrap-configuration.patch

Then apply the patch:

git apply bootstrap.patch

Alternatively, you can combine the above two steps into one:

curl https://raw.githubusercontent.com/keithrbennett/rails-bootstrap-example/master/0001-Add-Bootstrap-configuration.patch | git apply -

You can see the changes with a git diff.

Adding the JavaScript Libraries

At some point (either before or after the patch) you will need to add the necessary JavaScript libraries:

yarn add bootstrap jquery popper.js

This will make changes to package.json and yarn.lock that will need to be committed.

Testing that Bootstrap Works

We'll want to exercise Bootstrap to verify that it is working correctly.

The index.html.erb file in the repo uses Bootstrap colored border spinners requiring both CSS and JavaScript provided by Bootstrap, and is a good test. Of course, you can find many other components to use in the Bootstrap docs.

The rest of this section discusses setting up a sample app using the patches provided. Here is a list of all the patches:

# already used above, to configure Bootstrap
0001-Add-Bootstrap-configuration.patch                       

# These next 3 can be used to put something in the
# home page to test that Bootstrap is working:
0002-rails-g-controller-home-index.patch
0003-Change-root-route-to-go-to-new-page.patch
0004-Add-Bootstrap-color-border-spinners-to-page.patch

These patches are in the project root of the sample repo.

Applying patches #2 through #4 will set up the sample code to show that Bootstrap is working. Here are commands that will apply the patches directly from Github (without creating patch files on your local filesystem):

curl https://raw.githubusercontent.com/keithrbennett/rails-bootstrap-example/master/0002-rails-g-controller-home-index.patch | git apply -
curl https://raw.githubusercontent.com/keithrbennett/rails-bootstrap-example/master/0003-Change-root-route-to-go-to-new-page.patch | git apply -
curl https://raw.githubusercontent.com/keithrbennett/rails-bootstrap-example/master/0004-Add-Bootstrap-color-border-spinners-to-page.patch | git apply -

This is all you should need to do! At this point you can run rails s and connect to it in your browser. If all goes well you will see something like this:

Git's Patch Support

You may have noticed that the patch file names were numbered and contained the first part of the commit messages in the names. This was not something I did myself, this was done automatically by git. Git has great patch support, and all I needed to do to generate the patches was issue this command:

$ git format-patch HEAD~4
0001-rails-g-controller-home-index.patch
0002-Change-root-route-to-go-to-new-page.patch
0003-Add-Bootstrap-color-border-spinners-to-page.patch
0004-Add-patch-files-git-format-patch-HEAD-4.patch

The HEAD~4 told git how far back I wanted to start (4 commits).

As you saw above, applying a patch to a code base is as simple as:

git apply something.patch

Conclusion

I hope that the time I invested in creating and documenting this simplifying procedure will pay off in the form of time saved for you. If you think I could be useful on your project, or would just like to say hello, please give me a holler at kbennett@bbs-software.com.

This article may be improved over time. To see its revisions you can go to its Github commit history.

Note: The patch (#1) originally published with this article did not include some configuration code needed to expose JQuery and Popper, which could have been a problem implementing customizations. It was fixed on 2020-05-12. You can see the changes here.

Clinics in Your Meetups

Keith Bennett — Mon, 03 Feb 2020 10:47:52 +0000

Information sharing at technical meetings (aka "meetups") consists almost exclusively of presentations. Although we all understand the value of having an expert share expertise with the rest of us, our cultural bias towards one-to-many communication results in many lost opportunities for cross-fertilization of knowledge and cultivation of connections among the group.

In contrast, a many-to-many medium of communication such as a discussion or clinic provides an environment in which knowledge, ideas, and solutions can emanate from, and be received by, anyone.

I am not advocating replacing presentations with discussions. Rather, I am advocating an optimal balance of the two. Where this optimal point falls will depend on the group.

We are all familiar with presentations, so I will mainly talk about discussions and clinics here, and what they offer that presentations do not. First I will talk about discussions in general, and then clinics in particular.

Multiple Sources of Information

It's rare that a single person possesses all the knowledge in the room. When anyone can speak, we are more likely to get more high quality information. (Of course, we are also more likely to get more low quality information but the group can usually recognize and neutralize it.)

Dialog vs. Monologue

In a presentation, the information is assumed to be flowing in one direction, from the speaker to everyone else. Any discussion is normally for clarification of that unidirectional information flow.

In contrast, in a more conversational medium, people
are more likely to participate, resulting in greater depth and breadth of the information flow, as ideas bounce from person to person.

Synergy

There is a synergy in discussions that is absent in presentations. In discussions, one person's contribution can raise other questions or responses that can provide better information or lead the discussion to a valuable place that would not have otherwise been visited.

Greater Variety of Subject Matter

Even if a meetup includes two presentations, the topics covered are narrow. Including discussions even for, say, fifteen minutes, makes it more likely that people will find something in the meetup that is more relevant, and therefore more interesting, to them. Imagine, for example, the beginner who attends their first meetup, encounters specialized presentations that they do not understand, and then never returns, concluding that the meetings are "not for them".

Community Development

People who may have not uttered a word in months of presentations may feel more comfortable doing so in a discussion. They should be actively encouraged, even if it's to ask a question.

There will likely be "wizards" in the group, members who are recognizably much more knowledgeable than the others. It is natural for the others to feel intimidated and refrain from speaking. Wizards should step out of the way frequently, and the facilitator should encourage everyone to take part.

From discussion springs familiarity, and from familiarity springs friendship and community (not to mention jobs sometimes). It even enhances productivity, in that a relationship has formed such that people will feel more comfortable consulting each other outside of the discussions.

Clinics, in Particular

The clinic is a specialized form of discussion where people share their problems, challenges, and discoveries.

The key benefit of the clinic is that participants are likely to receive the information that is most valuable and helpful to them at the time.

In addition, the entire group is involved in finding the solution. Everyone is "on call". This can result in greater engagement and energy.

I describe at the bottom of this article my experience facilitating an internal organization Ruby clinic.^[1]

Structure

The clinic can be totally free form, but I recommend a meta-discussion at the beginning where people describe the topics they would like to discuss, and the group plans the discussion schedule. This is kind of like the opening of an open space conference but on a micro scale.

Advance Notice of Topics

It can be helpful for the group to propose topics in advance of the meetup. This has the following benefits:

ambiguities can be resolved before the meetup
some of the session planning can be done in advance
people can research the topics in advance
opportunities to refine or combine topics would be identified earlier

Infrastructure

It's helpful to have a shared screen. This is automatic if you are using collaboration software, but if you're in a physical room together, you'll need a projector or large shared monitor to be most productive.

Whether you have a single machine which will be shared by everyone, or everyone uses their own computer, I recommend bringing a wireless keyboard and mouse to the clinic; it lowers the barrier for other people to take the driver seat.

Level of Effort

One of many great things about clinics is that they require zero preparation. The group may want to propose topics in advance, but it's not absolutely necessary.

Frequency

At minimum, clinics can be used as a fallback activity when unable to find a speaker. However, I would argue that having a clinic for part of every meeting would be valuable and interesting for most groups.

Duration

There is no real minimum time, nor a need for its own sake that the time be consistent from meeting to meeting. For example, if a meeting normally has two presentations, the clinic could be limited to fifteen minutes. If there is only one presentation, the clinic could be expanded to fill that missing speaker's time slot.

Clinics for Dev Teams

Clinics aren't good only for meetups -- they can be even more valuable for development teams, where in addition to the shared languages and frameworks, they work with the same organizational code base and environment.

Cautions and Counterpoints

There is a risk that the clinics would gravitate towards beginner issues that would bore the more advanced users, or advanced subjects that would be incomprehensible to the more beginning users. Facilitators need to keep a close eye on this. If it's a short clinic this might be fine, but if it's a substantial portion of the meeting, it could drive people away. Other options could partially replace the clinic, such as a supplemental beginners meeting once a month.

The topics chosen (if there are enough to select from among them) could be selected to balance against the presentation. For example, if the presentation is very advanced, then the the more beginning clinic topics might be selected for discussion.

Darin Wilson points out: "The Q&A format can be good, but I think you have to put some effort into managing group dynamics.
The trouble with any Q&A session is that they self-select for folks who are comfortable speaking out loud in a group setting. The larger the group, the more pronounced this effect will be. Also, if you have one or two individuals who really like to hear themselves talk, they can end up dominating the discussion.
If you have a smallish group that already knows each other, this can be easier to manage. But if you have a larger group and/or folks don’t know each other very well, the moderator will probably need to make some extra effort to make sure that everyone feels welcome to participate, and that the discussion isn’t dominated by the over-talkers."

I agree that this is definitely something to look out for, but more applicable to general discussions than clinics. If the goal is to provide solutions, then there may indeed be a couple of experts who possess most of the relevant information. The facilitator needs to balance the need for solutions with the need to cultivate universal participation.

Joseph Glass says: "I’ve found clinics to be very engaging but they also require more “crowd management”. With a well-prepared presentation, the quality largely depends on the speaker, and will be pretty much the same wherever it's presented.

"Clinics on the other hand depend on the quality of the participants. It might turn out to be a fantastic conversation of people sharing expertise and forming connections. Or maybe one over-talkative participant will hijack the conversation into irrelevant tangents. It can be a bit of a gamble unless you know the community you’re working with or have a really good moderator."

Indeed, a really good moderator is the key. Perhaps that is a whole 'nother article. Anyone want to write it? :)

Philippines Ruby Meetup, January 2020

I suggested experimenting with this clinic format to the organizers of the Philippines Ruby Users Group. They agreed, and roughly half the meetup was allocated for it.

It turned out that only one person had a question, and several people answered it in complementary ways. Then there were no more questions. What to do? We morphed it into a general discussion. It was all engaging and interesting. Sometimes things don't go the way we expect. That's ok, we can adapt.

Conclusion

I describe in this article a formula that has worked for me, but of course there is an infinite number of combinations of attributes of a clinic, and your optimal mix may differ from mine. Any discussion can be engaging and informative, but for me the essentials of a clinic are:

help people get the information they most need at the time
strive for universal and balanced participation
balance the mix of levels of expertise for the questions within the clinic, and with the presentations

Presentations definitely have their place. There really is value in someone who has deeply researched a topic sharing their knowledge with the group. However, meetups can be improved when the one-to-many information flow is supplemented by the many-to-many flow provided by discussions in general, and clinics in particular.

Footnotes:

[1]A few years ago I was working at a network services company. At the time, they were automating their software testing to use Ruby and Cucumber. There were whole teams of software testers new to those technologies, and even programming in general. They were provided training, but after that training they were on their own. In addition to my own assignments to write software, one of my tasks was to help them get more productive in their programming.

They would sometimes come to me when stuck, and when they did, I dropped what I was doing. It felt great to help them, but I knew they needed more than just firefighting to be really productive. I was unable to get authorization for us to devote the time we needed for more intensive collaboration, so I scheduled weekly Ruby lunchtime "clinics" on our own time. People shared their challenges, questions, and conquests. We met in a meeting room with a large shared monitor. I brought an external keyboard and mouse with me for use by whomever was "driving". At the beginning we had a discussion of topics to cover. That gave me the information I needed to optimize the use of our hour together.

I encouraged them to show things they had learned, and to help each other. As time went on, their knowledge grew, and to my delight this happened more and more often. These sessions were my fondest memories of working there.

This article may be improved over time. To see its revisions you can go to its Github commit history.

The Case for Stabby Lambda Notation

Keith Bennett — Mon, 02 Dec 2019 14:45:32 +0000

The Stabby Lambda (`->`)

Although the -> "stabby lambda" notation has been available for creating lambdas since Ruby version 1.9, old habits die hard and acceptance and adoption has been slow. In this article I will explain why I recommend using it instead of the lambda notation.

Stabby Notation as an Indicator of Preferred and Default Proc Type

In a previous article, "lambdas Are Better Than procs", I proposed that lambdas should be used rather than procs in almost all cases, given that they are safer in terms of argument count checking and return behavior.

So it makes sense that -> should create a lambda and not a proc. (As an aside, it always puzzles me when people use the term stabby proc, when it creates a lambda.)

One way to look at it is, by using the stabby lambda notation, we are
saying "make me Ruby's implementation of an objectless function". This is at a level higher than "make me a lambda" or "make me a proc", and is probably a better interface to the programmer, especially the newer Rubyist.

`->`'s Picture-Like Notation

The picture-like notation -> is quite different from the lambda and proc forms, because although all result in method calls that create Proc instances, lambda and proc look like method calls, while -> does not, instead appearing more like a language construct. On the higher level, it really is a language construct, and the fact that a method needs to be called to create a lambda is an implementation detail that should not matter to the programmer.

The striking appearance of -> says to the reader "take note, something different is happening here, this marks the beginning of a definition of executable code that will probably be called somewhere else". If a picture is worth a thousand words, then a text picture like -> is worth, well, at least ten.

The Need for Visual Differentiation

Unlike other code in a method, a lambda's code is not called in sequence (unless it is immediately called as a self invoking anonymous function, but this is rare). Also, sometimes a lambda can be used as if it were a nested method, containing lower level code that may be called multiple times in the method in which it was defined. For these reasons, a pictorial indication setting it apart from other code in the method is especially helpful.

Rubocop

Rubocop is a very useful tool for normalizing code style. For better or worse though, Rubocop's defaults constitute implicit recommendations, and deviating from the defaults can require lengthy and contentious team discussions. Because of this potentially high cost of overriding the defaults, it is important that the basis in reasoning for the selection of the default be sound.

Rubocop's default setting for lambdas is to use -> with lambda one-liners but lambda for multiline lambdas. While this is not a matter of monumental importance, I believe it's misguided and should be changed.

My guess is that it is intended to mirror the Ruby code block notation convention of {..} for single line blocks and do...end for multi-line blocks. However, the code block case is different because the do and end are at the end and beginning of the line, respectively (though it is true that if there are arguments they will appear after the do). Although the indentation of the code block within the lambda do...end makes it easy to see that something is going on, it is easy to miss the lambda and assume it is a normal code block. The pictorial nature of -> reduces this risk.

I believe that the Rubocop default should be changed to prefer (or at minimum permit) -> in all cases.

Note: Since writing this article I posted an issue on the Rubocop project site here.

Conclusion

Lambdas are, thankfully, first class objects in Ruby. That is, they can be passed to and returned from methods, and can be assigned to variables. This is a pretty major construct, and I believe a special notation (->), rather than a method name (lambda) is justified and helpful. While it is true that class, module, and def also mark the beginning of major language constructs, they are likely to be the first token on a line, whereas lambdas are usually assigned to variables or passed to methods or other lambdas, and are not.

The conciseness and pictorial nature of -> encourage the use of lambdas, and in my opinion, that is a Good Thing. Lambdas are underused in the Ruby community, and many opportunities for cleaner and clearer code are missed.

This article may be improved over time. To see its revisions you can go to its Github commit history.

lambdas Are Better Than procs

Keith Bennett — Wed, 20 Nov 2019 09:53:12 +0000

Many Rubyists believe that lambda and nonlambda Procs are pretty much the same and that choosing which one to use is a subjective preference. This is an unfortunate fallacy.

This article will attempt to achieve two purposes:

1) to explain the difference between lambdas and procs

2) to persuade you to use lambdas unless there is a compelling reason not to

There are many resources available that explain lambdas and procs ^[1], and I will assume you know at least a little about them.

Before we look at some examples, here are some characteristics of lambdas and procs:

a Proc instance can be either lambda or a proc ^[2]
all lambdas are Procs
all procs are Procs
code blocks behave like procs
you can determine the kind of Proc by calling lambda? on it

Arity (Argument Count) Checking Behavior Differences

A lambda, like a method, strictly enforces its argument count, but a proc does not. When we call a proc with the wrong number of arguments, there are no complaints by the Ruby runtime ^[3]:

2.6.5 :006 > pfn = proc { |arg| }
 => #<Proc:0x00007f93828bd298@(irb):6>
2.6.5 :007 > pfn.call
 => nil

In contrast, when we do the same with a lambda, we get an error ^[4]:

2.6.5 :002 > lfn = ->(arg) {}
    => #<Proc:0x00007f9383118ed8@(irb):2 (lambda)>
   2.6.5 :003 > lfn.call
   ...
   ArgumentError (wrong number of arguments (given 0, expected 1))

Which behavior would you prefer?

Clearly, arity checking is helpful, and we abandon it at our peril.

Return Behavior Differences

What happens when you pass a code block somewhere, and it executes a return? Does it return from the block? Well, yes, but it does much more than that; it returns from the method that yielded to the block. procs behave the same way; in addition to returning from themselves, they will return from the method in which they were called:

def using_proc
  pfn = proc { return }
  puts "Before calling"
  pfn.call
  puts "After calling"
end

# ...
2.6.5 :015 > using_proc
Before calling

Before proceeding to the lambda behavior, I'd like to point out that this proc behavior is such that implicit and explicit returns do very different things. An implicit return will return from the proc, but an explicit return will return from the context that called it! Weird, eh? Here is the same code, but without the explicit return; the proc will end and exit naturally:

def using_proc_without_return
  pfn = proc { }
  puts "Before calling"
  pfn.call
  puts "After calling"
end
# ...
2.6.5 :007 > using_proc_without_return
Before calling
After calling

When we first learn Ruby, we learn that a return at the end of a method is redundant (it is, of course), but in the case of the proc (and code block) it is not!

In contrast, a lambda's return returns from itself to the context that called it:

def using_lambda
  lfn = -> { return }
  puts "Before calling"
  lfn.call
  puts "After calling"
end
# ...
2.6.5 :008 > using_lambda
Before calling
After calling

A `lambda` is More Method-Like Than a `proc`

In both of the above cases, the lambda behaves more like a method than a proc does. The newer ->(args) notation for creating a lambda reveals that intent by defining the arguments as a method does, in a parenthesized list, and is therefore preferable to the older lambda notation:

New:

fn = ->(arg1, arg2) { ... }

Old:

fn = lambda { |arg1, arg2| ... }

Conclusion

Here are some principles I've learned to code by:

prefer simplicity to complexity
limit things to the narrowest possible scope
specify things with minimal ambiguity
use language features that minimize the risk of errors

Regarding everything said so far, the lambda wins over the proc. There is no reason to use a proc unless you specifically need the odd and potentially hazardous behaviors described above.

You may think it will never matter in your case. Maybe you're never calling the lambda yourself, but passing it to a framework such as Rails that is doing all the calling. Nevertheless, if given the added protection for free, why would you not want it? Especially since the -> notation is somewhat pictorial and more concise?

Footnotes

[1] There are many good resources; here are some that I have produced (articles and a conference talk):

Using Lambdas to Simplify Varying Behaviors in Your Code
Ruby Enumerables Make Your Code Short and Sweet
Functional Programming in Ruby, video of a talk given at Functional Conf in Bangalore, India in 2014

↩

[2] This terminology is unfortunate, as Proc and proc, when spoken, sound identical.
↩

[3] In this article I've used the .call variant of calling a Proc because it is the most obvious for the reader, but in practice I prefer the shorthand notation .().
↩

[4] The shorthand -> can be used in place of the lambda keyword to more succinctly define a lambda.
↩

This article may be improved over time. To see its revisions you can go to its Github commit history.

Using Lambdas to Simplify Varying Behaviors in Your Code

Keith Bennett — Mon, 18 Nov 2019 10:10:45 +0000

(This article was first posted on my personal blog on 2018-08-04 at https://bbs-software.com/blog/2018/08/04/using-lambdas-to-simplify-varying-behaviors-in-your-code.html.)

The Problem with Non-Lambda Approaches

The procedural if-elsif-end or case clauses work when you have a small number of conditions and actions that are known in advance, but if you don't, they're pretty useless.

And although the object oriented approach of polymorphism by inheritance (2) can produce a correct result, in many cases it is unnecessarily verbose, ceremonial, and awkward.

Furthermore, though we're accustomed to thinking about this problem in the context of a single customizable behavior, what if there are several?

A better solution is using callables such as lambdas.

Callables as a Superset of Lambdas

This means that in Ruby, since the method used to call a lambda is `call`, any object that responds to `call` can be used in place of a lambda.

Since any object responding to call can be used in place of a lambda, I will use the term callable instead of lambda where applicable.

The Buffered Enumerable

To create an instance of this with callables, we call the BufferedEnumerable class method create_with_callables, which is defined as follows(4):

 def self.create_with_callables(chunk_size, fetcher, fetch_notifier = nil)
    instance = self.new(chunk_size)
    instance.fetcher = fetcher
    instance.fetch_notifier = fetch_notifier
    instance
  end

fetcher.(data, chunk_size)

(In Ruby, .( is an abbreviation for .call(.)

A trivial fetcher that merely fills the array of the requested chunk size with random numbers could look like this:

fetcher = ->(data, chunk_size) do
  chunk_size.times { |index| data[index] = Random.rand }
end

Below is a pry example that illustrates the call to that fetcher, and its effect on the passed array:

[7] pry("")> a = []; fetcher.(a, 2)
2
[8] pry("")> a
[
    [0] 0.4885287976297428,
    [1] 0.5493143769524284
]

After the buffer is filled, if a fetch notifier callable has been defined (unlike the fetcher, this is optional), it too is called, with the data buffer:

fetch_notifier.(data)

A trivial fetch notifier might look like this:

->(data) { puts "#{Time.now} Fetched #{data.size} objects" }

(data.size will not necessarily be equal to chunk_size, especially on the last fetch.)

This notifier might produce something looking like this:

2018-07-26 17:19:47 +0700 Fetched 1000 objects

After defining the fetcher and fetch_notifier lambdas, we could call the class method for creating a BufferedEnumerable shown above as follows:

buffered_enumerable = BufferedEnumerable.create_with_callables( \
    1000, fetcher, fetch_notifier)

This enumerable can be used to call each or any other of the rich set of methods available on the Enumerable module.

Using Predicate Callables to Implement Filters

Configurable predicates are another natural fit for using callables.

Both cases were an excellent fit for using callables as filters.

MockDnsServer::PredicateFactory.new.qname('example.com')

The qname method (i.e. the method that returns a filter for exactly one qname value) is defined as (roughly):

def qname(qname)
  ->(message, _protocol = nil) do
    eq_case_insensitive(message.qname, qname)
  end
end

Does storing state in the lambda make it any less functional? Not really; the state is immutable and used only for comparison.

pf = MockDnsServer::PredicateFactory.new
filter = pf.all(
    pf.qtype('A'),
    pf.qname('example.com'),
    pf.from_tcp
)

The all compound filter is nothing more than a simple wrapper around Ruby's Enumerable's all? method:

  def all(*predicates)
    ->(message, protocol = nil) do
      predicates.all? { |p| p.call(message, protocol) }
    end
  end

def from_tcp
  ->(_message, protocol) { protocol == :tcp }
end

In this case, we only care about the protocol so we can ignore the first (message) parameter.

Conclusion

I hope I have been successful in persuading you to consider using callables for implementing variable predicates and actions.

These are just two examples. I will stop here for the sake of brevity, but if you have any questions or suggestions for elaboration, please contact me.

You can find me on:

Also, for more information about lambdas, feel free to check out my Ruby lambdas presentation (Speakerdeck slideshow and YouTube video).

Footnotes

(1) One might ask what else is there about lambdas other than varying behavior, but there are other reasons to use them in Ruby. For example, they can be used:

as nested functions
as truly private functions (which, unlike private methods, cannot be accessed with send
for self invoking anonymous functions, to hide variables
for defining functions where methods cannot be defined (e.g. in some RSpec code)
for defining classes dynamically with the class...end notation which may be easier to understand than the Class.new approach.
for chaining operations, as in an array of lambdas

(2 (cont'd)) @kayess points out that the more precise terms in software engineering are nominal typing for polymorphism by inheritance, and structural typing for duck typing.

This article may be improved over time. To see its revisions you can go to its Github commit history.

Ruby Enumerables Make Your Code Short and Sweet

Keith Bennett — Wed, 13 Nov 2019 13:52:20 +0000

One of the most amazing things about Ruby is the richness of its Enumerable library; there are so many things it can do. Another is Ruby's ability to express intent with the utmost conciseness and clarity. However, out in the wild I very often see code that fails to take full advantage of these qualities.

As a contrived example, let's say we're keeping track of letter frequencies in a document. We define a class to contain them as:

LetterFrequency = Struct.new(:letter, :frequency, :vowel?)

I've seen a lot code that looks like this:

def filtered_and_transformed_records_1(records)
  results = []

  records.each do |record|
    next unless record.vowel?
    results << [record.letter, record.frequency]
  end

  results
end

In more primitive languages one must use these approaches, but in Ruby we have some major refactorings that can make this code much, much simpler.

First, we can use each_with_object to eliminate the need for the explicit initialization of the function-local variable containing the array and its explicit return, on the first and last lines of the method:

def filtered_and_transformed_records_2(records)
  records.each_with_object([]) do |record, results|
    next unless record.vowel?
    results << [record.letter, record.frequency]
  end
end

I say function-local because we do need the block-local variable results inside the each_with_object block. However, we've narrowed the scope of the results variable, and that's always a good thing.

each_with_object is like each except that it will pass two variables to the block instead of one. In addition to the object from the Enumerable that each passes, it passes the object you are using to accumulate results. You initialize the accumulator by passing its initial value to the each_with_object method. In this case we are passing a newly created empty array.

each_with_object's return value is the accumulator object, so you don't need to specify the accumulator explicitly for it to be the value returned by the method.

The each_with_object usage may not feel natural at first, but once you've seen it a few times your mind will parse it with almost zero effort. (By the way, I always had trouble remembering the order of its arguments until I realized that they were in the same order as in the method name itself; each for the enumerated object and object for the accumulator object.)

The second refactoring is instead of using control flow constructs like next, we can use the Enumerable methods select or reject. We could refactor the code further into:

def filtered_and_transformed_records_3(records)
  records.select(&:vowel?).each_with_object([]) do |record, results|
    results << [record.letter, record.frequency]
  end
end

After this refactoring, we see the filter where it is more appropriate and helpful. Instead of it being on a line inside the block, it's just a few characters immediately after the input array (records.select...).

We've already simplified this method quite a bit, but there's even more we can do. Because select returns the filtered array, we can simplify even further by using map instead of each_with_object!:

def filtered_and_transformed_records_4(records)
  records.select(&:vowel?).map { |record| [record.letter, record.frequency] }
end

Although as software developers our mission is to deliver functionality, the other side of that coin is to do so as simply as possible. Put otherwise, we need to remove accidental complexity (a.k.a. incidental complexity) so that only the essential complexity remains. The functional approaches described here are extremely effective at doing this. We've ended up with a simple one-liner.

Whenever you start feeling that your code is getting verbose or awkward, ask yourself "could I improve this code with Enumerable?" The answer may well be yes.

For your reference, here is a file that contains the methods in the article, and verifies that they all produce the same result.

[Note: This article may occasionally be improved. Its commit history is here.]

Automate the Little Things Too

Keith Bennett — Tue, 12 Nov 2019 10:46:08 +0000

(This article was originally published at on 2019-09-24 at:
https://bbs-software.com/blog/2019/09/24/automate-the-little-things-too.html)

We developers automate highly complex tasks, but when it comes to the smaller repetitive tasks, we tend to do things manually, or fail to do them at all. By combining Ruby with robust and richly functional command line tools such as MPlayer, we can save ourselves lots of time and have fun in the process.

I recently decided that it would be nice to trim my collection of many video files downloaded from my phones over the years. Realizing this would be quite tedious, I asked myself "would this be easier with Ruby?" The answer, of course, was Yes!

Integrating MPlayer and Ruby

MPlayer is a Unix command line multimedia player that can be installed with your favorite package manager (e.g. brew, apt, or yum). By driving MPlayer from Ruby, we can create a workflow that will enable you to view and decide about video files with a minimum of keystrokes, without needing to use the mouse.

Files to process are specified on the command line. Multiple arguments can be specified, either absolute or relative, and either with or without wildcards. All filespecs are normalized to their absolute form so that duplicates can be eliminated.

MPlayer plays each file for the user, responding to cursor keys to move forward and backward in time, change the speed, etc. I recommend viewing the man page (man mplayer), but here are the most relevant options:

keyboard control
      LEFT and RIGHT
           Seek backward/forward 10 seconds.
      UP and DOWN
           Seek forward/backward 1 minute.
      PGUP and PGDWN
           Seek forward/backward 10 minutes.
      [ and ]
           Decrease/increase current playback speed by 10%.
      { and }
           Halve/double current playback speed.
      BACKSPACE
           Reset playback speed to normal.

When the user has seen enough to make a decision, q or [ESC] can be pressed, and MPlayer returns control to the Ruby script, which accepts a one character response to mark it to be saved (s), deleted (d), or marked as undecided (u) for future reprocessing; or q to quit the application.

The High Level View

Here is the highest level method in the script:

def main
  check_presence_of_mplayer
  create_dirs
  puts greeting
  files_to_process.each do |filespec|
    play_file(filespec)
    print disposition_prompt(filespec)
    destination_subdir = get_disposition_from_user
    `mv #{filespec} #{destination_subdir}`
    log(filespec, destination_subdir)
  end
end

Using Subdirectories

For simplicity of implementation and added safety, the application "marks" each multimedia file by moving it to one of the three subdirectories it has created, based on the user's choice. The user selects d for deletes, s for saves, or u for undecideds. create_dirs creates the three subdirectories:

def create_dirs
  %w{deletes  saves  undecideds}.each { |dir| FileUtils.mkdir_p(dir) }
end

When the user is finished processing all files, they will probably want to move any files that have been moved to ./undecided back to . and run the program again.

Finally, when there are no files left in undecided, one will probably want to do something like this:

rmdir undecideds
rm -rf deletes
mv saves/* .
rmdir saves

An Example

For example, let's say you run the following command:

organize-av-files 'video/*mp4' 'audio/*mp3'

MPlayer will begin playing the first file. When you are ready to finish viewing it, you will press q or ESC, and be presented with a prompt like this:

/Users/kbennett/android/video/20160102_234426.mp4:
s = save, d = delete, u = undecided, q = quit:

Type your response choice and then [Enter]. The program will move the file as appropriate, and immediately start playing the next file.

Shell vs. Ruby Wildcard Expansion

Be careful when using wildcards. If you enter *mp4 in a directory with 10,000 MP4 files, the shell will try to expand it into 10,000 arguments, which might exceed the maximum command line size and result in an error. You can instead quote the filemask (as '*mp4'), and it will then be passed to Ruby as a single argument, and Ruby will perform the expansion. You can usually use double quotes, but be aware that the single and double quotes behavior differs (see this helpful StackOverflow article).

One case where the shell's expansion would be preferable is with the use of environment variables in the filespec ($FOO is more concise than ENV['FOO']), and in the case of using ~ for users other than the current user (e.g. ~someoneelse).

Also...

This workflow can be used with any multimedia files recognized by MPlayer, and that includes audio files.
There are many, many nice-to-have features that have not been implemented, since speed of implementation was a high priority. Feel free to add your own!
Although using Ruby probably enables writing the most concise and intention-revealing code, other languages such as Python would do fine as well.
The code for this script ("organize-av-files") is currently at https://gist.github.com/keithrbennett/4d9953e66ea35e2c52abae52650ebb1b.

Conclusion

I hope you can see that with a modest amount of code you can build a highly useful (albeit not fancy) automation tool. The amount of expected use and the benefit per use determines the optimum amount of effort, and you have the freedom to choose any point in that continuum. The notion that all applications need to be feature-rich is not a useful one, and often results in inaction altogether.

Ruby is a great tool for this sort of thing. Why not use it?

--- The End ---

[Note: This article is occasionally improved. Its commit history is here.]

For your convenience, the script is displayed below:

#!/usr/bin/env ruby

# organize-av-files - Organizes files playable by mplayer
# into 'saves', 'deletes', and 'undecideds' subdirectories
# of the current working directory.
#
# Be careful, if you specify files to process in multiple directories,
# they will all be moved to the same subdirectories, so they will no
# longer be organized by directory, and if there are multiple files
# of the same name, some may be lost if overwritten.
#
# stored at:
# https://gist.github.com/keithrbennett/4d9953e66ea35e2c52abae52650ebb1b


require 'date'
require 'fileutils'
require 'set'

LOG_FILESPEC = 'organize-av-files.log'

def create_dirs
  %w{deletes  saves  undecideds}.each { |dir| FileUtils.mkdir_p(dir) }
end


def check_presence_of_mplayer
  if `which mplayer`.chomp.size == 0
    raise "mplayer not detected. "
        "Please install it (with apt, brew, yum, etc.)"
  end
end


# Takes all ARGV elements, expands any wildcards,
# converts to normalized (absolute) form,
# and eliminates duplicates.
def files_to_process

  # Dir[] does not understand ~, need to process it ourselves.
  # This does *not* handle the `~username` form.
  replace_tilde_if_needed = ->(filespec) do
    filespec.start_with?('~/')                    \
        ? File.join(ENV['HOME'], filespec[2, -1]) \
        : filespec
  end

  # When Dir[] gets a directory it returns no files.
  # Need to add '/*' to it.
  add_star_to_dirspec_if_needed = ->(filespec) do
    File.directory?(filespec)      \
        ? File.join(filespec, '*') \
        : filespec
  end

  # Default to all nonhidden files in current directory
  # but not its subdirectories.
  ARGV[0] ||= '*'

  all_filespecs = ARGV.each_with_object(Set.new) do |filemask, all_filespecs|
    filemask = replace_tilde_if_needed.(filemask)
    filemask = add_star_to_dirspec_if_needed.(filemask)

    Dir[filemask]                          \
        .map { |f| File.absolute_path(f) } \
        .select { |f| File.file?(f) }      \
        .each do |filespec|
      all_filespecs << filespec
    end
  end
  all_filespecs.sort
end


def greeting
  puts <<~GREETING
      organize-av-files

      Enables the vetting of audio and video files. 

      For each file, plays it with mplayer, and prompts for what you would like to do 
      with that file, moving the file to one of the following subdirectories:

      * deletes
      * saves
      * undecideds

      This software uses mplayer to play audio files. Use cursor keys to move forwards/backwards in time.
      Press 'q' or 'ESC' to abort playback and specify disposition of that file.

      Run `man mplayer` for more on mplayer.

      Assumes all files specified are playable by mplayer.
      Creates subdirectories in the current directory: deletes, saves, undecideds.
      Logs to file '#{LOG_FILESPEC}'

  GREETING
end


def play_file(filespec)
  # If you have mplayer problems, remove the redirection ("2> /dev/null")
  # to see any errors.
  `mplayer #{filespec} 2> /dev/null`
end


def disposition_prompt(filespec)
  "\n\n#{filespec}:\ns = save, d = delete, u = undecided, q = quit: "
end


def get_disposition_from_user
  loop do
    response = $stdin.gets.chomp.downcase

    if response == 'q'
      exit
    elsif %w(s d u).include?(response)
      return {
          's' => 'saves',
          'd' => 'deletes',
          'u' => 'undecideds'
      }[response]
    else
      print "s = save, d = delete, u = undecided, q = quit: "
    end
  end
end


def log(filespec, destination_subdir)
  dest_abbrev = destination_subdir[0].upcase # 'S' for saves, etc.
  log_message = "#{dest_abbrev}  #{Time.now}  #{filespec}"
  `echo #{log_message} >> #{LOG_FILESPEC}`
end


def main
  check_presence_of_mplayer
  create_dirs
  puts greeting
  files_to_process.each do |filespec|
    play_file(filespec)
    print disposition_prompt(filespec)
    destination_subdir = get_disposition_from_user
    `mv #{filespec} #{destination_subdir}`
    log(filespec, destination_subdir)
  end
end


main

Boost Your Shell Scripting with Ruby and Rexe

Keith Bennett — Mon, 15 Apr 2019 06:11:54 +0000

[Caution: This is a long article!
Sections are mostly self contained, so feel free to skim or skip them.]

Rexe is a Ruby script and gem that multiplies Ruby's usefulness and conciseness on the command line by:

automating parsing and formatting using JSON, YAML, Ruby marshalling, Awesome Print, and others
simplifying the use of Ruby as a shell filter, optionally predigesting input as lines, an enumerator, or one big string
extracting the plumbing from the command line; requires and other options can be set in an environment variable
enabling the loading of Ruby helper files to keep your code DRY and your command line code high level
reading and evaluating a ~/.rexerc file on startup for your shared custom code and common requires

Shell scripting is great for simple tasks but for anything nontrivial it can easily get cryptic and awkward (pun intended!).

This problem can often be solved by writing a Ruby script instead. Ruby provides fine grained control in a language that is all about clarity, conciseness, and expressiveness.

Unfortunately, when there are multiple OS commands to be called, then Ruby can be awkward too.

Sometimes a good solution is to combine Ruby and shell scripting on the same command line. Rexe multiplies your power to do so.

Using the Ruby Interpreter on the Command Line

Let's start by seeing what the Ruby interpreter already provides. Here we use ruby on the command line, using an intermediate environment variable to simplify the logic and save the data for use by future commands. An excerpt of the output follows the code:

➜  ~   export EUR_RATES_JSON=`curl https://api.exchangeratesapi.io/latest`
➜  ~   echo $EUR_RATES_JSON | ruby -r json -r yaml -e 'puts JSON.parse(STDIN.read).to_yaml'

---
rates:
  MXN: 21.96
  AUD: 1.5964
  HKD: 8.8092
  ...
base: EUR
date: '2019-03-08'

Unfortunately, the configuration setup (the requires) along with the reading, parsing, and formatting make the command long and tedious, discouraging this approach.

Rexe

Rexe [see footnote ^1 regarding its origin] can simplify such commands. Among other things, rexe provides switch-activated input parsing and output formatting so that converting from one format to another is trivial. The previous ruby command can be expressed in rexe as:

➜  ~   echo $EUR_RATES_JSON | rexe -mb -ij -oy self

Or, even more concisely (self is the default Ruby source code for rexe commands):

➜  ~   echo $EUR_RATES_JSON | rexe -mb -ij -oy

The command options may seem cryptic, but they're logical so it shouldn't take long to learn them:

-mb - mode to consume all standard input as a single big string
-ij - parse that input with JSON; self will be the parsed object
-oy - output the final value as YAML

If input comes from a JSON or YAML file, rexe determines the input format from the file's extension, and it's even simpler:

➜  ~   rexe -f eur_rates.json -oy

Rexe is at https://github.com/keithrbennett/rexe and can be installed with gem install rexe. Rexe provides several ways to simplify Ruby on the command line, tipping the scale so that it is practical to do it more often.

Here is rexe's help text as of the time of this writing:

rexe -- Ruby Command Line Executor/Filter -- v1.0.1 -- https://github.com/keithrbennett/rexe

Executes Ruby code on the command line, 
optionally automating management of standard input and standard output,
and optionally parsing input and formatting output with YAML, JSON, etc.

rexe [options] [Ruby source code]

Options:

-c  --clear_options        Clear all previous command line options specified up to now
-f  --input_file           Use this file instead of stdin for preprocessed input; 
                           if filespec has a YAML and JSON file extension,
                           sets input format accordingly and sets input mode to -mb
-g  --log_format FORMAT    Log format, logs to stderr, defaults to none
                           (see -o for format options)
-h, --help                 Print help and exit
-i, --input_format FORMAT  Input format
                             -ij  JSON
                             -im  Marshal
                             -in  None (default)
                             -iy  YAML
-l, --load RUBY_FILE(S)    Ruby file(s) to load, comma separated;
                             ! to clear all, or precede a name with '-' to remove
-m, --input_mode MODE      Input preprocessing mode (determines what `self` will be):
                             -ml  line; each line is ingested as a separate string
                             -me  enumerator (each_line on STDIN or File)
                             -mb  big string; all lines combined into one string
                             -mn  none (default); no input preprocessing; 
                                  self is an Object.new 
-n, --[no-]noop            Do not execute the code (useful with -g);
                           For true: yes, true, y, +; for false: no, false, n
-o, --output_format FORMAT Output format (defaults to puts):
                             -oi  Inspect
                             -oj  JSON
                             -oJ  Pretty JSON
                             -om  Marshal
                             -on  No Output (default)
                             -op  Puts
                             -os  to_s
                             -oy  YAML
-r, --require REQUIRE(S)   Gems and built-in libraries to require, comma separated;
                             ! to clear all, or precede a name with '-' to remove

---------------------------------------------------------------------------------------

In many cases you will need to enclose your source code in single or double quotes.

If source code is not specified, it will default to 'self', 
which is most likely useful only in a filter mode (-ml, -me, -mb).

If there is a .rexerc file in your home directory, it will be run as Ruby code 
before processing the input.

If there is a REXE_OPTIONS environment variable, its content will be prepended
to the command line so that you can specify options implicitly 
(e.g. `export REXE_OPTIONS="-r awesome_print,yaml"`)

Simplifying the Rexe Invocation

There are two main ways we can make the rexe command line even more concise:

by extracting configuration into the REXE_OPTIONS environment variable
by extracting low level and/or shared code into helper files that are loaded using -l, or implicitly with ~/.rexerc

The REXE_OPTIONS Environment Variable

The REXE_OPTIONS environment variable can contain command line options that would otherwise be specified on the rexe command line:

Instead of this:

➜  ~   rexe -r wifi-wand -oa WifiWand::MacOsModel.new.wifi_info

you can do this:

➜  ~   export REXE_OPTIONS="-r wifi-wand -oa"
➜  ~   rexe WifiWand::MacOsModel.new.wifi_info
➜  ~   # [more rexe commands with the same options]

Putting configuration options in REXE_OPTIONS effectively creates custom defaults, and is useful when you use the same options in most or all of your commands. Any options specified on the rexe command line will override the environment variable options.

Like any environment variable, REXE_OPTIONS could also be set in your startup script, input on a command line using export, or in another script loaded with source or ..

Loading Files

The environment variable approach works well for command line options, but what if we want to specify Ruby code (e.g. methods) that can be used by your rexe code?

For this, rexe lets you load Ruby files, using the -l option, or implicitly (without your specifying it) in the case of the ~/.rexerc file. Here is an example of something you might include in such a file:

# Open YouTube to Wagner's "Ride of the Valkyries"
def valkyries
  `open "http://www.youtube.com/watch?v=P73Z6291Pt8&t=0m28s"`
end

To digress a bit, why would you want this? You might want to be able to go to another room until a long job completes, and be notified when it is done. The valkyries method will launch a browser window pointed to Richard Wagner's "Ride of the Valkyries" starting at a lively point in the music [see footnote ^2 regarding autoplay]. (The open command is Mac specific and could be replaced with start on Windows, a browser command name, etc.) [see footnote ^3 regarding OS portability].

If you like this kind of audio notification, you could download public domain audio files and use a command like player like afplay on Mac OS, or mpg123 or ogg123 on Linux. This approach is lighter weight, requires no network access, and will not leave an open browser window for you to close.

Here is an example of how you might use the valkyries method, assuming the above configuration is loaded from your ~/.rexerc file or an explicitly loaded file:

➜  ~   tar czf /tmp/my-whole-user-space.tar.gz ~ ; rexe valkyries

(Note that ; is used rather than && because we want to hear the music whether or not the command succeeds.)

You might be thinking that creating an alias or a minimal shell script (instead of a Ruby script) for this open would be a simpler and more natural approach, and I would agree with you. However, over time the number of these could become unmanageable, whereas using Ruby you could build a pretty extensive and well organized library of functionality. Moreover, that functionality could be made available to all your Ruby code (for example, by putting it in a gem), and not just command line one liners.

For example, you could have something like this in a gem or loaded file:

def play(piece_code)
  pieces = {
    hallelujah: "https://www.youtube.com/watch?v=IUZEtVbJT5c&t=0m20s",
    valkyries:  "http://www.youtube.com/watch?v=P73Z6291Pt8&t=0m28s",
    wm_tell:    "https://www.youtube.com/watch?v=j3T8-aeOrbg&t=0m1s",
    # ... and many, many more
  }
  `open #{Shellwords.escape(pieces.fetch(piece_code))}`
end

...which you could then call like this:

➜  ~   tar czf /tmp/my-whole-user-space.tar.gz ~ ; rexe 'play(:hallelujah)'

(You need to quote the play call because otherwise the shell will process and remove the parentheses. Alternatively you could escape the parentheses with backslashes.)

One of the examples at the end of this articles shows how you could have different music play for success and failure.

Logging

A log entry is optionally output to standard error after completion of the code. This entry is a hash representation (to be precise, to_h) of the $RC OpenStruct described in the $RC section below. It contains the version, date/time of execution, source code to be evaluated, options (after parsing both the REXE_OPTIONS environment variable and the command line), and the execution time of your Ruby code:

➜  ~   echo $EUR_RATES_JSON | rexe -gy -ij -mb -oa -n self

---
:count: 0
:rexe_version: 1.0.0
:start_time: '2019-04-15T13:12:15+08:00'
:source_code: self
:options:
  :input_filespec:
  :input_format: :json
  :input_mode: :one_big_string
  :loads: []
  :output_format: :awesome_print
  :requires:
  - awesome_print
  - json
  - yaml
  :log_format: :yaml
  :noop: true
:duration_secs: 0.050326

We specified -gy for YAML format; there are other formats as well (see the help output or this document) and the default is -gn, which means don't output the log entry at all.

The requires you see were not explicitly specified but were automatically added because Rexe will add any requires needed for automatic parsing and formatting, and we specified those formats in the command line options -gy -ij -oa.

This extra output is sent to standard error (stderr) instead of standard output (stdout) so that it will not pollute the "real" data when stdout is piped to another command.

If you would like to append this informational output to a file(e.g. rexe.log), you could do something like this:

➜  ~   rexe ... -gy 2>>rexe.log

Input Modes

Rexe tries to make it simple and convenient for you to handle standard input, and in different ways. Here is the help text relating to input modes:

-m, --input_mode MODE      Input preprocessing mode (determines what `self` will be):
                             -ml  line; each line is ingested as a separate string
                             -me  enumerator (each_line on STDIN or File)
                             -mb  big string; all lines combined into one string
                             -mn  none (default); no input preprocessing; 
                                  self is an Object.new

The first three are filter modes; they make standard input available to your code as self.

The last (and default) is the executor mode. It merely assists you in executing the code you provide without any special implicit handling of standard input. Here is more detail on these modes:

-ml "Line" Filter Mode

In this mode, your code would be called once per line of input, and in each call, self would evaluate to each line of text:

➜  ~   echo "hello\ngoodbye" | rexe -ml puts reverse
olleh
eybdoog

reverse is implicitly called on each line of standard input. self is the input line in each call (we could also have used self.reverse but the self. would have been redundant).

Be aware that, in this mode, if you are using an automatic output mode (anything other than the default -on no output mode), although you can control the content of output records, there is no way to selectively exclude records from being output. Even if the result of the code is nil or the empty string, a newline will be output. To prevent this, you can do one of the following:

use -me Enumerator mode instead and call select, filter, reject, etc.
use the (default) -on no output mode and call puts explicitly for the output you do want

-me "Enumerator" Filter Mode

In this mode, your code is called only once, and self is an enumerator dispensing all lines of standard input. To be more precise, it is the enumerator returned by the each_line method, on $stdin or the input file, whichever is applicable.

Dealing with input as an enumerator enables you to use the wealth of Enumerable methods such as select, to_a, map, etc.

Here is an example of using -me to add line numbers to the first 3 files in the directory listing:

➜  ~   ls / | rexe -me "first(3).each_with_index { |ln,i| puts '%5d  %s' % [i, ln] }"

    0  AndroidStudioProjects
    1  Applications
    2  Desktop

Since self is an enumerable, we can call first on it. We've used the default output mode -on (no output mode), which says don't do any automatic output, just the output explicitly specified by puts in the source code.

-mb "Big String" Filter Mode

In this mode, all standard input is combined into a single (possibly large and possibly multiline) string.

A good example of when you would use this is when you need to parse a multiline JSON or YAML representation of an object; you need to pass all the standard input to the parse method. This is the mode that was used in the first rexe example in this article.

-mn "No Input" Executor Mode -- The Default

In this mode, no special handling of standard input is done at all; if you want standard input you need to code it yourself (e.g. with STDIN.read).

self evaluates to a new instance of Object, which would be used if you defined methods, constants, instance variables, etc., in your code.

Filter Input Mode Memory Considerations

If you are using one of the filter modes, and may have more input than would fit in memory, you can do one of the following:

use -ml (line) mode so you are fed only 1 line at a time
use an Enumerator, either by a) specifying the -me (enumerator) mode option, or b) using -mn (no input) mode in conjunction with something like STDIN.each_line. Then:
- Make sure not to call any methods (e.g. map, select) that will produce an array of all the input because that will pull all the records into memory, or:
- use lazy enumerators

Input Formats

Rexe can parse your input in any of several formats if you like. You would request this in the input format (-i) option. Legal values are:

-ij - JSON
-im - Marshal
-in - None
-iy - YAML

Except for -in, which passes the text to your code untouched, your input will be parsed in the specified format, and the resulting object passed into your code as self.

The input format option is ignored if the input mode is -mn ("no input" executor mode, the default), since there is no preprocessing of standard input in that mode.

Output Formats

Several output formats are provided for your convenience:

-oa - Awesome Print - calls .ai on the object to get the string that ap would print
-oi - Inspect - calls inspect on the object
-oj - JSON - calls to_json on the object
-oJ - Pretty JSON calls JSON.pretty_generate with the object
-on - (default) No Output - output is suppressed
-op - Puts - produces what puts would output
-os - To String - calls to_s on the object
-oy - YAML - calls to_yaml on the object

All formats will implicitly require anything needed to accomplish their task (e.g. require 'yaml').

The default is -on to produce no output at all (unless explicitly coded to do so). If you prefer a different default such as -op for puts mode, you can specify that in your REXE_OPTIONS environment variable.

If two letters are provided, the first will be used for tty devices (e.g. the terminal when not redirected or piped), and the second for block devices (e.g. when redirected or piped to another process).

You may wonder why these formats are provided, given that their functionality could be included in the custom code instead. Here's why:

The savings in command line length goes a long way to making these commands more readable and feasible.
It's much simpler to switch formats, as there is no need to change the code itself.
This approach enables parameterization of the output format.

Reading Input from a File

Rexe also simplifies getting input from a file rather than standard input. The -f option takes a filespec and does with its content exactly what it would have done with standard input. This shortens:

➜  ~   cat filename.ext | rexe ...

...to...

➜  ~   rexe -f filename.ext ...

This becomes even more useful if you are using files whose extensions are .yml, .yaml, or .json (case insensitively). In this case the input format and mode will be set automatically for you to:

-iy (YAML) or -ij (JSON) depending on the file extension
-mb (one big string mode), which assumes that the most common use case will be to parse the entire file at once

So the example we gave above:

➜  ~   export EUR_RATES_JSON=`curl https://api.exchangeratesapi.io/latest`
➜  ~   echo $EUR_RATES_JSON | rexe -mb -ij -oy self

...could be changed to:

➜  ~   curl https://api.exchangeratesapi.io/latest > eur_rates.json
➜  ~   rexe -f eur_rates.json -oy self

Another possible win for using -f is that since it is a command line option, it could be specified in REXE_OPTIONS. This could be useful if you are doing many operations on the same file.

If you need to override the input mode and format automatically configured for file input, you can simply specify the desired options on the command line after the -f:

➜  ~   rexe -f eur_rates.json -mb -in 'puts self.class, self[0..20]'
String
{"base":"EUR","rates"

'self' as Default Source Code

To make rexe even more concise, you do not need to specify any source code when you want that source code to be self. This would be the case for simple format conversions, as in JSON to YAML conversion mentioned above:

➜ ~  rexe -f eur_rates.json -oy
# or
➜ ~  echo $EUR_RATES_JSON | rexe -mb -ij -oy

---
rates:
  JPY: 126.63
  BRL: 4.3012
  NOK: 9.6915
  ...

This feature is probably only useful in the filter modes, since in the executor mode (-mn) self is a new instance of Object and hardly ever useful as an output value.

The $RC Global OpenStruct

For your convenience, the information displayed in verbose mode is available to your code at runtime by accessing the $RC global variable, which contains an OpenStruct. Let's print out its contents using YAML:

➜  ~   rexe -oy '$RC'

--- !ruby/object:OpenStruct
table:
  :count: 0
  :rexe_version: 1.0.0
  :start_time: '2019-04-15T13:25:56+08:00'
  :source_code: "$RC"
  :options:
    :input_filespec:
    :input_format: :none
    :input_mode: :none
    :loads: []
    :output_format: :yaml
    :requires:
    - yaml
    :log_format: :none
    :noop: false
modifiable: true

Probably most useful in that object at runtime is the record count, accessible with both $RC.count and $RC.i. This is only really useful in line mode, because in the others it will always be 0 or 1. Here is an example of how you might use it as a kind of progress indicator:

➜  ~   find / | rexe -ml -on \
'if $RC.i % 1000 == 0; puts %Q{File entry ##{$RC.i} is #{self}}; end'

...
File entry #106000 is /usr/local/Cellar/go/1.11.5/libexec/src/cmd/vendor/github.com/google/pprof/internal/driver/driver_test.go
File entry #107000 is /usr/local/Cellar/go/1.11.5/libexec/src/go/types/testdata/cycles1.src
File entry #108000 is /usr/local/Cellar/go/1.11.5/libexec/src/runtime/os_linux_novdso.go
...

Note that a single quote was used for the Ruby code here; if a double quote were used, the $RC would have been interpreted and removed by the shell.

Implementing Domain Specific Languages (DSL's)

Defining methods in your loaded files enables you to effectively define a DSL for your command line use. You could use different load files for different projects, domains, or contexts, and define aliases or one line scripts to give them meaningful names. For example, if you had Ansible helper code in ~/projects/ansible-tools/rexe-ansible.rb, you could define an alias in your startup script:

➜  ~   alias rxans="rexe -l ~/projects/ansible-tools/rexe-ansible.rb $*"

...and then you would have an Ansible DSL available for me to use by calling rxans.

In addition, since you can also call pry on the context of any object, you can provide a DSL in a REPL (shell) trivially easily. Just to illustrate, here's how you would open a REPL on the File class:

➜  ~   ruby -r pry -e File.pry
# or
➜  ~   rexe -r pry File.pry

self would evaluate to the File class, so you could call class methods using only their names:

➜  ~   rexe -r pry File.pry

[6] pry(File)> size '/etc/passwd'
6804
[7] pry(File)> directory? '.'
true
[8] pry(File)> file?('/etc/passwd')
true

This could be really handy if you call pry on a custom object that has methods especially suited to your task:

➜  ~   rexe -r wifi-wand,pry  WifiWand::MacOsModel.new.pry

[1] pry(#<WifiWand::MacOsModel>)> random_mac_address
"a1:ea:69:d9:ca:05"
[2] pry(#<WifiWand::MacOsModel>)> connected_network_name
"My WiFi"

Ruby is supremely well suited for DSL's since it does not require parentheses for method calls, so calls to your custom methods look like built in language commands and keywords.

Quotation Marks and Quoting Strings in Your Ruby Code

One complication of using utilities like rexe where Ruby code is specified on the command line is that you need to be careful about the shell's special treatment of certain characters. For this reason, it is often necessary to quote the Ruby code. You can use single or double quotes to have the shell treat your source code as a single argument. An excellent reference for how they differ is on StackOverflow at https://stackoverflow.com/questions/6697753/difference-between-single-and-double-quotes-in-bash.

Personally, I find single quotes more useful since I usually don't want special characters in my Ruby code like $ to be processed by the shell.

Sometimes it doesn't matter:

➜  ~   rexe 'puts "hello"'
hello
➜  ~   rexe "puts 'hello'"
hello

We can also use %q or %Q, and sometimes this eliminates the needs for the outer quotes altogether:

➜  ~   rexe puts %q{hello}
hello
➜  ~   rexe puts %Q{hello}
hello

Sometimes the quotes to use on the outside (quoting your command in the shell) need to be chosen based on which quotes are needed on the inside. For example, in the following command, we need double quotes in Ruby in order for interpolation to work, so we use single quotes on the outside:

➜  ~   rexe puts '"The time is now #{Time.now}"'

The time is now 2019-03-29 16:41:26 +0800

In this case we also need to use single quotes on the outside, because we need literal double quotes in a %Q{} expression:

➜  ~   rexe 'puts %Q{The operating system name is "#{`uname`.chomp}".}'

The operating system name is "Darwin".

We can eliminate the need for any quotes in the Ruby code using %Q{}:

➜  ~   rexe puts '%Q{The time is now #{Time.now}}'

The time is now 2019-03-29 17:06:13 +0800

Of course you can always escape the quotes with backslashes instead, but that is probably more difficult to read.

No Op Mode

The -n no-op mode will result in the specified source code not being executed. This can sometimes be handy in conjunction with a -g (logging) option, if you have are building a rexe command and want to inspect the configuration options before executing the Ruby code.

Mimicking Method Arguments

You may want to support arguments in your rexe commands. It's a little kludgy, but you could do this by piping in the arguments as rexe's stdin.

One of the previous examples downloaded currency conversion rates. To prepare for an example of how to do this, let's find out the available currency codes:

➜  /   echo $EUR_RATES_JSON | \
rexe -ij -mb -op "self['rates'].keys.sort.join(' ')"

AUD BGN BRL CAD CHF CNY CZK DKK GBP HKD HRK HUF IDR ILS INR ISK JPY KRW MXN MYR NOK NZD PHP PLN RON RUB SEK SGD THB TRY USD ZAR

The codes output are the legal arguments that could be sent to rexe's stdin as an argument in the command below. Let's find out the Euro exchange rate for PHP, Philippine Pesos:

➜  ~   echo PHP | rexe -ml -op -rjson \
        "rate = JSON.parse(ENV['EUR_RATES_JSON'])['rates'][self];\
        %Q{1 EUR = #{rate} #{self}}"

1 EUR = 58.986 PHP

In this code, self is the currency code PHP (Philippine Peso). We have accessed the JSON text to parse from the environment variable we previously populated.

Because we "used up" stdin for the PHP argument, we needed to read the JSON data explicitly from the environment variable, and that made the command more complex. A regular Ruby script would handle this more nicely.

Using the Clipboard for Text Processing

For editing text in an editor, rexe can be used for text transformations that would otherwise need to be done manually.

The system's commands for pasting to and copying from the clipboard can handle the moving of the text between the editor and rexe. On the Mac, we have the following commands:

pbcopy - copies the content of its stdin to the clipboard
pbpaste - copies the content from the clipboard to its stdout

Let's say we have the following currency codes displayed on the screen (data abridged for brevity):

AUD BGN BRL PHP TRY USD ZAR

...and we want to turn them into Ruby symbols for inclusion in Ruby source code as keys in a hash whose values will be the display names of the currencies, e.g "Australian Dollar").

We could manually select that text and use system menu commands or keys to copy it to the clipboard, or we could do this:

➜  ~   echo AUD BGN BRL PHP TRY USD ZAR | pbcopy

After copying this line to the clipboard, we could run this:

➜  ~   pbpaste | rexe -ml -op \
        "split.map(&:downcase).map { |s| %Q{    #{s}: '',} }.join(%Q{\n})"
    aud: '',
    bgn: '',
    brl: '',
    # ...

If I add | pbcopy to the rexe command, then that output text would be copied into the clipboard instead of displayed in the terminal, and I could then paste it into my editor.

Using the clipboard in manual operations is handy, but using it in automated scripts is a very bad idea, since there is only one clipboard per user session. If you use the clipboard in an automated script you risk an error situation if its content is changed by another process, or, conversely, you could mess up another process when you change the content of the clipboard.

Multiline Ruby Commands

Although rexe is cleanest with short one liners, you may want to use it to include nontrivial Ruby code in your shell script as well. If you do this, you may need to add trailing backslashes to the lines of Ruby code.

What might not be so obvious is that you will often need to use semicolons as statement separators. For example, here is an example without a semicolon:

➜  ~   cowsay hello | rexe -me "print %Q{\u001b[33m} \
puts to_a"

rexe: (eval):1: syntax error, unexpected tIDENTIFIER, expecting '}'
...new { print %Q{\u001b[33m} puts to_a }
...                           ^~~~

The shell combines all backslash terminated lines into a single line of text, so when the Ruby interpreter sees your code, it's all in a single line:

➜  ~   cowsay hello | rexe -me "print %Q{\u001b[33m} puts to_a"

Adding the semicolon fixes the problem:

➜  ~   cowsay hello | rexe -me "print %Q{\u001b[33m}; \
puts to_a"

 _______
< hello >
 -------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Clearing the Require and Load Lists

There may be times when you have specified a load or require on the command line or in the REXE_OPTIONS environment variable, but you want to override it for a single invocation. Here are your options:

1) Unspecify all the requires or loads with the -r! and -l! command line options, respectively.

2) Unspecify individual requires or loads by preceding the name with -, e.g. -r -rails. Array subtraction is used, and array subtraction removes all occurrences of each element of the subtracted (subtrahend) array, so:

➜  ~   rexe -n -r rails,rails,rails,-rails -gP
...
   :requires=>["pp"],
...

...would show that the final -rails cancelled all the previous rails specifications.

We could have also extracted the requires list programmatically using $RC (described above) by doing this:

➜  ~   rexe -oP -r rails,rails,rails,-rails '$RC[:options][:requires]'
["pp"]

Clearing All Options

You can also clear all options specified up to a certain point in time with the clear options option (-c). This is especially useful if you have specified options in the REXE_OPTIONS environment variable, and want to ignore all of them.

Comma Separated Requires and Loads

For consistency with the ruby interpreter, rexe supports requires with the -r option, but also allows grouping them together using commas:

                                    vvvvvvvvvvvvvvvvvvvvv
➜  ~   echo $EUR_RATES_JSON | rexe -r json,awesome_print 'ap JSON.parse(STDIN.read)'
                                    ^^^^^^^^^^^^^^^^^^^^^

Files loaded with the -l option are treated the same way.

Beware of Configured Requires

Requiring gems and modules for all invocations of rexe will make your commands simpler and more concise, but will be a waste of execution time if they are not needed. You can inspect the execution times to see just how much time is being consumed. For example, we can find out that rails takes about 0.63 seconds to load on one system by observing and comparing the execution times with and without the require (output has been abbreviated using grep):

➜  ~   rexe -gy -r rails 2>&1 | grep duration
:duration_secs: 0.660138
➜  ~   rexe -gy          2>&1 | grep duration
:duration_secs: 0.027781

(For the above to work, the rails gem and its dependencies need to be installed.)

Operating System Support

Rexe has been tested successfully on Mac OS, Linux, and Windows Subsystem for Linux (WSL). It is intended as a tool for the Unix shell, and, as such, no attempt is made to support Windows non-Unix shells.

More Examples

Here are some more examples to illustrate the use of rexe.

Using Rexe as a Simple Calculator

To output the result to stdout, you can either call puts or specify the -op option:

➜  ~   rexe puts 1 / 3.0
0.3333333333333333

or:

➜  ~   rexe -op 1 / 3.0
0.3333333333333333

Since * is interpreted by the shell, if we do multiplication, we need to quote the expression:

➜  ~   rexe -op '2 * 7'
14

Of course, if you put the -op in the REXE_OPTIONS environment variable, you don't need to be explicit about the output:

➜  ~   export REXE_OPTIONS=-op
➜  ~   rexe '2 * 7'
14

Outputting ENV

Output the contents of ENV using AwesomePrint [see footnote ^4 regarding ENV.to_s]:

➜  ~   rexe -oa ENV

{
...
  "LANG" => "en_US.UTF-8",
   "PWD" => "/Users/kbennett/work/rexe",
 "SHELL" => "/bin/zsh",
...
}

Reformatting a Command's Output

Show disk space used/free on a Mac's main hard drive's main partition:

➜  ~   df -h | grep disk1s1 | rexe -ml \
"x = split; puts %Q{#{x[4]} Used: #{x[2]}, Avail: #{x[3]}}"
91% Used: 412Gi, Avail: 44Gi

(Note that split is equivalent to self.split, and because the -ml option is used, self is the line of text.

Formatting for Numeric Sort

Show the 3 longest file names of the current directory, with their lengths, in descending order:

➜  ~   ls  | rexe -ml -op "%Q{[%4d] %s} % [length, self]" | sort -r | head -3
[  50] Agoda_Booking_ID_9999999 49_–_RECEIPT_enclosed.pdf
[  40] 679a5c034994544aab4635ecbd50ab73-big.jpg
[  28] 2018-abc-2019-01-16-2340.zip

When you right align numbers using printf formatting, sorting the lines alphabetically will result in sorting them numerically as well.

Print yellow (trust me!):

This uses an ANSI escape code to output text to the terminal in yellow:

➜  ~   cowsay hello | rexe -me "print %Q{\u001b[33m}; puts to_a"
➜  ~     # or
➜  ~   cowsay hello | rexe -mb "print %Q{\u001b[33m}; puts self"
➜  ~     # or
➜  ~   cowsay hello | rexe "print %Q{\u001b[33m}; puts STDIN.read"

  _______
 < hello >
  -------
         \   ^__^
          \  (oo)\_______
             (__)\       )\/\
                 ||----w |
                 ||     ||`

More YouTube: Differentiating Success and Failure

Let's take the YouTube example from the "Loading Files" section further. Let's have the video that loads be different for the success or failure of the command.

If we put this in a load file (such as ~/.rexerc):

def play(piece_code)
  pieces = {
    hallelujah: "https://www.youtube.com/watch?v=IUZEtVbJT5c&t=0m20s",
    rick_roll:  "https://www.youtube.com/watch?v=dQw4w9WgXcQ&t=0m43s",
    valkyries:  "http://www.youtube.com/watch?v=P73Z6291Pt8&t=0m28s",
    wm_tell:    "https://www.youtube.com/watch?v=j3T8-aeOrbg",
  }
  `open #{Shellwords.escape(pieces.fetch(piece_code))}`
end


def play_result(success)
  play(success ? :hallelujah : :rick_roll)
end


# Must pipe the exit code into this Ruby process, 
# e.g. using `echo $? | rexe play_result_by_exit_code`
def play_result_by_exit_code
  play_result(STDIN.read.chomp == '0')
end

Then when we issue a command that succeeds, the Hallelujah Chorus is played [see footnote ^2]:

➜  ~   uname; echo $? | rexe play_result_by_exit_code

...but when the command fails, in this case, with an executable which is not found, it plays Rick Astley's "Never Gonna Give You Up":

➜  ~   uuuuu; echo $? | rexe play_result_by_exit_code

Reformatting Source Code for Help Text

Another formatting example...I wanted to reformat this source code...

         'i' => Inspect
         'j' => JSON
         'J' => Pretty JSON
         'n' => No Output
         'p' => Puts (default)
         's' => to_s
         'y' => YAML

...into something more suitable for my help text. Admittedly, the time it took to do this with rexe probably exceeded the time to do it manually, but it was an interesting exercise and made it easy to try different formats. Here it is, after copying the original text to the clipboard:

➜  ~   pbpaste | rexe -ml -op "sub(%q{'}, '-o').sub(%q{' =>}, %q{ })"
         -oi  Inspect
         -oj  JSON
         -oJ  Pretty JSON
         -on  No Output
         -op  Puts (default)
         -os  to_s
         -oy  YAML

Currency Conversion

I travel a lot, and when I visit a country for the first time I often get confused by the exchange rate. I put this in my ~/.rexerc:

# Conversion rate to US Dollars
module Curr
  module_function
  def myr;      4.08  end  # Malaysian Ringits
  def thb;     31.72  end  # Thai Baht
  def usd;      1.00  end  # US Dollars
  def vnd;  23199.50  end  # Vietnamese Dong
end

If I'm lucky enough to be at my computer when I need to do a conversion, for example, to find the value of 150 Malaysian ringits in US dollars, I can do this:

➜  rexe git:(master) ✗   rexe puts 150 / Curr.myr
36.76470588235294

Obviously rates will change over time, but this will give me a general idea, which is usually all I need.

Reformatting Grep Output

I was recently asked to provide a schema for the data in my rock_books accounting gem. rock_books data is intended to be very small in size, and no data base is used. Instead, the input data is parsed on every run, and reports generated on demand. However, there are data structures (actually class instances) in memory at runtime, and their classes inherit from Struct. The definition lines look like this one:

class JournalEntry < Struct.new(:date, :acct_amounts, :doc_short_name, :description, :receipts)

The grep command line utility prepends each of these matches with a string like this:

lib/rock_books/documents/journal_entry.rb:

So this is what worked well for me:

➜  ~   grep Struct **/*.rb | grep -v OpenStruct | rexe -ml -op \
"a =                            \
 gsub('lib/rock_books/', '')    \
.gsub('< Struct.new',    '')    \
.gsub('; end',           '')    \
.split('.rb:')                  \
.map(&:strip);                  \
                                \
%q{%-40s %-s} % [a[0] + %q{.rb}, a[1]]"

...which produced this output:

cmd_line/command_line_interface.rb       class Command (:min_string, :max_string, :action)
documents/book_set.rb                    class BookSet (:run_options, :chart_of_accounts, :journals)
documents/journal.rb                     class Entry (:date, :amount, :acct_amounts, :description)
documents/journal_entry.rb               class JournalEntry (:date, :acct_amounts, :doc_short_name, :description, :receipts)
documents/journal_entry_builder.rb       class JournalEntryBuilder (:journal_entry_context)
reports/report_context.rb                class ReportContext (:chart_of_accounts, :journals, :page_width)
types/account.rb                         class Account (:code, :type, :name)
types/account_type.rb                    class AccountType (:symbol, :singular_name, :plural_name)
types/acct_amount.rb                     class AcctAmount (:date, :code, :amount, :journal_entry_context)
types/journal_entry_context.rb           class JournalEntryContext (:journal, :linenum, :line)

Although there's a lot going on in this code, the vertical and horizontal alignments and spacing make the code straightforward to follow. Here's what it does:

grep the code base for "Struct"
exclude references to "OpenStruct" with grep -v
remove unwanted text with gsub
split the line into 1) a filespec relative to lib/rockbooks, and 2) the class definition
strip unwanted space because that will mess up the horizontal alignment of the output.
use C-style printf formatting to align the text into two columns

Conclusion

Rexe is not revolutionary technology, it's just plumbing that removes parsing, formatting, and low level configuration from your command line so that you can focus on the high level task at hand.

When we consider a new piece of software, we usually think "what would this be helpful with now?". However, for me, the power of rexe is not so much what I can do with it in a single use case now, but rather what will I be able to do over time as I accumulate more experience and expertise with it.

I suggest starting to use rexe even for modest improvements in workflow, even if it doesn't seem compelling. There's a good chance that as you use it over time, new ideas will come to you and the workflow improvements will increase exponentially.

A word of caution though -- the complexity and difficulty of sharing your rexe scripts across systems will be proportional to the extent to which you use environment variables and loaded files for configuration and shared code. Be responsible and disciplined in making this configuration and code as clean and organized as possible.

Footnotes

[1]: Rexe is an embellishment of the minimal but excellent rb script at https://github.com/thisredone/rb. I started using rb and thought of lots of other features I would like to have, so I started working on rexe.

[2]: It's possible that when this page opens in your browser it will not play automatically. You may need to change your default browser, or change the code that opens the URL. Firefox's new (as of March 2019) version 66 suppresses autoplay; you can register exceptions to this policy: open Firefox Preferences, search for "autoplay" and add "https://www.youtube.com".

[3]: Making this truly OS-portable is a lot more complex than it looks on the surface. On Linux, xdg-open may not be installed by default. Also, Windows Subsystem for Linux (WSL) out of the box is not able to launch graphical applications.

Here is a start at a method that opens a resource portably across operating systems:

  def open_resource(resource_identifier)
    command = case (`uname`.chomp)
    when 'Darwin'
      'open'
    when 'Linux'
      'xdg-open'
    else
      'start'
    end

    `#{command} #{resource_identifier}`
  end

[4]: It is an interesting quirk of the Ruby language that ENV.to_s returns "ENV" and not the contents of the ENV object. As a result, many of the other output formats will also return some form of "ENV". You can handle this by specifying ENV.to_h.

DEV Community: Keith Bennett

AutoPilot, CoPilot, or AI Assistant?

AI Downsides

Resisting Oversimplification

The Optimum Balance

Organizational Responsibility

Conclusion

Installing International Language Keyboard Input Methods on Linux Mint 20 Cinnamon & Mate

Language Support

If...

Install IBus

Add IBus to the Startup Applications List

Configure IBus

Add a keyboard shortcut

Add input methods

Reboot and Test

The Language Panel Applet

Cautions

You're done!

Using Lambdas to Simplify Varying Behaviors in Your Code

The Problem with Non-Lambda Approaches

Callables as a Superset of Lambdas

This means that in Ruby, since the method used to call a lambda is call, any object that responds to call can be used in place of a lambda.

The Buffered Enumerable

Using Predicate Callables to Implement Filters

Conclusion

Footnotes

Add Bootstrap to Rails 6 with Two Shell Commands

Introduction

Even Simpler -- A Patch

Adding the JavaScript Libraries

Testing that Bootstrap Works

Git's Patch Support

Conclusion

Clinics in Your Meetups

Multiple Sources of Information

Dialog vs. Monologue

Synergy

Greater Variety of Subject Matter

Community Development

Clinics, in Particular

Structure

Advance Notice of Topics

Infrastructure

Level of Effort

Frequency

Duration

Clinics for Dev Teams

Cautions and Counterpoints

Philippines Ruby Meetup, January 2020

Conclusion

Footnotes:

The Case for Stabby Lambda Notation

The Stabby Lambda (->)

Stabby Notation as an Indicator of Preferred and Default Proc Type

->'s Picture-Like Notation

The Need for Visual Differentiation

Rubocop

Conclusion

lambdas Are Better Than procs

Arity (Argument Count) Checking Behavior Differences

Return Behavior Differences

A lambda is More Method-Like Than a proc

Conclusion

Footnotes

Using Lambdas to Simplify Varying Behaviors in Your Code

The Problem with Non-Lambda Approaches

Callables as a Superset of Lambdas

This means that in Ruby, since the method used to call a lambda is call, any object that responds to call can be used in place of a lambda.

The Buffered Enumerable

Using Predicate Callables to Implement Filters

Conclusion

Footnotes

Ruby Enumerables Make Your Code Short and Sweet

Automate the Little Things Too

Integrating MPlayer and Ruby

The High Level View

Using Subdirectories

An Example

Shell vs. Ruby Wildcard Expansion

This means that in Ruby, since the method used to call a lambda is `call`, any object that responds to `call` can be used in place of a lambda.

The Stabby Lambda (`->`)

`->`'s Picture-Like Notation

A `lambda` is More Method-Like Than a `proc`

This means that in Ruby, since the method used to call a lambda is `call`, any object that responds to `call` can be used in place of a lambda.