DEV Community: Piotr Murach

My 2021 Annual Review: Seeking Serenity

Piotr Murach — Sun, 06 Feb 2022 22:27:55 +0000

As I sat down to write this review I wondered if there is much to talk about. 2021 has finished way too quickly. Looking back feels like staring at a blurred image and trying to make out what it's about. But that's exactly why I needed to write this review. Without reflection time, it's easy to get busy and forget why we do what we do and be caught in "Groundhog Day" every year.

I tried to do something a bit different in 2021 - slow down and seek serenity. I don't want this to sound grandiose as if I aimed to reach some state of enlightenment. It is simply to say that I decided to limit my busyness to gain some perspective and find more quiet time for myself.

This is the fourth time I'm doing this annual review. To honour long-standing tradition, my review will follow these three questions:

What went well this year?
What didn't go so well this year?
What did I learn?

I hope that you enjoy reading about my highs and lows and what these twists and turns have taught me. Let's dive in.

1. What went well this year?

Let's look at the good stuff first.

Open-source. This is a constant for me. Something I can lose myself in. I got into open-source more than a decade ago and it still provides me with a lot of growth both technically and personally. There are many talented people involved in the community that inspire me daily to be a better coder, writer and communicator.

In February I published a curated list of Awesome Ruby CLI apps with my aim to create more community resources. I want this to be a central place for Ruby developers to find great command-line applications. It has about a couple dozen categories with tools from managing git hooks, linting code to converting between data formats. I took a lot of time to research the list and make it useful. I want to continue to grow it even more in 2022.

I made 16 releases of my Ruby gems. This is a bit less compared with previous years but still a decent effort. The downloads have grown an extra 100 million in 2021 to an outstanding 250 million in total. I'm not attached to these numbers but I am grateful for everyone who puts their trust in my libraries. This is truly humbling.

As for the TTY toolkit suite of gems, I expanded it by the tty-sparkline. I also worked on many features in different tty components that I intend to wrap up and release in the coming year. The word completion in the tty-reader gem is one of these features. It is also long demanded and awaited by the Ruby community. I want to continue on the path of making terminal applications more visually attractive, intuitive and easier to build. I'm curious to see what's possible.

Later in the year, I created the minehunter terminal game inspired by the classic Microsoft Minesweeper. Its purpose, apart from obvious fun, is to demo how quickly you can put together terminal applications with a handful of tty components. Projects like this are an opportunity for me to see how well different components work together. I admit, I spent (or 'wasted'), many hours playing the game.

Playing minehunter on a 20x10 gird with 30 mines

Reading. I set my new record - 66 books read! My eclectic taste meant that I meandered through various subjects. In particular, I was keen to learn about our capacity for creativity and how innovative ideas come about. But also to see where our analytical thinking fails us.

The "Fooled by Randomness" by Nassim Table provides one such alternative approach to thinking about life events. We tend to attribute major achievements in our life to grit and hard work but dismiss how much luck plays a role. One random event can completely change our life's trajectory. You can be easily fooled by such abnormal events. Then succumb to the survivorship bias where you create a convenient narrative that it was always meant to be. In the software industry, we celebrate big successes and skip mentioning the many projects and companies that failed. I picked up many insights from the book. But I'd like to caution you. It's not a book for everybody given the author's uncompromising, direct and anecdotal style.

On a more software engineering theme, I enjoyed reading the Staff Engineer by Will Larson. It shows possible paths for those in senior software engineering roles that wonder what to do next. Taking the next step doesn't have to involve becoming a "people manager". Larson explores job types and responsibilities that higher-level roles may entail. Regardless of the role, you will write less code and think more about the strategy and vision for the team. A lesson that "the most effective leaders spend more time following than they do leading" is a good taste of what the book is about. I found many interviews with staff engineers from well-known companies very helpful. Their perspectives gave me a better feel for what to expect.

Exercise & Fitness. Keeping fit is one of the key areas for me that I didn't want to neglect. But, due to gym restrictions I needed to find a different strategy. The answer came on one of my daily lunch walks. I discovered an exercise area with various metal bars and ramps perfect for doing callisthenics exercises. I got hooked. I started slowly and built up my strength step by step. I didn't do any crazy moves. Instead, I stuck to basic pullups, chin-ups and pushups increasing intensity and repetitions each week. I found that training outdoors appeals to me. You're close with nature, don't need to waste time changing clothes and most importantly it's free. All great benefits.

2. What didn't go so well this year?

Here's where my past twelve months proved challenging:

Writing. This will be quick and simple to summarise - I haven't written any articles on my blog in 2021. This is the worst year when it comes to writing so far. The blog has practically started gathering virtual cobwebs. Now I know the feeling and the reason why there are so many abandoned blogs. I partly blame this on my increased reading time. The year wasn't a total write-off though. I wrote some future article outlines and as I like to call them 'solid' drafts on a few ideas. I plan to create a natural writing cadence next year and push myself to blog more.

Travel. I hardly left my city. I had itchy feet the whole year and visited holiday booking websites regularly. But my worries about Covid and restrictions that seemed to change daily kept everything in the realm of dreams. Instead, I opted for kayaking in the lakes and weekend sessions in a deck chair reading and staring at the sky. My mind and wallet were probably much happier with this type of holiday.

Social media. I did a few occasional tweets in 2021 but apart from that, I have mostly stayed away from social media. One of the reasons I kept social media on hold was to unplug and attempt to do more deep and meaningful work. This provided an immediate benefit of having more time and fewer disruptions. Unfortunately, the downside was also that I lost contact with the developer community. One of the things that I enjoy about social media is the many spontaneous interactions I can have. Also, I missed seeing what other interesting projects people are working on. In 2022 I want to plug back into the social machine and share more of what I'm up to.

Coding. I had no adventures with new programming languages in 2021. Instead, I turned my attention towards different topics indirectly related to coding. I've spent my hours reading articles, books and watching talks around technical leadership, software development methodologies and innovation. These prompted me to consider many types of questions. How do you introduce technical changes in a team successfully? What does great technical leadership look like? What are the principles and techniques that make a team effective? That's why I picked up the Staff Engineer book in the first place.

3. What did I learn?

By slowing down and making more room for reflection, I discovered a little bit more about myself and what works for me. And here's what I've learnt.

Make peace with your plans. By "making peace", I don't mean sitting still on a pillow, meditating and being unconcerned about how your plans unfold. To me, it is knowing what you want, being patient and working towards your goals without obsessing about the outcome or what other people may think. It's a way of being in agreement with your desires and fulfilling them in a way that fits your character and life circumstances. This also means trusting yourself to do your best without burning out.

Act but don't despair when your plans overextend or cease to be important to you anymore. I wanted to write more on my blog and contribute more to open-source in 2021. But my circumstances made it difficult to schedule consistent time. Getting wound-up about it all would be futile. Peacefully accepting reality and staying the course will work better in the long run.

Carefully prune the excessive ideas and tasks. I tend to dilute my focus by thinking of many ideas and things I want to do and learn. My curiosity naturally takes me to many places. As a result, I spread myself too thin to achieve all of my plans. The ambitious list of things I would like to do doubles each time I look at it.

I decided it was time to end this behaviour. Life is finite. I cannot do everything. Some things on second reflection probably shouldn't be done in the first place. Time to eliminate doing tasks and learning subjects that don't align with my future plans. On my personal board, I added a new column "Maybe" that I aggressively moved many items over. One day I may revisit the maybe list, though I doubt it.

Take the time to find your blind spots and improve on what you already know. There are domains of knowledge, tools and techniques with which I am comfortable. I can do things without much effort. But here lies the problem, I get complacent once I reach a "good enough" level. For years, I've been using Vim as my editor of choice. I reached a certain level of fluency and stopped at that. But Vim itself didn't stop getting better. Why should I? So I spent more time this year improving my Vim editing skills.

It is easy to continue without realising that some of the things we know or do are outdated, inefficient or we simply don't understand them well enough. Our culture encourages learning new shiny things and constant growth. Especially if you work in the software industry. But do we only have to learn new things? What about being better at what we already know? There is always another depth we can go into to glean new understanding. This is best expressed by great physicist Richard Feynman:

"You only think you know, as a matter of fact. And most of your actions are based on incomplete knowledge and you really don't know what it is all about, or what the purpose of the world is, or know a great deal of other things. It is possible to live and not know."

Outro

Another year in the books. There was far lower intensity to many things I did. No international travel hence no anxiety booking flights and hotels. No conference speaking. Nearly no posts written on social media. I replaced it all with walks in nature, plenty of reading time and contemplation. This slowdown offered me a chance for self-discovery and reevaluation of my plans. I learned more about the things I want to continue doing to grow as an individual and software developer. Equally, I learned what I need to eliminate to make room for everything else that I hope to do.

Let's see what 2022 has in the cards for me!

This article was originally published on PiotrMurach.com. Cover photo by Piotr Murach.

My 2020 Annual Review: Strange Ride

Piotr Murach — Sun, 24 Jan 2021 21:56:28 +0000

For a few days I debated whether there is much point in writing this review. I had a lot of enthusiasm going into 2020 and couldn't wait to begin working on my ideas. But then this C letter thing happened, spelling the end for all of my best laid plans. I went from feeling what can best be described as an emoji face with starry eyes to looking like the Edvard Munch's screamer.

Despite all the turmoil, there were many good events and experiences that had a positive influence on me and provided valuable insights worth sharing. Similar to the previous year, my process will be guided by the following questions:

What went well this year?
What didn't go so well this year?
What did I learn?

So, grab your favourite beverage and make yourself comfortable. We're going to look through some of my personal highlights and disappointments. We will finish by trying to tease out some lessons and talk about my aspirations for the future.

1. What went well this year?

Here's how I turned these chaotic times into positive outputs:

Open-sourcing software. This was probably one of the most productive times I had in open source. Compared to previous years, I didn't publish as many new Ruby gems. Instead, my focus shifted to maintenance and improving the quality of what I already created.

I resolved many outstanding issues. Without being exact, I think I've dealt with more than a hundred open tickets and pull requests. On top of that, I worked on many long requested features. As a result, there are many projects without any open issues now. An open source maintainer dream turning into reality.

All this activity resulted in 56 Ruby gem releases. To my surprise, by the end of the year the total gem download count crossed the 150 million mark. A 50 million increase in one year alone feels crazy. Knowing that more and more people use my libraries motivates me.

My main focus went to updating all of the TTY toolkit components. That's nearly 25 gems. Whilst doing the various updates, I couldn't resist creating new components. So in February, I released the tty-exit gem. This is a relatively small but essential gem that will fill a need in the TTY framework for more readable exit codes.

In May, I published the tty-option gem for parsing command line arguments, flags and environment variables. I put tons of work into the release. More than 300 commits made it into the first 0.1 version. This is probably the most solid first release I've ever done. It brings many powerful features like parsing map arguments which is a common way of specifying flag values in terminal tools like the Docker.

Now that all of the TTY components are updated(as evidenced in the image below), the plan is to release a new version of the toolkit in 2021. I'm excited about the future and where the TTY ecosystem of gems is going to be!

Sponsorship. GitHub made it super easy for individual developers and now companies to sponsor open source projects and their creators. This has a huge potential to make open source more sustainable. I'm grateful and feel fortunate to have gained a few new sponsors. I appreciate their generosity and support!

To me, the sponsorship is more than money, it's a real vote of confidence and shows approval that what I create matters. I'm humbled to learn that there are people wanting for the TTY and the newer Strings ecosystem of gems to succeed. I hope that companies will also contribute to supporting the future development.

Reading. What do you do when there is nowhere to go? You stay home and read books! I read more than 50 books. Honestly, I haven't planned to read that many but they served as a great escape. The topics that I've picked were a true mixture. Sticking with the more technical theme of this review, a couple of books stood out for me the most.

The first book that I enjoyed a lot is the Phoenix Project. A tale about DevOps that didn't disappoint and lived up to its hype. I never read a book that introduces technical concepts by telling a fictional story. Who could expect that telling a story is such a good concept, heh? Though the characters were made up, the story felt real and very believable. I could relate to many issues found in bridging the gap between IT services and the Development team. I'd even say that's where I spent most of my energy in the recent years. So yes, sometimes, the dialogues and problems really felt close to the bone.

The Working in Public was also a very relatable book. Nadia Eghbal did a great job analysing the open source community and discussing the evolution of open source. I felt a bit like a lab rat in an experiment. It was a weird feeling being in a way analysed as part of an open source ecosystem. The quadrant diagram that explains four different ways that open source projects are structured was particularly novel and interesting. As I recall, I classified myself as the Stadium type open source developer. The book definitely grew my awareness and strengthened my thoughts around maintenance and long-term sustainability in open source.

Coding. Given the extra hours in the day, I dove into Crystal programming language. Even though Crystal is still below major release, I haven't had any issues with it. The experience felt rather smooth. The language has Ruby inspired syntax but you would be wrong to think that it doesn't offer much beyond that. Quite the contrary. Crystal is a compiled language and thus super fast. It also provides strong type guarantees that lead to features not present in Ruby. For example, you can overload method definitions. In most cases, the compiler doesn't force you to provide type signatures as the type inference is very good.

At last, I also dipped my toes into a Lisp-like language. I chose to learn the Racket language. This experience certainly felt like a bigger departure than Crystal. Even though it is a dynamic language, it's hard to find parallels with any languages that I had a chance to code in so far. The syntax feels rather strange to type and look at in my editor. Writing an equivalent of a class is a journey into many nested methods in an ever growing forest of brackets. It is an aesthetic that I haven't had to deal with before. I like it though when a language stretches my coding skills.

2. What didn't go so well this year?

Now let's look at some things from my “not so great” list.

Blogging. I started the year with an intention of publishing one article per month. This seemed like a reasonable goal given my workloads and past experience. But I managed to only write three articles which includes a yearly review.

Out of the three articles, the "Writing a Ruby Gem Specification" article attracted the most interest. It was featured in a prominent Ruby community newsletter Ruby Weekly underneath a biblical image with a prophetic feel to it. I got a chuckle out of it. Based on comments, emails and reviews a lot of people found the article useful. In the third and, as it turned out, the last article of 2020 titled "Looking Inside a Ruby Gem ", I further built on the topic of explaining RubyGems packaging system.

Then my enthusiasm waned. I lost momentum and found it hard to get back into writing again. I tried writing for a while on a few other topics but nothing felt solid enough or ready to be published. Maybe the weight of expectations that I put on myself gave me the famous writer's block? Not sure. Despite my meagre effort my blog readership has tripled. I want to turn the corner in 2021 and write more but without expectation of a regular publishing schedule. The bar is set very low indeed.

Travel and conferences. By the end of 2019, I made a grand plan to visit a few countries mostly in Europe. France, Finland, Russia and Japan were top of my list. I was excited! Well... as you probably expect the grand tour of Europe didn't happen. I visited only one country from the list, France. In the middle of February, I boarded the Eurostar train from London to Paris to attend the ParisRB conference. The first and only conference I attended in person that year. At that point I was still blissfully unaware.

The ParisRB was a two day conference held in a university setting. I travelled on the crowded metro and trams to reach the venue which now feels surreal. The conference hosted many great speakers. In particular, Paolo "Nusco" Perrotta told a great origin story of what we now know as a deep learning domain. I was impressed with his presentation skills and the ability to draw in the audience. In another talk I liked a lot, a duet of Mélanie and Alexandre from the Doctolib company talked about challenges with onboarding new developers to a large codebase. I enjoyed hearing about how they structure their inhouse mentoring programs and automate workflows to keep their large codebase consistent.

In between the presentations, I talked to many developers and had a chance to speak with some of the speakers. Being able to talk to like-minded people in real life is what I miss the most from the conference. Only now I realise what a privilege that was. The conference ended in quite an emotional way and a few people had tears as if they knew that this could be the last such conference in a long time.

Exercising. I continued going to the gym until the middle of March. I kept my gym membership for a few months longer expecting this whole pandemic will end soon and normal life resume. It shouldn't be a surprise that my general well being plummeted. I continued exercising at home, but lack of weights and a small space made it impossible to match my prior training routines.

During the summer, I bought a basketball and started playing at a nearby court. Initially, I had little stamina and gassed out pretty quickly. I didn't give up though as basketball was always my number one sport. Playing has rekindled my passion and I even got to do a few dunks. In autumn, I bought some cycling gear and rode my bike regularly around woods and lakes.

3. What did I learn?

My monk-like living has made me realise and stressed the importance of some things.

Keep reaching out. I can be a bit of hermit and I'm generally fine with minimal social interactions. This pandemic made me realise that more than ever it's important to keep frequently in touch with friends and loved ones. But it is equally key to seek out virtual opportunities to meet new people. In 2021 I want to engage more with the Ruby community via social media and other means.

Cook your meals. Self-isolation has made it clear to me that one cannot subsist for a very long time on porridge and fish and chips diet. If anything, eating the same thing every day gets boring. The nutritious value of my meals was also questionable. I developed a craving for soups. So with the help of my mum, I dedicated myself to learning how to cook a different soup every week. Every time, I cooked a big pot that lasted me a few days. Cooking is one of the survival skills that I view as essential to living a better quality of life. It can also save you a bit of money.

Lower your standards. Hear me out before you cast your look of disdain at me. I'm not suggesting you release crappy work. Far from it. This is to say that whatever you wish to do, whether it is writing a piece of code or a blog article, you need to let go of expectations and dive straight in. Leave perfection out the door. Don't worry about applying design patterns or figuring out domain models. Just write that piece of code and have fun. You can always come back and make things better. Once an initial version exists, you can improve the design and refactor messy code.

Becomome more decisive. I can be stuck in a perpetual circle of indecision. Debating, sometimes for days, whether I should go ahead and do something or not. Should I include this feature in a library? Should I write this article? Should I buy this? What's worse, I may never actually pull the trigger and instead let the decision sit there for weeks. I need to stop this and learn how to make quicker and better decisions. The guidance from basketball's greatest really resonates with me:

"I always say, decisions I make, I live with them. There are always ways you can correct them or ways you can do them better. At the end of the day, I live with them." - LeBron James

Execute on your ideas. I have ideas. You have them. Everyone has them. Time passes and nothing happens. I research, think and procrastinate. The list of ideas I wish to tackle is constantly growing. The only way out is to start working on the stuff. Create a quick code spike. Write an article outline. Record a video intro. Whatever. Nothing will give you more clarity than a dose of action. My new motto comes from Thomas Edison who said "Vision without execution is hallucination.".

Onwards and upwards

What a year it was! As I mentioned in my last review I'm not keen on setting goals. After 2020 that's probably even more true than ever. But this doesn't mean I don't have a vision for what I'd like 2021 to be like. I'd like to dedicate more time to help and encourage more people to write command line applications in Ruby. I want to build resources and a community of Ruby terminal applications enthusiasts.

I have many thoughts around how this can be done. I will try to promote and highlight Ruby projects that are meant to be used in the terminal. I plan to increase my participation in other open source projects. I'd especially like to provide support for people using TTY components. I receive questions on how to use my various libraries and I feel I could be doing more to help. Making more code contributions to other projects, giving feedback or making suggestions for improvements are a few that come to mind.

I think we can all agree that 2020 has been a strange ride, but one thing I've learned for sure is to embrace the uncertainty. Whatever comes my way I will do my best to turn it into a positive. Let's make 2021 a good one!

Intro photo of all attendees at the end of the two-day ParisRB conference.

Looking Inside a Ruby Gem

Piotr Murach — Tue, 21 Apr 2020 21:09:29 +0000

If you have used Ruby for any length of time you have probably, knowingly or not, used a gem. It could've been already installed on your system as part of your Ruby release. You may also have downloaded it with the RubyGems package manager. Irrespective of how you got hold of a gem, you required it as a dependency in your file and used it. But, have you ever wondered what a gem is? What's inside a Ruby gem? Having written a Ruby gem manifest file in the Writing a Ruby Gem Specification article, it's only natural to use it to learn how a Ruby package is built.

In this article, I'd like to take you to the RubyGems factory to see where it all begins. We will first learn how to create a Ruby gem and then unpack it and look inside the internals. Whether you intend to share your own gems or continue using third-party ones, knowing the inner workings of a gem will be useful. If anything it will make you appreciate the elegance with which the whole Ruby ecosystem works.

Now, on to the fun.

What is a Gem?

Any popular programming language has an ecosystem of packages that solve various problems. These packages capture reusable functionality and save tons of time for everyone else. To be accessible to everyone, libraries need a central storage place. A service for automated uploading and downloading of named packages referred to as - a registry.

Examples of popular registries are NPM (www.npmjs.com) for JavaScript, Hex (hex.pm) for Elixir and Erlang code, and Crates (crates.io) for Rust. Ruby also has an official place to store and distribute its packages: the rubygems.org. This is a service that provides you with the ability to browse and download hundreds of thousands of Ruby packages called a gem.

Each gem in a registry has a unique name and a release number. The Ruby community has a taste for quirky and whimsical names. Don't believe me, look at these:

Cucumber - tool for running automated tests written in plain language.
Jekyll - a static site generator.
Nokogiri - HTML and XML parser.
Sinatra - a minimal web framework.
Unicorn - a web server.

The vibrant and large ecosystem of Ruby gems plays a considerable role in making the Ruby language as popular as it is now. To be part of this success and to create our own gems, we need to learn about the tool that makes it all possible, the RubyGems package manager.

The RubyGems Package Manager

When a Ruby language is installed on your system it includes a handful of gems. One of these gems is the RubyGems gem (this is a bit meta, a gem to manage gems). The RubyGems is a powerful package manager that will do things for you like:

Installation of third-party gems.
Dependencies resolution.
Searching local and remote packages.
Inspection of gems including their files and metadata.
Building and serving RDoc documentation.
Creation and publishing of gems.

This is not an exhaustive list but it should give you a taste of how many tasks a full-blown package manager handles. As of this writing, Ruby 2.7 installs RubyGems 3.1.2 on my system. The RubyGems gem provides you with a command-line tool that will support all the before listed tasks called the gem. I agree this is a bit confusing. Both a Ruby package and the tool to manage these packages are named the same.

You may also have used a tool called Bundler to install gems. Both Bundler and RubyGems are partners in the same crime. They help you above all manage gem dependencies. Bundler's focus is on dealing with the installation of gems for a local application. In contrast, RubyGems manages all the gems installed on your system under a given Ruby version. The plot thickens even more starting from Ruby version 2.6. In this version, the Bundler has been merged into the RubyGems as a default gem alongside packages like irb, logger or fileutils.

For us, the important feature of RubyGems is that it can convert a gem specification into a package. That's what we're going to take a closer look at next.

Building a Gem

Most likely you used the gem command to install or uninstall a dependency. There are more than 30 commands available at your fingertips. In particular, the gem build command is what we're going to look at. Its job is to take a manifest file and turn a bunch of files into a Ruby gem. The typical layout of a gem follows a well-established pattern for organising files. For example, the layout of the emoticon gem that we used to write a manifest file for looks like this:

emoticon/
├── exe
│   └── emote
├── lib
│   ├── emoticon
│   │   └── version.rb
│   └── emoticon.rb
├── CHANGELOG.md
├── LICENSE.txt
├── README.md
└── emoticon.gemspec

For our demonstration, we're going to use the already created emoticon.gemspec that contains all the information necessary to bake a new gem.

To create a gem from our gemspec, run the gem build command:

$ gem build emoticon.gemspec

This will display confirmation information in the console:

Successfully built RubyGem
Name: emoticon
Version: 0.1.0
File: emoticon-0.1.0.gem

At this point, you will have the emoticon-0.1.0.gem file in the same directory. Nice!

Under the covers the build command delegates its work to the Gem::Package class and the class level build method. This method accepts as an argument Gem::Specification instance and few options. That's exactly the instance we used to describe metadata about the emoticon gem in our manifest file. As a reminder here is the emoticon.gemspec snippet, for the sake of clarity:

# frozen_string_literal: true

require_relative "lib/emoticon/version"

gemspec = Gem::Specification.new do |spec|
  spec.name        = "emoticon"
  spec.version     = Emoticon::VERSION
  ...
end

Instead of relying on the gem build command, we will create a package manually to learn what's involved. To begin, we create a new file build.rb and load the "rubygems/package" dependency. As it happens, all we need to do to build a gem is add a couple lines of code. First, we load the gem specification. Next, we invoke the Gem::Package's build method with a gemspec as an argument:

require "rubygems/package"

gemspec = Gem::Specification.load("emoticon.gemspec")
Gem::Package.build(gemspec)

With these two lines in place, we can now execute our build.rb file:

$ ruby build.rb

This produces exactly the same result as before - an emoticon-0.1.0.gem file. Now, we have a gem for further exploration.

What's Inside a Gem?

The emoticon-0.1.0.gem has an unusual .gem extension name. But, this file is nothing more than an archived directory packaged using the tar Unix utility. We can take a look inside of the archive and list its content to the console with the tar utility. All we need to do is pass the -t flag for printing to the terminal and -f option for reading from a file:

$ tar -tf emoticon-0.1.0.gem

Inside the tar archive file, we find a set of three gzipped files:

checksums.yaml.gz
data.tar.gz
metadata.gz

If you build your gem and sign it with a cryptographic key then the archive will have a total of six files. Each gzipped file will have a matching signature file:

checksums.yaml.gz
checksums.yaml.gz.sig
data.tar.gz
data.tar.gz.sig
metadata.gz
metadata.gz.sig

What are these signature files?

To create a signature file, the RubyGems selects a digest algorithm from the standard library:

# lib/rubygems/security.rb

module Gem::Security
 DIGEST_ALGORITHM =
    if defined?(OpenSSL::Digest::SHA256)
      OpenSSL::Digest::SHA256
    elsif defined?(OpenSSL::Digest::SHA1)
      OpenSSL::Digest::SHA1
    else
      require 'digest'
      Digest::SHA512
    end
end

In the first step, the digest algorithm is applied to every file in the archive to create a summary of its content. The summary is a unique value of a fixed-length often referred to as a file checksum or digest. Then, a signing key and a certificate are combined into a signing instance of Gem::Security::Signer. The signer signs the file's digest and creates a digital signature that is then saved to a file with .sig extension. To see the content of the signature, a base64 encoded binary file, we can use the base64 command:

$ base64 data.tar.gz.sig

In this case, we see some random characters in the terminal output, truncated to save the space:

bAclqtMmrTKFSMFn74w8kwjRrmv/8Wlc1prNXxHfYtqHypaFdnoIOKKPhYAp3D5MKiqmqIsY8VTb8gQvbT/rEcMm40C57fh5JEQ3I7Tkvs0D76nL+cIaqnnqJ1H5Irknhkj+fu...

The signature can be thought of as a software equivalent of what we do in everyday life when we sign a paper document with a pen. By signing a document we verify its accuracy and authenticity. In our case, the signature files provide an extra layer of protection when installing gems in two ways. First, they allow us to verify that the gem source files haven't changed since they were originally signed. Second, they ensure that the signature belongs to the gem's author who owns the private signing key.

By default gems are installed as unsigned without checking their file signatures. This leaves the possibility open that a gem may have been tampered with by an attacker. To take advantage of the signatures during the gem's installation process, you need to use the - -trust-policy option or short flag -P with the highest and most secure policy:

$ gem install emoticon -P HighSecurity

With the signatures files out of the way, we can come back to our checksums, data and metadata compressed files. Our good friend the Gem::Package has more to teach us about how it performs its packaging process. Diving into the build method, we discover that all these three archived files are generated using the Gem::Package::TarWriter class. The tar writer does its work by exposing a few helper methods via an internal DSL (Domain Specific Language) that add content to the main gem archive file:

# lib/rubygems/package.rb

@gem.with_write_io do |gem_io|
  Gem::Package::TarWriter.new gem_io do |gem|
    add_metadata gem
    add_contents gem
    add_checksums gem
  end
end

Let's go ahead and extract all the files from the emoticon gem archive:

$ tar -xf emoticon-0.1.0.gem

One by one, we're going to examine files from the emoticon gem archive so we can learn their purpose.

Inspecting the Files

The first file that we're going to take under a microscope is a compressed YAML file called checksums.yaml.gz. The file compression is done with a lesser-known zlib gem from the Ruby standard library. This gem provides an interface to general purpose and portable zlib utility for compressing files. Among the formats it supports is the gzip (.gz) format.

In more detail, the Gem::Package offloads compression of all the archived files to the gzip_to method. This method accepts as an argument a file content represented as IO stream. Then it uses the Zlib::GzipWriter to turn the stream into a gzipped version:

# lib/rubygems/package.rb

def gzip_to(io)
  gz_io = Zlib::GzipWriter.new io, Zlib::BEST_COMPRESSION
  gz_io.mtime = @build_time
  yield gz_io
ensure
  gz_io.close
end

Since the checksums.yaml.gz is a binary file we need to do something to view its content. Fortunately, we don't have to unpack it to look inside, as we can use the gzcat utility:

$ gzcat checksums.yaml.gz

What we see inside at the top level are the keys that correspond with the names of the digest algorithms used. In our case, the sha256 and sha512 algorithms. Underneath each algorithm name are pairs of gzipped file name and its checksum. The checksums are digital summaries of the file content generated with the particular algorithm. For example, the file checksums under the sha256 key are represented as a string of 64 hexadecimal encoded characters. Thus it all looks something like this:

---
SHA256:
  metadata.gz: bdac75ac0dab99ec3cdb10692ef70f8b0966ef0aff7fb2c40b3c0ebf0440667d
  data.tar.gz: f0004274a83869c47e487eeb06c655b13c14601350eb8f6f4299ea6e0259b2d4
SHA512:
  metadata.gz: 03d8c338cee67683cc5e032a4e54153c9a73496e8960173a60b74b31fbe372e8786b86979efd6a923e0842dc132b2d4bdf020eff299b13a970018a31b5503325
  data.tar.gz: c5c52e2f82d90cc9d249b2dcc3500b3e29216f5db4049557b365f5051f34df19e5152519793b60b9691e1b951ec54fb6fa8e6c39ef673522cc1e19d72bb01bbf

The purpose of the checksums file is to help verify the integrity of the data.tar.gz and metadata.gz files during the gem installation process.

The next file we're going to zoom in on is the data.tar.gz. Since this is a gzipped tar archive, to see inside we use the familiar technique:

$ tar -ztf data.tar.gz

This file contains the source code files that were listed in the gem specification. Here is the list from emoticon gem:

CHANGELOG.md
LICENSE.txt
README.md
exe/emote
lib/emoticon.rb
lib/emoticon/version.rb

The final metadata.gz file is a YAML archive that includes all the metadata extracted from the gem's manifest file. If we look inside the file:

$ gzcat metadata.gz

We're greeted with a long output akin to the following truncated version:

--- !ruby/object:Gem::Specification
name: emoticon
version: !ruby/object:Gem::Version
  version: 0.1.0
platform: ruby
authors:
- Piotr Murach
autorequire: 
bindir: exe
cert_chain: []
date: 2020-03-01 00:00:00.000000000 Z
dependencies:
- !ruby/object:Gem::Dependency
  name: tomlrb
  requirement: !ruby/object:Gem::Requirement
    requirements:
    - - "~>"
      - !ruby/object:Gem::Version
        version: '1.2'
  type: :runtime
  prerelease: false
  version_requirements: !ruby/object:Gem::Requirement
    requirements:
    - - "~>"
      - !ruby/object:Gem::Version
        version: '1.2'
...

What's interesting here are all the !ruby/object notations. What are these? The YAML format allows us to serialize simple values like strings, arrays or hashes. But it also can serialize pretty much any Ruby object. By default, YAML will serialize all the object's instance variables and their values. For example, if we create a Gemspec class from a Struct object with some attributes:

require "yaml"
Gemspec = Struct.new(:name, :date, :version)
gemspec = Gemspec.new("emoticon", Time.now, "0.1.0")

And then print the object content in a YAML format to the console:

puts YAML.dump(gemspec)

We will see all the instance variables serialized:

--- !ruby/struct:Gemspec
name: emoticon
date: 2020-04-10 14:00:41.077868000 +01:00
version: 0.1.0

Yet, we're not stuck with only serializing object's attributes. We can control what attributes YAML is going to use with the to_yaml_properties method. For example, this is how the Gem::Requirement specifies it's YAML transformation under the hood:

# lib/rubygems/requirement.rb

class Gem::Requirement
  def to_yaml_properties # :nodoc:
    ["@requirements"]
  end
end

In our case, the entire Gem::Specification is serialized this way. All its attributes that may point to other objects are recursively serialized and saved in the YAML file.

If you wish to take a look at the specification of any gem installed on your system, you can run the gem spec command:

$ gem spec package-name

This will print all the gem's metadata information in your console.

You should also notice that the gem specification file is only needed when building a gem. It shouldn't be part of your release. The metadata is extracted to a YAML file and that's what is used during a gem installation.

Conclusion

You may have thought of gems as nondescript blobs of functionality that somehow appear on your system under some weird and wonderful name. My hope is that this article clarified the structure of a gem and how it accomplishes its functionality in the Ruby ecosystem.

After all, we learnt that there is nothing new under the sun. The old and trusted Unix tools for archiving and compression serve as the foundation for packaging gems. We experienced how the succinct YAML format keeps the data readable and machine friendly. We saw how the OpenSSL toolkit adds a security layer to make our gems resilient and dependable.

I don't know about you but I like to know the nitty-gritty details of the tools I use. I'm in awe and have nothing but great admiration for the developers who provide such solutions. It takes an ingenious effort to turn the concept of reusable code into a practical solution that scales and makes millions of Ruby projects possible!

Thanks for taking the time to read the article and I hope you found it useful.

Stay hungry and creative!

This article was originally published on PiotrMurach.com.

Photo by Jason D on Unsplash.

Writing a Ruby Gem Specification

Piotr Murach — Mon, 02 Mar 2020 22:50:36 +0000

I created my very first Ruby gem in 2009. Since then I have released many packages. It's been a long ride. A lot of things have changed in a Ruby gem manifest file. Some configuration options are no longer necessary. A few new options appeared. Despite having a Bundler to help you generate a new gemspec, there are still questions you need to answer for yourself. The net effect is that everyone seems to be a little bit unsure about what to include and what to keep out from their gem specification. None of the gemspecs you will see in the wild will be alike. Confusing, isn't it?

In this article, I will examine the most common gemspec attributes and clarify the intention behind them. To help us, we will write a gemspec by hand and cover best practices related to each attribute and offer practical advice. As we learn, we will discover ways to make our gemspec simple and neat like Marie Kondo's drawers. Once we're done, we will have a manifest file that can serve as a reasonable starting point for any gem author. The final gemspec won't have the ultimate list of attributes that works in every scenario. But, I'm doing this in the hope that my practical experience will help you avoid issues that I've been bitten by myself. Whether you're a Ruby newcomer or a veteran with battle scars to show, I'm certain you will pick a few useful nuggets of information. Ready?

There's no better way to learn than to write our own gem manifest from scratch!

A Manifest File By Any Other Name

To build a Ruby gem, at the very least you need to have a manifest file. A file that will describe how to put source files together to do useful things. In the Ruby world, this file is often named package.gemspec. The convention is to use *.gemspec suffix but you could name the file anything you want. Inside the manifest file, we use Ruby to specify metadata and release information about our source code. Soon, we will explore each configuration attribute in more detail. Hold tight!

To aid our discussion, we are going to write a gemspec for an emoticon gem. This fictitious gem will help display ASCII emoticons in a terminal. Life is so much better when we can express our frustration with a bit of table-flipping (╯°□°)╯︵ ┻━┻. Instead of relying on gem generators like bundler or jeweler, we're going to take small steps and see what's necessary to write a gemspec by hand.

The Essential Ingredients

Any modern Ruby file starts with a magic comment that ensures all strings are immutable. This means that we cannot accidentally alter string content after its creation. So let's open emoticon.gemspec in an editor and add the following line:

# frozen_string_literal: true

There are many pieces of metadata we can provide about our gem. The good news is that despite the long list of possible attributes, we don't need to include all of them. There is a subset of only a handful that we need to use as the bare minimum. These attributes are:

name
version
summary
author(s)

To start describing our gem metadata, all we need to do is to create Gem::Specification instance and pass a configuration block as an argument. Within the block, we can start defining various attributes:

# frozen_string_literal: true

Gem::Specification.new do |spec|
end

Now, we are ready to add a minimal set of attributes. First, we add a name for our gem. A valid gem name can only include letters, numbers, dashes, underscores and dots. No other funky characters are allowed. You cannot start your gem name with a dash, underscore or dot either. Any gem name you pick needs to be unique. So like domain names, good names are often at a premium. You may need to be a little bit creative here. To see the naming conventions read "Name your gem" guide.

Bearing all this in mind, we add a name:

spec.name = "emoticon"

Versioning

Next in line, we need to specify a version. This is a three-digit number separated by dots in the format of "major.minor.patch". It's common to start a gem development with the "0.1.0" version. Then use what is referred to as "semantic versioning" to continue releasing new versions. You can find out more on how to manage this number on semver.org. In my experience, it's best to stay safe and do at least one release before the first stable release “1.0.0”. When you think about it, a lot of Ruby infrastructure depends on this little innocuous number. It's equally impressive and scary.

We can provide a version number directly as a string value. However, most often this number is read from another file located inside the lib/ folder. This way other libraries can also read a version number directly from the source code. For example, for our emoticon gem the file would look like this:

# lib/emoticon/version.rb

module Emoticon
  VERSION = "0.1.0"
end

It is common for gems to load the version file with the following three lines:

lib = File.expand_path("../lib", __FILE__)
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
require "emoticon/version"

That's a mouthful to just load a number. What is going on here?

Enter The $LOAD_PATH

We start by getting the hold of a path to the lib/ directory relative to our gemspec location. The lib/ folder contains all our source code files. Next, we check Ruby's global variable $LOAD_PATH to see if it already has the lib/ directory in its list of paths. This variable contains a list of paths that point to installation directories where Ruby files are saved. For example, on my system, one of the paths points to where Rbenv keeps Ruby 2.7 installation "~/.rbenv/versions/2.7.0/lib/ruby/2.7.0".

Ruby uses $LOAD_PATH when you call require statement. For example, if we wanted to load the YAML library in our code, we would do the following:

require "yaml"

The above would cause a search through all the paths in the $LOAD_PATH global variable. Since YAML gem is part of Ruby, it will be located in the language installation directory like "~/.rbenv/versions/2.7.0/lib/ruby/2.7.0/yaml.rb". Once found, Ruby will load the YAML for us automatically. In our case, we add our local lib/ folder to the searched paths list so that the require method can load our version file.

You must admit that's a lot of ceremony just to read a number! I agree.

This was a popular method to load a version file during the 1.8.7 and 1.9.2 Ruby era. It still remains popular, for example, for loading binary files. However, we can do better now. That's where the require_relative method comes handy. It will load a file relative to the current location. Exactly what we need. All the three lines can be replaced in one fell swoop with:

require_relative "lib/emoticon/version"

Nice. Once the version is loaded we can access it to populate the version number:

spec.version = Emoticon::VERSION

Explaining The Purpose

The next piece of information we're required to give is the summary. There is no limit on how long it should be but my rule of thumb is to use no more than ten words. I kind of see this as a tagline for a gem. A sales pitch if you wish. Let's look at examples of summaries from popular gems:

rake - "Rake is a Make-like program implemented in Ruby"
sequel - "The Database Toolkit for Ruby"
rails - "Full-stack web application framework."

With that said, we add a plain and simple summary:

spec.summary = "Display emoticons in your terminal"

The summary, for example, is used to display information when you run the gem list command for any installed gem package:

$ gem list -d package

Hand in hand with the summary goes description. It is an optional attribute but I do recommend you add it as well. This is where we're expected to provide more details. You can often see developers use heredoc to format many lines to keep their description readable:

  spec.description = <<-DESC
Display emoticons in your terminal. Communicate with your
users not only through words but also emotions.
  DESC

If you want to keep your text indented, you can use the “squiggly” operator introduced in Ruby 2.3. It removes the extra whitespace at the start of each line:

  spec.description = <<~DESC
    Display emoticons in your terminal. Communicate with your
    users not only through words but also emotions.
  DESC

Another approach, for those who wish to keep their spec attributes aligned, is to use a line continuation:

spec.description = "Display emoticons in your terminal. " \
                   "Communicate with your users not only " \
                   "through words but also emotions."

It's your choice. Once you publish your gem, the description will be displayed on the rubygems.org site when users search for your gem. For example, the tty-prompt gem description looks like this:

Attribution

The final bit of information necessary to make our gemspec build successful is the author field. We have the choice of two methods here - author and authors. Regardless of which one we use, the author names will be stored in an array. The author is a convenient shortcut that creates a single element array. Because of this, my preference is to always use the authors method. You never know when you will share the gem development responsibility with others. The following are equal:

spec.author = "Firstname Lastname"
spec.authors = ["Firstname Lastname"]

The Minimal Viable Gemspec

Putting it all together, our minimal gemspec should look like this:

# frozen_string_literal: true

require_relative "lib/emoticon/version"

Gem::Specification.new do |spec|
  spec.name        = "emoticon"
  spec.version     = Emoticon::VERSION
  spec.summary     = "Display emoticons in your terminal"
  spec.description = <<~DESC
    Display emoticons in your terminal. Communicate with your
    users not only through words but also emotions.
  DESC
  spec.authors     = ["Firstname Lastname"]
end

Nice! Now we have everything to build, an albeit useless, gem. If we were to try to build our gem, we would succeed but get quite a few warnings about missing some useful information tidbits and actual source code files! Let's change this.

Choosing a License

One of the extremely useful bits of information is license. From a legal perspective, if you don't provide any license with your gem, it will be assumed that you're the sole copyright holder. In turn, nobody will be able to use your gem in their code.

There are many open-source licenses to choose from. It is common to use a license identifier in place of a full name. The ones I tend to use are MIT and AGPL-3.0. The former to allow anyone to use and modify a gem for any purpose they may like. The latter for when I want any changes to the source to benefit the project and keep it freely available.

Similar to the author option, there are license and licenses methods. For our example, we pick permissive MIT license:

spec.license = "MIT"

Occasionally, gems are released under more than one license. Depending on the use case, one or more licenses may be applicable. For example, the rouge gem is distributed with MIT and BSD-2-CLAUSE licenses. Choosing the right license for a gem is important so take your time in deciding. The choosealicense.com may help you make up your mind.

Contact Information

Being part of an open-source community means that it's good to make yourself available in case people wish to contact you. You can do this by providing your email address with the email attribute:

spec.email = "email@example.com"

You can also provide more than one email address:

spec.email = ["email@example.com"]

A port of call for anyone wishing to learn more about your gem is a homepage. I tend to use a GitHub repository link here as it includes a readme file with project documentation:

spec.homepage = "https://github.com/your-handle/emoticon"

All the configuration options that we used so far are great. But, what if we need a little bit more flexibility and wish to add some custom information?

Metadata Inception

Since RubyGems version 2.0, you can add arbitrary information to your gem release. For example, we can tell people where they can find a changelog or submit an issue. This can be done using the metadata key. I know this is a bit like the Inception film - metadata in the metadata specification file. But not as mind-bending.

The metadata attribute must be a hash. Both keys and values need to be strings. Also, each key cannot exceed 128 characters, and the value cannot be longer than 1024 characters. Otherwise, you will get validation errors. Apart from that, there is no limit on what you can put inside the hash. Specifying metadata is optional but I'd strongly encourage you to use it to help others find out more about your gem.

There is a set of predefined keys whose values are expected to point to various web resources. You can use the following metadata keys to link to sources like a bug tracker or a changelog:

bug_tracker_uri
changelog_uri
documentation_uri
homepage_uri
mailing_list_uri
source_code_uri
wiki_uri
funding_uri

Out of the above, I tend to use the following five keys:

spec.metadata = {
  "bug_tracker_uri"   => "https://github.com/your-handle/emoticon/issues",
  "changelog_uri"     => "https://github.com/your-handle/emoticon/blob/master/CHANGELOG.md",
  "documentation_uri" => "https://www.rubydoc.info/gems/emoticon",
  "homepage_uri"      => spec.homepage,
  "source_code_uri"   => "https://github.com/your-handle/emoticon"
}

The gem's profile page on rubygems.org site will provide links based on the metadata information.

We said a lot about our gem already but one essential bit we haven't added is the source code itself.

What Files to Include?

This is one of the most important but sometimes controversial attributes. What files should you include as part of the release? Should test files be part of it or not? Let me take a step back, and consider what a Ruby gem is and what it isn't. When we share a gem with others, we want them to use our code. Test files, build files like Rakefile or Gemfile, CI configuration files and other such artefacts don't mean anything to our users. These files are helpful when developing but don't serve any purpose in a packaged gem. Similar to, for example, a game distribution. You don't need all the files that were used to create the game. You want only the source files - the game itself!

But things are not as straightforward. In other programmings communities like Perl or Python, it is common to find the test files as part of the package. Tests are used to verify that the package works correctly on a given operating system. Publishing a gem to rubygems.org is not the only way to make your gem available for download. For example, my gems are distributed and repackaged on systems like Debian or ArchLinux. The package maintainers use custom scripts that convert a gem into a suitable package on that platform. As part of this process, they run tests.

I did a survey recently on Twitter to see whether people prefer to keep or remove test files from a gem release. As it turns out, around 64% of devs are in favour of dropping test files.

This mirrors my own thoughts. After all, gems are downloaded and installed most of the time and so it makes sense to keep them as small as possible. By distributing only the source code, which often means files in lib/ directory, you will reduce a gem size significantly. In my case, it often means a reduction between 25% and 50% in size when I skip the tests.

To package files as part of a gem release, we can use the files attribute. There are other attributes that will add files as well which we will cover a little bit later. The value needs to be an Enumerable object that responds to each call.

Most often you will see gems that list their files using git and its ls-files command:

spec.files = `git ls-files -z`.split("\x0").reject do |f|
  f.match(%r{^(test|spec|features)/})
end

The advantage of this approach is that git already knows about all the files in the project. You can exclude files from the release with the gitignore file. The disadvantage is that git is the one that knows about all the files. As we discussed, there are many files that shouldn't be included in the release. Relying on git also makes the gemspec more brittle. The code needs to call out to an external dependency which may or may not be present on the system when the gem is built.

There is a simpler way!

We can use Ruby and the Dir class to help us list source files. We can provide a list of shell glob patterns to seek files that we wish to include. For example, to add all the files from the lib/ directory recursively, we can call the glob method like this:

Dir.glob("lib/**/*")

But there is an even more convenient array-like access method:

Dir["lib/**/*"]

A note of warning about the * wildcard character in the globbing pattern. Any files that start with a dot are considered hidden and excluded from matching the star pattern by default. This is in line with how the shell works on Unix systems. To include hidden files from the lib/ folder you need to use the File::FNM_DOTMATCH flag. Pass the flag as a second argument to the glob method like so (this won't work with the bracket alias):

spec.files = Dir.glob("lib/**/*", File::FNM_DOTMATCH)

Occasionally, you will see gemspecs that declare test files separately with the test_files attribute:

spec.test_files = gem.files.grep(%r{^(test|spec|features)/})

My recommendation is to skip this attribute as this is a thing of the past and no longer needed. Especially, given what we have discussed about the test files.

Autoloading Files

After declaring which files we wish to package, we can also tell RubyGems which files to make readily available from our gem. We met the $LOAD_PATH variable before. To add our lib/ directory location to it and make our code available via the require statement, we use the require_paths attribute:

spec.require_paths = %w[lib]

However, if adding the lib/ folder is all you need then you can skip this attribute as this is the default anyway.

Sometimes though, we also need to distribute an executable file with our gem similar to what bundler or rake gems do. How can we achieve this?

Executables

For a long time, it was common to store binary files in the bin/ directory. But, more recent bundler versions started generating development scripts in the bin/ folder. I've seen people add other scripts for profiling or running tests in there as well. The unfortunate scenario is when a gem's executable finds its way into this folder. If you're not careful a gem's executable can be released with all the development scripts as executables as well. Oops!

The new place to keep the gem's executables is in the exe/ folder. It is common to see executable names deduced with the following:

spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }

There is nothing wrong with this code. But to my taste, it's overly complex. Instead, I recommend being explicit about the executable names. Chances are that's a single file anyway that won't change. So instead consider listing names like this:

spec.executables = %w[emote]

If we attempt to build a gem right now, we will get an error that the bin/emote file cannot be found. That's because by default RubyGems assumes that all executables are in the bin/ folder. Using the bindir attribute, we can prepend each executable name with the new location of the binary folder. In our case, this is the exe/ directory:

spec.bindir = "exe"

Once you specify the bin folder name, all the executable files will be automatically added to the gem files list. There is no need to do it yourself.

Executables and source code files are not the only files we can add to our gem. There are other files that will help people get started and learn more about our project.

API Documentation

The RDoc tool takes any comments in your source code and turns them into a comprehensive documentation. It is compatible with many document formats like Markdown or inline code comments like YARD. By default, it only processes files in the lib/ folder. But, a lot of gems include other standard files like README.md, CHANGELOG.md, LICENSE.txt. These files provide a good source of information to begin using our code. We can instruct RDoc to add extra documentation files via the extra_rdoc_files attribute:

spec.extra_rdoc_files = Dir["README.md", "CHANGELOG.md", "LICENSE.txt"]

The above files will automatically get combined and included in a gem. There is no need to add them to the files attribute listing separately.

During a gem's installation process, the rdoc executable is run on the user's system to generate documentation. To control what users will see when browsing API examples, we can provide various options. Below is an example of a few self-explanatory options you may use to format documentation:

spec.rdoc_options += [
  "--title", "Emoticon - emotions in terminal",
  "--main", "README.md",
  "--line-numbers",
  "--inline-source",
  "--quiet"
]

At this point, we have included all the files. It's time to turn our attention to system requirements. It would be good to tell others which platform, Ruby or RubyGems version our library is compatible with.

System Requirements

We can specify the minimum version of Ruby that our gem works with using the required_ruby_version attribute. As a value, it accepts a string that conforms to the regular version comparison rules. For example, to specify Ruby version 2.4 or higher we can do the following:

spec.required_ruby_version = ">= 2.4.0"

If you don't provide any Ruby version then the default ">= 0" value will be used allowing for any Ruby version. Under the covers, the number is converted into Gem::Requirement object. Our previous example is the same as saying:

spec.required_ruby_version = Gem::Requirement.new(">= 2.4.0")

What version will you support depends on a few considerations. One constraint may be the type of Ruby features you wish to use that are not backwards compatible. The maintenance burden of supporting the older versions may lead you to choose more modern Ruby versions. Given these, I tend to support as many versions as I comfortably can. Even though a version of Ruby may be officially deprecated, it's important to remember that the business world moves slowly.

Next, you can force a minimum supported version of RubyGems. The required_rubygems_version attribute works exactly the same way as the Ruby version one. However, I tend to skip specifying this attribute. Why? By default Ruby installation is packaged with rubygems that limits its compatible versions.

There is a more esoteric attribute called the platform that allows you to limit your gem installation to certain systems only. With it, you can specify an exact processor architecture and system your gem is designed for. For example, to limit your gem to x86 CPU on the Windows system with VC8, you could specify it this way:

spec.platform = "x86-mswin32-80"

And if we wanted to automate the architecture detection to the current system, we could do the following:

spec.platform = Gem::Platform.local

By default, a gem's platform is Ruby which means that the gem can be installed pretty much anywhere:

spec.platform = Gem::Platform::RUBY

Chances are you won't have to ever worry about this attribute. I recommend you skip it.

Our code doesn't live in the vacuum. It often needs other gems for its functionality.

External Dependencies

Our emoticon gem needs to store the icons somewhere in a file. We decided that the most convenient format will be TOML. There is a dependency named tomlrb that does the job. To ensure this dependency is present on the user's system, we can use the add_runtime_dependency attribute. I prefer to use shorter add_dependency alias:

spec.add_dependency "tomlrb"

But declaring a dependency without a version is dangerous. It will make the gem build process issue warnings. You should always strive to select a version your gem is compatible with. Otherwise, you're almost guaranteed that some future release of the dependency will introduce breaking changes and your gem will stop working. There is an art to choosing a version number. It's a risk management strategy. It's good to check if the dependency author uses semantic versioning. Given this, I tend to use the pessimistic operator to constrain a version number:

spec.add_dependency "tomlrb", "~> 1.2"

I won't cover how to specify different constraints. For more details please read Declaring Dependencies official guide.

Apart from runtime dependencies, you can add development dependencies with the add_development_dependency attribute. The process is exactly the same. This attribute was added at the time when Bundler didn't exist. It was a way to setup a gem manually to start contributing. You can run gem install package - - development to install all the development gems. Now, we have Gemfile for declaring our development dependencies. What's the solution?

Based on my recent discussions, I realised that not everyone uses Bundler to develop their Ruby gems. Some developers even have a strong aversion to using it. Whether you like it or not, a few years from now Bundler may not be the tool that everyone uses to install their dependencies. But, you want to be able to continue building your gem. Because of this, I tend to include the most essential development dependencies necessary to test and build a gem. For me these are rake and rspec gems.

spec.add_development_dependency "rake", "~> 13.0"
spec.add_development_dependency "rspec", "~> 3.0"

If you depend on a piece of software to build your gem - add it to your development list. This means that gems like rubocop, simplecov, pry or bundler, though useful, are not essential to the process. So I keep them separate in a Gemfile.

Keeping Secure

Given recent abuses of rubygems.org service by nefarious types, I suggest you do anything possible to protect your gems. At the gem level, the solution is to sign your gem with a cryptographic key. There is a security guide that explains all the details on how to generate your private and public keys. It will also tell you how to verify a gem's integrity. Here, we are only concerned with including what's needed in a gemspec.

By default, a gem will be distributed with your public key if both the public and private keys are generated in the ~/.gem folder. You don't have to do anything else here. Alternatively, you can link to your public and private keys from different locations. It is common to distribute a gem with a public key in the certs/ folder and link to it with cert_chain attribute:

spec.cert_chain = ["certs/yourhandle.pem"]

Then keep the signing key together with other secure keys in the ~/.ssh directory (applies to Unix compatible systems). And only assign the private key via the signing_key attribute when a gem is built, that is, when the gem build package.gempsec command runs:

if $PROGRAM_NAME.end_with?("gem") && ARGV == ["build", __FILE__]
  spec.signing_key = File.expand_path("~/.ssh/gem-private_key.pem")
end

The Final Reveal

You have seen almost all of the available RubyGems attributes and code snippets demonstrating their usage. It may feel a little bit overwhelming at first. What you will also notice is that I introduced each attribute in order matching how I tend to place them in a gemspec. I like to group attributes together based on their conceptual affinity. Let's take a last final look at the entire gem specification in its full glory:

# frozen_string_literal: true

require_relative "lib/emoticon/version"

Gem::Specification.new do |spec|
  spec.name        = "emoticon"
  spec.version     = Emoticon::VERSION
  spec.summary     = "Display emoticons in your terminal"
  spec.description = <<~DESC
    Display emoticons in your terminal. Communicate with your
    users not only through words but also emotions.
  DESC
  spec.authors    = ["Your Name"]

  spec.license  = "MIT"
  spec.email    = "email@example.com"
  spec.homepage = "https://github.com/your-handle/emoticon"
  spec.metadata = {
    "bug_tracker_uri"   => "https://github.com/your-handle/emoticon/issues",
    "changelog_uri"     => "https://github.com/your-handle/emoticon/blob/master/CHANGELOG.md",
    "documentation_uri" => "https://www.rubydoc.info/gems/emoticon",
    "homepage_uri"      => spec.homepage,
    "source_code_uri"   => "https://github.com/your-handle/emoticon"
  }

  spec.files       = Dir["lib/**/*"]
  spec.bindir      = "exe"
  spec.executables = %w[emote]

  spec.extra_rdoc_files = Dir["README.md", "CHANGELOG.md", "LICENSE.txt"]
  spec.rdoc_options    += [
    "--title", "Emoticon - emotions in terminal",
    "--main", "README.md",
    "--line-numbers",
    "--inline-source",
    "--quiet"
  ]

  spec.required_ruby_version = ">= 2.4.0"

  spec.add_dependency "tomlrb", "~> 1.2"

  spec.add_development_dependency "rake", "~> 13.0"
  spec.add_development_dependency "rspec", "~> 3.0"
end

Sorted. Now our gem is ready to be released into the world.

Conclusion

My goal in this article was to remove the confusion that surrounds the RubyGems specification file. I selected the attributes I believe are necessary to make your gem release process smooth. If you stick to the advice in this article, your gemspec will surely spark joy! It won't be cluttered with unnecessary configuration options, cryptic method calls or spurious files. You will keep the gem in good shape by reducing its size significantly and help make your gem installation super quick. You will also avoid annoying your users with unrequested executable files.

I hope you enjoyed this whirlwind tour of the RubyGems specification. If you already have some open-source Ruby gems or plan to release one soon, I hope you have learnt enough to apply it in your working code! Next time, we will take a closer look at the process of turning a specification into a gem.

Stay hungry and creative!

This article was originally published on PiotrMurach.com.

Photo by Jas on Unsplash

My 2019 Annual Review: Not too shabby

Piotr Murach — Wed, 08 Jan 2020 22:02:40 +0000

This is the second time I'm doing this, a review of the past year. I find this process helpful in reminding myself about what I've been up to and increasing my gratitude. My memory is usually quite patchy about the past and I tend to forget the smaller details. However, without those details, the whole wouldn't be possible. So I want to document all those little achievements, failures and events to create a track record that I can look back at and say I went through all this.

Similar to last year, I'm following the format of three questions. Unlike last year, I decided to swap the last question "What am I working toward?" for "What did I learn?". I'll explain the rationale behind this decision a little bit later.

Here are the questions I like to structure my thoughts around:

What went well this year?
What didn't go so well this year?
What did I learn?

Many parts of my life revolve around the technology world and software. This is the lens I'm going to use to filter what to share and what to leave out. With that said, get your favourite drink and strap yourself in, I'd like to walk you through some ups and downs of the past year!

1. What went well this year?

Alright, let’s start with the positives:

Blog writing. This is the most pivotal change for me. For the past five years or so, I harboured a dream of blogging but the mental blocks were too hard to tackle. Finally, at the start of 2019, I decided to take the plunge and try. Funnily enough, with a yearly review. This felt like a good way in.

As with any new habit, I didn't know how to go about it. It turned out that writing is much harder than anticipated. Especially when you're a foreigner writing in another language. Who could've known - ehh?

Early on my writing muse got crippled by many questions. How many articles should I write per month? Do I have anything interesting to write? What should I be writing about? Should the articles be short or long-form? I pushed these questions aside and decided that the best approach is to write widely on many subjects before focusing on specific themes. In the end, I wrote 8 articles in total on different topics. Some articles were more general about software development practices. Whilst others were more technical tutorials. The one article that proved to resonate the most with the readers was "Streaming Large ZIP Files in Rails". The second to follow was "Working with Capistrano: Tasks, Roles, and Variables".

Turns out that there is an appetite for long and informative technical content. Content that dives deep into explaining not only the “how” but “why” of a technical problem. In the sea of three minutes articles, long-form articles seem like a breath of fresh air.

Recording Screencasts. I'm not going to hide it, I felt immensely privileged to work with Avdi Grimm and his team from rubytapas.com. We recorded two professional screencasts about performance testing in Ruby. These episodes distil my thinking about the value of automated performance tests on refactoring your code for speed:

It was great to peek behind the curtains and see how screencasts are made. Avdi's team was very welcoming with everyone being a consummate expert in their role. I enjoyed seeing how a single tapas episode goes from an idea to a polished screencast. Having a deadline and required deliverables each week kept me focused. This was a moment that showed me that focusing on one thing is key to being productive.

Open-sourcing software. I created a whopping nine new gems! The highest number ever. The gem names reveal my meandering path through different subjects this year. I started with the creation of a cryptocurrency-related gem. Then switched to memory optimisation. Next, I devoted a patch of time to terminal related gems. Finally, I finished the year with some text processing gems:

At the end of the year, the total download of gems reached 80 million! Not so long ago, I remember I was very happy to have 2 million downloads. It's humbling to think that my software powers so many Ruby libraries and applications.

In addition, I bought a ttytoolkit.org domain to signal my commitment to all things connected with TTY Ruby gems. The site had 10,000 organic visits this year! It also underwent a significant redesign to make it easier to find components needed. I added a new projects section to showcase terminal applications built with TTY gems. I want this to become a central place for the Ruby community to find resources to craft powerful terminal apps.

Last but not least, thanks to GitHub sponsorship, I got my very first two supporters. You are amazing! Having this support provides a validation to me that what I'm creating is useful and helps others.

International Travel and Conferences. I beat my personal flying record - I boarded planes 10 times in a single month! For the first time, I travelled to Belarus, Thailand, Ukraine and the United States.

America was always on top of my list of countries to visit. I proverbially wet my pants when I was accepted to speak at the Abstractions Conference in Pittsburgh. On my way to the conference, I decided to stopover in New York for a couple of days and turn into a full-on tourist. So, landing in New York was a dream come true. The three-day conference in Pittsburgh was a blast. It was jam-packed with star speakers like Chris Coyier, Zed Shaw and Simone Giertz. The weirdest thing about Pittsburgh is how very familiar it felt. I could find many parallels with Sheffield where I currently live. An industrial city, next to a river with an ambition to become something more.

I like to combine speaking at a conference with sightseeing when visiting a new country. Spending one week in Thailand, Bangkok ticked all the boxes. The Ruby conference had a great community feel. The speaker's treatment was stellar. We went on a night boat ride around Bangkok. I made new friends and even more connections. Thai food didn't disappoint. I had so much fresh fruit. Now, I'm a big fan of durian fruit. I bought virtually anything durian flavoured. Peanuts, sweets, cakes - you bet! One thing I missed out on during my stay is a Thai massage! Hopefully, I will be back soon to amend this.

Trips to Belarus and Ukraine were motivated by my desire to discover more of Eastern Europe. This is the region of the world where I'm from and I want to get to know it better. Both Minsk and Kyiv left me in awe with their architecture. I love big brutalist architecture of the communist era. As expected, the Slavic people were warm and welcoming. I also spent much more time in Poland - nearly half of the year. This is another place I want to explore more. This time with my parents, we travelled to the city of Lodz. A place where the Polish industrial revolution started - the equivalent of Manchester.

Reading. I read a total of 37 books. Not too bad, but also not a very good year for reading. The books that stood out for me the most are “Bad Blood” by John Carreyrou and “Masters of Doom” by David Kushner. The former compelled me to write a review and follow up article that explored the culture of fake software products. The latter was more of a nostalgia trip to my childhood days spent shooting Nazis in Wolfenstein 3D and monsters in Doom's Mars laboratories.

2. What didn't go so well this year?

Here's where things weren’t as rosy:

Conference speaking. Even though I have spoken a handful of times at conferences, they had a significant impact on me. Among the positives were that I got to listen and talk to talented developers. I got supercharged with ideas on what to learn and create. However, I also realised that there is a hidden cost to giving international talks. You need to find money for travel, food and accommodation as well as take time off work. All these costs spiral into large chunks of money which are difficult to sustain.

Speaking at conferences is a huge privilege and I want to continue sharing my experiences. No doubt that I do enjoy presenting and meeting people face to face. But, I need to change my approach before I drain my savings! From now on, I intend to be a bit more selective with my applications whilst still scratching my itch for conference speaking. Plus speaking is just one of the ways to share your experience and knowledge with others. I can do other impactful stuff like writing articles or recording screencasts.

Health. Somewhat corollary to conference speaking, taking care of my health took a back seat. It's embarrassing to think how little effort I made to exercise. I hardly went to the gym, did very little swimming and cycling. This needs to change! My short-term thinking and working myself to the ground is not a good life strategy. Duh!

After my return from speaking at Ruby conference in Belarus, I came down with pneumonia and ended up in a hospital. A long course of antibiotics put me back on my feet. In my recent memory, I don't recall experiencing anything more agonising. It was pure pain even moving in bed. I'm not sure what I can do to prevent this though. One idea is to purchase an antibacterial mask that I could take with me on planes. They are a breeding ground for bacteria.

Programming. You may look at my open source contributions and be a bit perplexed here. Indeed, I did a lot of programming in and outside of work. At work, I worked with Vue framework for the first time to create a small game. So learning new concepts entered the picture. But, I haven’t done much conscious practice to get better as a programmer. One of my failures was not learning a new programming language, in particular, a Lisp inspired one like Racket. Most of the time the best ideas and solutions come out from studying a new language. I like how this provides me with a new perspective. So I need to turn my wishes into an actionable plan and prioritise regular practice.

3. What did I learn?

In the previous annual review, I tried to predict the future. You know what? It doesn't work. For starters, I'm poor at predicting what I'm going to do the next month, let alone next year. A public announcement of goals is counterproductive for me. It saps my motivation and creates a weird sort of pressure to deliver things I'm not fully sure are possible. I agree with Jason Fried's sentiment expressed in the article "I've never had a goal". I prefer to see a new year as a continuation of my past efforts - but stronger and better. I also read somewhere that stating goals publicly makes you already feel as though you have achieved them. So that's that. What's worse, failing at your goals will make you feel bad at the end of the next year when you realise you failed most of them. That sucks.

Instead, what I like is the idea to "learn in public". I want to get into the habit of sucking a little less every day whilst learning and being open about it. So, this year I'm going to focus on lessons learnt. I split my learnings into two broad categories: general habits and technical subjects.

From my experience I have learnt that:

Iteration wins over any attempt at perfection. Whatever I'm working on, I try to get to the first draft, release or whatever quickly. Reaching the end of what I plan to do means following the shortest path possible. Basically, I try to get the stuff done. Slow code can always get faster. Wrong API method can be changed. Article paragraphs can be rewritten. Mistakes can be amended. Nothing is set in stone. So there is no need to wait until all is perfect. My blog writing experience confirmed to me that similar to programming, writing is an iterative process of constant improvement.

“If you want to be a writer, you must do two things above all others: read a lot and write a lot.” - Stephen King

Finishing things leads to progress. I have a dozen or so articles that I've started and never finished. I felt inspired to begin writing but quickly changed to do something else. Same for open source, I created a few projects that never saw the light of day. Don't even get me started on reading books. The list of unfinished books taunts me anytime I open my Kindle. I have this weird habit of reading simultaneously many books. Apparently, this leads to “ideas sex” but I'm not sure my brain is a very fertile ground. The Forbes list is safe from me for now.

Later last year, I got into the habit of working on one project at a time. Don't get me wrong, I still have a few things going on simultaneously. I can be writing an article or coding a project. But within the respective domain, I only do one thing. I'm allowed to write only one article, create one open source project, read one book. You get the idea. Once whatever I'm up to is finished, I'm allowed to move on to something else. This has already led me to get more things done. If you see me break this rule, can you virtually slap me with a reminder of this?

Quality requires patience. If I don't get to do something immediately, not all is lost. Like wine, projects need time to mature and become good. I try not to rush the process. Thinking what my article should be about or scoping software project requirements take time. I let things marinate for a bit before jumping in on them. This has the side benefit of avoiding creating absolute crap. In the jungle of digital age content and projects, quality matters. Shipping subpar stuff may do more harm than good. Legacy is a thing, and I don't want it to be full of things I'm not happy about.

From the technical side, I filled gaps in my knowledge about:

Bitcoin Architecture. I decided to dig more into underlying data structures in cryptocurrency. As part of my learning, I created merkle_tree Ruby gem. It cements my understanding of how a binary tree can provide tamper-proof data verification. Whether you believe that cryptocurrencies will replace fiat money or not, the underpinning technology is fascinating. It demonstrates the top thinking in securing distributed systems.

Linux Interface. This operating system underlines everything I do as a programmer. Recently, I'm drawn more and more towards DevOps. I know this is a mixed bag of many terminologies, practices and approaches. But, it aligns well with my own inclination to create tooling for other developers. Continuous delivery is part of it. There are many tools like Docker and Kubernetes for improving software delivery. I want to be part of it. Because of this, I decided to dive deeper into Linux, especially networking layers. You can probably guess what the future articles will be about.

Conclusion

As suggested by my title, this wasn't the greatest year ever. Equally, it wasn't a total write-off either. I was hoping to get accepted to speak at more conferences. Unfortunately, or fortunately, I channelled rejections into producing more content like open source software and blog articles. All in all, it was a year of fast learning that I hope to continue this year. Please don't hesitate to contact me with any questions, comments or ideas you may have. Thank you for reading. Enjoy 2020!

This article was originally published on PiotrMurach.com.

Photo of me speaking at Abstractions Conference in Pittsburgh, USA.

Working with Capistrano: Environment Variables and Remote Commands

Piotr Murach — Mon, 30 Dec 2019 11:51:26 +0000

In the "Working with Capistrano: Tasks, Roles, and Variables" article, we explored Capistrano's fundamental concepts. We wrote two useful tasks. One for logging into a server and another for accessing Rails console in a production system. This time around, we're going to write a more involved script composed of many tasks. By themselves, these tasks won't accomplish too much. But the sum of them all will help us automate the installation and upgrading of Ruby throughout the lifetime of our application.

Not so long ago, every deployment server that I worked with had only one version of Ruby installed. This system-wide installation forced all Rails applications to use a very specific version of Ruby, for example, 2.1.5. You were not only stuck with a Ruby version but also married to a particular Rails release that had a minimum Ruby requirement. It doesn't take much to realise the shortcomings of this approach. Any Ruby upgrade requires a new production server setup and a costly migration effort. Who has the time and money for this? From a reliability point of view, Ruby becomes a single point of failure. If Ruby were to become corrupted in some way then this would affect all the web applications on the same server.

The way to ease this pain is to use a Ruby version manager. A tool that will allow many different versions of Ruby coexist on a single server. What follows, each Rails application can have its own Ruby environment and be updated independently. Developers can easily change Ruby during the lifetime of their application without affecting other projects - a nice bonus.

There are many great tools such as RVM or chruby that can help you manage Rubies. But, rbenv is my preferred choice. It strikes a perfect balance between features and simplicity. The project is also maintained by Ruby core team members and that's what I will demonstrate in this article. While we explore rbenv, we will lift the veil and examine more Capistrano techniques to add to your toolbox.

In the code samples that follow, we'll install files on a server in some places and read them back from others. I'll assume a Unix like operating system such as Linux since that's what I'm most familiar with. You may need to adjust a few bits according to your operating system of choice.

With that out of the way, let's dive into rbenv installation.

Where to Install rbenv?

Before we decide where to install rbenv, we will want to name and organise our new task first. A great feature that Capistrano inherits from Rake build tool is the namespace method. It gives us a way to logically group tasks under one name. A side benefit is that we can reuse similar task names that otherwise could conflict with the built-in ones. I tend to group the tasks based on the overall function that ties them together. The rbenv as a namespace seems to fit well. We'll go ahead and inside lib/capistrano/tasks directory create a file called rbenv.rake with the :rbenv namespace:

# lib/capistrano/tasks/rbenv.rake

namespace :rbenv do
  ...
end

To install rbenv, we need to decide where we want it installed first. There are two primary ways to install rbenv: system-wide and per user account. What's the difference? The system-wide installation means that a given Ruby version is installed only once per server in a central location. For example, on a Unix system that would be /usr/local/rbenv. Conversely, user account installation restricts a given Ruby to locations accessible to that particular user. Usually, this is a hidden folder inside the user's home directory, for example, $HOME/.rbenv. Choosing the latter may lead to having similar Ruby versions exist on the same server. It is a balancing act.

As we discussed before, the system-wide installation suffers from a single point of failure issue. Because of this, by default, we're going to install rbenv for each deployment user account. But we will expose :rbenv_path variable to allow configuration of a custom path. To do so, we will create a helper method rbenv_path that will hold a rbenv installation path, by default, set to the user's home directory:

def rbenv_path
  fetch(:rbenv_path, "$HOME/.rbenv")
end

The fetch method accepts as a second argument, a default value that will be used when no value is set in your script. There is another way we could do this by using a block.

def rbenv_path
  fetch(:rbenv_path) { "$HOME/.rbenv" }
end

The difference is that the default value provided via the block will be lazy evaluated. This means that the evaluation of default value will only happen when the rbenv_path method is first used. This comes handy in a situation when you have, for example, a dynamic value that depends on the evaluation of other methods. We are going to stick with the second argument option here.

One aside: when working with the filesystem paths, it's better to represent them as data types rather than strings. What does this mean? A string is a poor representation for a system path. It doesn't give you any extra information, for example, whether a path is relative or absolute. Ruby's standard library includes Pathname that can wrap path string representation. Unlike string, the pathname object can tell us information about the path it represents and enhance our path operations. We can build paths in a system-independent way. Joining two paths is as easy as adding them together. The pathname object will automatically use the correct file path separator. If you want to learn more, I recommend Paths aren't strings article.

Armed with this knowledge, we change rbenv_path to return a pathname object instead:

# lib/capistrano/tasks/rbenv.rake

require "pathname"

namespace :rbenv do
  def rbenv_path
    Pathname.new fetch(:rbenv_path, "$HOME/.rbenv")
  end
end

Now, we need to find a way to tell Capistrano to use the rbenv_path method to find rbenv executable and make it available for other tasks to use.

The PATH Variable

When deploying applications, Capistrano uses what is called a non-login shell. In other words, Capistrano doesn't assume anything about the underlying system apart from establishing an SSH session. There is no visual terminal attached to this session. None of the shell configuration files, like a Bash profile file, are loaded. This is very much desirable behaviour that makes our deployments safe and repeatable. But it also means that we cannot rely on any configuration environment variables to be preloaded and available for our scripts.

Specifically, we cannot rely on the system to tell our scripts the location of rbenv binary installation. How can we work around this? We need to roll up our sleeves and show Capistrano where rbenv is installed ourselves. The way we can do this is by modifying the $PATH environment variable. This variable holds a list of directories delimited by a colon that the Unix shell uses to search for executable files when running commands. To check the value of your path, you can run the following in your shell:

$ echo $PATH

Your path may look something like this:

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

This begs the question...how can we instruct Capistrano to change what locations does the $PATH variable point to? Remember, we're trying to make rbenv available for the non-login shell.

Mapping Binaries with SSHKit

The heart of Capistrano - the sshkit Ruby gem - is a toolkit for working with servers via SSH. The sshkit is responsible for doing the heavy lifting in terms of defining and executing commands on a server. Besides, it provides access to the current SSH session configuration with the SSHKit.config method. In particular, the config exposes the default_env method that, by default, returns an empty hash. When a command gets executed, all the keys and their values are converted into the corresponding environment variables. Bear in mind that only the keys provided as symbols will change to an uppercase name:

SSHKit.config.default_env[:path] = "$HOME/.rbenv/bin:$PATH"

This is equal to:

SSHKit.config.default_env["PATH"] = "$HOME/.rbenv/bin:$PATH"

Both examples will result in a command being prefixed with a path:

$ PATH="$HOME/.rbenv/bin:$PATH"

The default_env is exactly the place to change the $PATH variable to include the location of rbenv installation path. Specifically, we want to add two directories with executables:

$HOME/.rbenv/bin
$HOME/.rbenv/shims

Now, we're ready to get our hands dirty and make Capistrano locate rbenv. We begin by adding a task called map_bins that will make common Ruby binaries use rbenv executable - but we're getting ahead of ourselves. Let's change the $PATH variable first.

# lib/capistrano/tasks/rbenv.rake

require "pathname"

namespace :rbenv do
  desc "Map Ruby binaries to use rbenv"
  task :map_bins do
    ...
  end

  def rbenv_path
    Pathname.new fetch(:rbenv_path, "$HOME/.rbenv")
  end
end

With the task defined, we read the current value for the :path key to preserve what other tasks may have already configured:

desc "Map Ruby binaries to use rbenv"
task :map_bins do
  path = SSHKit.config.default_env[:path]
end

You may have noticed that we haven't used the on scope. The scope is only needed when we want to execute remote commands on a server or access the server's configuration. None of this is necessary for the task at hand.

Next, we alter the path with the location of rbenv directories. We use rbenv_path helper to add our two rbenv locations. If the path isn't configured, we prepend the two directories directly to the $PATH variable. If the path is configured, we prepend on top of it:

desc "Map Ruby binaries to use rbenv"
task :map_bins do
  path = SSHKit.config.default_env[:path]
  path_env = "#{rbenv_path.join('shims'}:#{rbenv_path.join('bin')}:" +
             (path.nil? ? "$PATH" : path)
end

Finally, we override the SSHKit default_env hash with a new modified environment path:

desc "Map Ruby binaries to use rbenv"
task :map_bins do
  path = SSHKit.config.default_env[:path]
  path_env = "#{rbenv_path.join('shims'}:#{rbenv_path.join('bin')}:" +
             (path.nil? ? "$PATH" : path)
  SSHKit.config.default_env.merge!(path: path_env)
end

But, we're not done yet. If we're going to do anything useful a server like installing Rails, we need to have a way to run Ruby commands like gem or bundle. This is necessary so that we can later issue the following commands to bootstrap a Rails application:

$ gem install bundler
$ bundle install

Let's say we want to run "bundle install" command but there are many Ruby versions installed already. Which version of Ruby would provide the bundle executable? We need to somehow tell our shell which Ruby version to use. Again the $PATH variable comes to the rescue. We can extend the $PATH with the correct Ruby version before running our executable. For example, to use bundle executable from the Ruby 2.6.3 installation, we would run the following in the Unix shell:

$ PATH="$HOME/.rbenv/versions/2.6.3/bin:$PATH" bundle install

But, this command is long, ugly, and hard to automate. Thankfully, the rbenv provides rbenv exec command that can detect the currently loaded Ruby version and include it in our path for us. Since we have already configured Capistrano to find rbenv binary, we can use exec to rewrite our previous command like this:

$ rbenv exec bundle install

Now we know how to write our commands by hand. But how do we tell Capistrano to prepend common Ruby executables like bundle with the rbenv exec command for us? Before we answer this question, it's helpful to understand how an execute method works first.

Running Remote Commands

The sshkit provides an execute method for running any shell command on a server. This is a powerful method that behaves differently depending on the arguments passed in. A peek under the covers reveals a rather succinct and short implementation:

# lib/sshkit/backends/abstract.rb

def execute(*args)
  options = args.extract_options!
  create_command_and_execute(args, options).success?
end

The above snippet doesn't look like much but there is a whole machinery hidden behind the execute method. Conceptually, the process of running any command is split into two parts. The first part takes the command and converts it into an SSHKit::Command object. This object handles assembling a full, shell-ready command from various bits like prefixes, flags and arguments. The second part takes care of executing the command which can be carried away by any of the SSHKit::Backends. This approach lets you create command once and run it using different backends on many servers. This is such a great design decision!

When run with a 'simple' string without any whitespace, the execute hands over the command first to SSHKit::CommandMap. Internally, the CommandMap holds a hash of commands, empty by default, that allow additional configuration:

# lib/sshkit/command_map.rb

module SSHKit
  class CommandMap
    def initialize(value = nil)
      @map = CommandHash.new(value || defaults)
    end
  end
end

By default, if there are no custom prefixes specified, a command is prefixed with "/usr/bin/env". Except for if, test, time and exec, commands that receive no prefixes. For example, if we were to check Ruby version using ruby executable as the first argument:

execute :ruby, "--version"

This would result in the default lookup and the following output:

/usr/bin/env ruby --version

If, on the other hand, we were to provide a command as a raw string with whitespace:

execute "ruby --version"

It would be executed as-is without any prefixing, shell escaping or directory changes. Basically no involvement from Capistrano or SSHKit whatsoever:

ruby --version

This often tricks newcomers to Capistrano who scratch their head in puzzlement as to why a command fails to run on a server. So consider yourself warned.

Prefixing Commands

To change commands before they are run, we can use the command_map method exposed by the SSHKit configuration object. We can then take the CommandMap and chain it together with the prefix method to add new prefixes. Like CommandMap, the returned PrefixProvider stores a hash of command to prefix mappings. By default, any key lookup for non-existing command returns an empty array. Knowing this, we can override a default ruby command prefix:

SSHKit.config.command_map.prefix[:ruby] << "rbenv exec"

Then run our ruby command again:

execute :ruby, "--version"

But this time, we get a very different result:

rbenv exec ruby --version

That was a long but necessary diversion from the map_bins task implementation. Now, we know how to prefix common executables like bundle, gem, or rake with rbenv exec. First, we will add rbenv_map_bins method to allow for the configuration of Ruby binaries defaulting them to the most common:

def rbenv_map_bins
  fetch(:rbenv_map_bins, %w[bundle gem rails ruby rake])
end

Armed with our knowledge of customising commands, we can prefix each binary and ensure we don't change any existing prefixes:

rbenv_map_bins.each do |exe|
  SSHKit.config.command_map.prefix[exe.to_sym].unshift("rbenv exec")
end

Having prefixed Ruby binaries, the complete task for mapping commands will look like this:

desc "Map Ruby binaries to use rbenv"
task :map_bins do
  path = SSHKit.config.default_env[:path]
  path_env = "#{rbenv_path.join('shims')}:#{rbenv_path.join('bin')}:" +
             (path.nil? ? "$PATH" : path)
  SSHKit.config.default_env.merge!(path: path_env)

  rbenv_map_bins.each do |exe|
    SSHKit.config.command_map.prefix[exe.to_sym].unshift("rbenv exec")
  end
end

Now running "bundle install" (and similar) commands will work because they will be executed with the correct Ruby version. We can apply our task by running:

$ cap [stage] rbenv:map_bins

What we have configured so far is for the non-login shell. But, if a user wants to log into a server, the rbenv executable won't be available and none of the Ruby binaries will work. Let's change this!

Modifying Shell Startup File

It would be annoying to force anyone who logs into a server to manually run rbenv setup instructions before they could run, for example, a bundle command.

So what's the solution?

When a user logs into an operating system, this triggers loading of the shell startup files. A shell startup file allows for execution of arbitrary shell scripts, setting environment variables and so on. Sounds like just what we need. We could modify the shell startup file to set rbenv paths for us. This would make Ruby and any related executables available inside the logged-in user shell.

Many Linux distributions like Debian, typically, use a Bash shell as their default shell. When started as an interactive shell, Bash reads a startup file called bash_profile. The following Bash script is all we need to insert into a bash startup file to load rbenv automatically:

if [ -d ~/.rbenv ]; then
  export PATH="$HOME/.rbenv/bin:$PATH"
  eval "$(rbenv init -)"
fi

This snippet of code first checks that rbenv directory exists. When it does, it adds rbenv binary to the $PATH variable and makes it available in the shell. It then runs the rbenv initialisation script.

Now, we have everything we need to create a task called modify_shell_file that will, as you've probably guessed, modify a shell startup file. We use the on scope with all role to specify the servers affected by our changes:

# lib/capistrano/tasks/rbenv.rake

namespace :rbenv do
  desc "Map Ruby binaries to use rbenv"
  task :map_bins do
    ...
  end

  desc "Change shell startup file to setup rbenv"
  task :modify_shell_file do
    on release_roles(:all) do
    end
  end
end

Testing Remote Commands

What we don't want to do is to add rbenv initialisation script to a startup file more than once. If the script is already present, we want to skip adding it. We could use the execute method to run this check but that wouldn't get us far. Why is that? The execute method, though powerful, is used for running commands where we don't care about capturing the final output. We only care that a command succeeds and the script continues or it fails and terminates a task. But, in a situation when we need to decide if a command ran successfully or failed without stopping a task, Capistrano provides the test method. This method is equal to execute with the difference that it doesn't raise an error for non zero exit code. There is even :raise_on_non_zero_exit option you can use to get this behaviour with execute method:

execute ..., raise_on_non_zero_exit: false

To scan a file for matching content, we will use the grep command. It is a basic Unix utility that should already be installed on a server. If a match is found, the grep will finish with a zero exit code - exactly what we need. To skip the task and prevent further execution, we call next. If this task is part of many, the script will continue with the next task. This is a common idiom used in Capistrano plugins. It's worth noting that break or exit would immediately terminate the entire script and raise an error.

So the check to see if a startup file contains rbenv initialisation script looks like this:

desc "Change shell startup file to setup rbenv"
task :modify_shell_file do
  on release_roles(:all) do
    next if test "grep -qs 'rbenv init' ~/.bash_profile"
  end
end

We need to jump through some Unix shell hoops to programmatically insert a multiline string. When executing a multiline command, Capistrano replaces every newline character with a semicolon to essentially collapse a command into a single line. To prevent this and insert rbenv script line by line into the bash startup file, we're going to use the printf utility. If you used Ruby's printf method, the Unix printf utility is going to be very familiar. They both use a format string followed by input arguments, the difference being shell uses whitespace to split inputs.

Let's append the rbenv initialisation script to the Bash profile file:

desc "Change shell startup file to setup rbenv"
task :modify_shell_file do
  on release_roles(:all) do
    next if test "grep -qs 'rbenv init' ~/.bash_profile"

    execute %q{printf "%s\n  %s\n  %s\n%s" 'if [ -d ~/.rbenv ]; then' 'export PATH="$HOME/.rbenv/bin:$PATH"' 'eval "$(rbenv init -)"' 'fi' >> ~/.bash_profile}
  end
end

Having configured rbenv for the login and non-login shell alike, we're ready to install rbenv itself. It was quite a bit of a setup but I promise it was worth it.

Installing Rbenv & Rbenv Build

At this point, we're not going to reinvent the wheel and instead use a rbenv installer. The rbenv installer includes a doctor script that will verify that rbenv is correctly configured. If it isn't then the installation script won't proceed any further and return non-zero exit code. This will stop the installation process in its tracks.

The benefit of using rbenv installer is that it will also install the rbenv-build tool for compiling and installing Ruby. Rbenv by default doesn't include any scripts for installing Ruby, it limits its functionality to only managing Ruby versions. However, in our case, we want to install Ruby as well.

Let's modify the rbenv.rake file and add a new run_installer task:

# lib/capistrano/tasks/rbenv.rake

namespace :rbenv do
  desc "Map Ruby binaries to use rbenv"
  task :map_bins do
    ...
  end

  desc "Change shell startup file to setup rbenv"
  task :modify_shell_file do
    ...
  end

  desc "Install rbenv and rbenv-build tools" 
  task :run_installer do
  end
end

Similar to the task for modifying a shell startup file, we want to skip installing rbenv if it's already setup. To do so, we're going to use bash shell to check for rbenv directory existence:

desc "Install rbenv and rbenv-build tools"
task :run_installer do
  on release_roles(:all) do
    next if test("[ -d #{rbenv_path} ]")
  end
end

We start by saving the rbenv installer location in a method rbenv_installer_url:

def rbenv_installer_url
  "https://github.com/rbenv/rbenv-installer/raw/master/bin/rbenv-installer"
end

One word of caution here. Before using a third-party script, it's best to read through and verify that a script doesn't contain any malicious code first. For extra measure, it's advisable to store such script on an internal network and only download it from there.

Then, installing rbenv is just a matter of downloading the installation script using the wget utility in quiet mode. Once the installation script is downloaded, we're going to pipe its content into the Bash shell to perform the installation process:

desc "Install rbenv and rbenv-build tools"
task :run_installer do
  on release_roles(:all) do
    next if test(" [ -d #{rbenv_path} ] ")

    execute :wget, "-q", rbenv_installer_url, "-O-", "|", :bash
  end
end

And with that we have successfully installed rbenv utility on our server. This way we have opened the doors for automating Ruby installation.

Summary

In this article, we wrote a few fairly complex tasks that gave us a much bigger taste of Capistrano capabilities. We modified environment variables, updated a shell startup file and installed rbenv utility. As we did all this, we explored more of the Unix shell concepts and learnt how rbenv works.

As a side benefit, this article demonstrated to us the extreme flexibility that Capistrano architecture gives us. It lets you provision the application infrastructure the way you need. You describe your requirements using task definitions and, viola! Not sure about you, but I'm impressed with Capistrano design. It embraces Ruby's dynamic nature and object model to provide you with an intuitive way of expressing deployment concepts.

Nice. We’re another level up on our path to Capistrano proficiency. In the next article, we will continue to explore how to combine all the tasks to install a brand-new Ruby.

This article was originally published on PiotrMurach.com.

Photo by Tincho Franco on Unsplash

Working with Capistrano: Tasks, Roles, and Variables

Piotr Murach — Sun, 17 Nov 2019 19:22:20 +0000

Capistrano is a great tool for automating the deployment and maintenance of web applications. I use it to deploy Ruby on Rails and other Rack-based applications. It takes care of all the tedious bits such as installing dependencies, compiling assets or migrating database schemas. The alternative would be to manually log into a server via SSH and run all the necessary commands ourselves. This would open a path to terrible sins and consume a lot of time and patience. Thankfully we don’t have to.

The modular design of Capistrano means that there are plenty of plugins and scripts available. They will make even the thorniest deployment needs possible. Even if you don’t find what you want off the shelf, Capistrano flexible nature will help you write easily your own tasks. You will be able to tell Capistrano exactly how you want your deployment to look like. It will then prepare the server and continue to help you maintain updates to your application.

In this series of articles, my goal is to provide core concepts and explore more advanced features of the Capistrano deployment tool. To do so, we will dive into different real-life examples of solutions to common problems maintaining web applications in Ruby. These examples will teach you the ins and outs of how to write Capistrano scripts to get the job done quickly. You should also develop a better sense of the internals of Capistrano.

All the code samples in this article assume Capistrano 3. At the time of this writing, this means installing Capistrano 3.11.2. Please read the official guides on how to set up Capistrano in your project.

With that said, let's start!

Capistrano Tasks

At its core, Capistrano uses tasks provided by the Rake build tool to describe various actions that will update and deploy your application. It then supercharges these tasks with its own intuitive DSL that helps execute commands in Unix shell on a local machine or a remote server.

For a long time, I used to manually log into different servers over an SSH connection to perform one-off updates. This, in turn, meant that I needed to know and type in the shell the deployment user and server name each time I wanted to, for example, access a staging server. This is tedious and error-prone!

Surely, there is a better way?

Ideally, I would like to have a consistent way to login to any server based on the deployment environment without needing to know the details. The equivalent of saying "I want to login into the staging server". In Capistrano, this would translate into executing the following command:

$ cap staging login

Let's see how we could implement this command as a task.

By default, Capistrano will automatically load files from lib/capistrano/tasks directory. We'll go ahead and create login.rake file inside that directory. Notice the *.rake extension which is typically used to name Ruby files that contain Rake tasks. Rake provides a method called desc that can be used to add a task description. The description will be displayed when the user runs cap - -tasks to see all available tasks. Rake also exposes a method called task that can be used to express the behaviour of a task. Immediately after the description, we use the task block to register a new, albeit empty for now, :login task:

# lib/capistrano/tasks/login.rake

desc "Login into a server based on the deployment stage"
task :login do
  ...
end

So far the login task will do nothing. To be able to log in, we need to gather information about our application server. This requires us to talk about the concept of roles.

Capistrano Roles

To give us a fine-grained control over which tasks are run on which servers, Capistrano uses a concept of a role. For example, you may want to apply a task only to a database server and skip it when deploying to a web server. The role acts as a filter that tells Capistrano to group tasks with the same role together before executing them on a server matching a specific role.

There are three common roles used in Capistrano tasks:

The app role is used by any task that runs on an application server, a server that generates dynamic content. In Rails this is puma server. Capistrano's built-in tasks deploy:check, deploy:publishing or deploy:finished are all run in this role.
The db role is used by tasks that execute against a database server. For example, the capistrano-rails plugin provides the deploy:migrate task for migrating the Rails database schema.
The web role is used by tasks that deal with web servers that serve static content, think Nginx here. A community created capistrano3-nginx plugin uses this role in all of its tasks like nginx:start, nginx:reload or nginx:site:disable.

We can also define custom roles of our own. For example, a role called redis that will only run tasks related to Redis database instance:

role :redis, "redis-server"

If we want our task to match any server and we don't care what role a server has, we can use the all role. Capistrano's built-in tasks use this role to stay flexible.

In our case, we will be deploying to a single machine that hosts the application, web and database servers. To apply all three roles app, db and web to our server, we use a shorthand definition via the server method:

# config/deploy/staging.rb

server "staging-server.example.com", roles: %w[app db web], primary: true

If you examine the above server definition, you'll notice a property called primary. This property will let Capistrano know the order of precedence when running tasks. The tasks with roles associated with the primary server will run first. This becomes especially useful when your application is split between many hosts. If that's the case, then we could expand our definition to focus on roles instead:

# config/deploy/staging.rb

role :app, "app-server.example.com"
role :db, "db-server.example.com", primary: true
role :web, "static-server.exmaple.com"

Moving ahead, we’ll see how to apply our defined roles in a task.

Getting Host Configuration

To access our server configuration, we will use the roles helper method. This method accepts one or more role definitions and returns a list of all matching host instances. We specify an :app role to get, in our case, a list with only a single server configuration instance of Capistrano::Configuration::Server.

Among many innovations, Capistrano 3 introduces more modular architecture. It offloads management of SSH session to another dependency, a Ruby gem called sshkit. The SSHKit via its DSL provides an on method that allows us to run commands described in a block scope on many servers. As an argument, the on takes an array of host configuration objects and then uses SSHKit::Coordinator to run commands in parallel on each host:

# lib/sshkit/dsl.rb

module SSHKit
  module DSL
    def on(hosts, options={}, &block)
      Coordinator.new(hosts).each(options, &block)
    end
  end
  ...
end

If you have more than one host configured, it's probably better to instruct Capistrano to login to each server in a sequence:

on roles(:app), in: :sequence do |server|
  ...
end

Inside the on scope we gain access to the host instance that contains information about our server. Essentially, we're saying "on this server do the following things":

# lib/capistrano/tasks/login.rake

desc "Login into a server based on the deployment stage"
task :login do
  on roles(:app) do |server|
    ...
  end
end

Capistrano Variables

To ssh into a server we need to know the user name, the server name and the path to log the user into. For your configuration needs, Capistrano provides a method called set that allows you to configure variable like settings globally or for specific tasks and make them available to other parts of your script. For example, we can set the :user and :deploy_to variables in the following way:

# config/deploy/staging.rb

set :user, "deploy-user"
set :deploy_to, "/path/to/deploy/directory"

To help us read our configuration variables, Capistrano makes available a method called fetch. For example, we can use it to get the user name and deployment path:

# lib/capistrano/tasks/login.rake

desc "Login into a server based on the deployment stage"
task :login do
  on roles(:app) do |server|
    user = fetch(:user)
    path = fetch(:deploy_to)
    ...
  end
end

Now that we know the user name and path to login into, we can construct a URI that we will use as an argument to ssh command utility. We build the URI string by concatenating the user name, server name and port number. The code will handle the cases where the user name or port are missing:

# lib/capistrano/tasks/login.rake

desc "Login into a server based on the deployment stage"
task :login do
  on roles(:app) do |server|
    user = fetch(:user)
    path = fetch(:deploy_to)

    uri = [
      user,
      user && ‘@’,
      server.hostname,
      server.port && “:”,
      server.port
    ].compact.join
  end
end

The SSH Command

We're now ready to run the ssh command with the URI and deployment path. Before we do that though, there are a few interesting things to note here. In particular, the -t flag which instructs ssh utility to run in teletype mode. What is teletype mode? By default, Capistrano deploys via SSH connection without attaching a visual terminal. There is no need to show beautiful interfaces as nobody can see anything anyway. But, in our case we want the user to have access to all the shell capabilities.

After the -t flag we specify the URI location of the server. Then in quotes enclose shell commands to run once logged in. We string together two shell commands. First, we navigate to the website root directory and then execute a $SHELL variable. The $SHELL is an environment variable that holds the location for the default shell, usually Bash. Why do we need to start the shell? Without executing the shell, the ssh command would finish and log us out. But we want to remain logged in, that’s the whole point of this task! In case of bash, we use -l flag to invoke it as if a user logged in. We do this to preload all the configurations in hidden files such as the Bash profile.

The final command string will look like this:

"ssh -t #{uri} 'cd #{path}/current && exec $SHELL -l'"

We use Ruby's exec to run our ssh command locally:

exec("ssh -t #{uri} 'cd #{path}/current && exec $SHELL -l'")

Putting everything together, the entire task looks like this:

desc "Login into a server based on the deployment stage"
task :login do
  on roles(:app) do |server|
    user = fetch(:user)
    path = fetch(:deploy_to)

    uri = [
      user,
      user && '@',
      server.hostname,
      server.port && ":",
      server.port
    ].compact.join

    exec("ssh -t #{uri} 'cd #{path}/current && exec $SHELL -l'")
  end
end

We talked about logging quickly into a server from our machine. But this is not the only way to enhance our everyday workflow with Capistrano.

Remote Rails Console

Often I need to jump straight into Rails console on a server and run some queries against a live database. I could use the login task and then once logged in, type in a full command to open the Rails console. But this would get annoying quickly. How about we script Capistrano to log us straight into a production Rails console instead? The way we would like to do this is by running the following command:

$ cap production console

To implement the above, inside the same login.rake file, we write a bare-bones :console task with a description:

# lib/capistrano/tasks/login.rake

desc "Open Rails console on a server based on the deployment stage"
task :console do
    ...
end

Like the login task, to access our server, we need to know the user name and deployment path. Besides, to open the Rails console we also need to know the environment that the Rails application runs in. But, we can glean this from the :rails_env configuration variable which is configured by the capistrano-rails gem:

# lib/capistrano/tasks/login.rake

desc "Open Rails console on a server based on the deployment stage"
task :console do
  on roles(:app) do |server|
    env = fetch(:rails_env)
    user = fetch(:user)
    path = fetch(:deploy_to)
    ...
  end
end

Finding Rails Executable

Before we can open Rails console via SSH, we need to deal with one extra issue. Because we don't have access to a pseudo-terminal, none of the user profile configuration files will be loaded. We need to somehow tell Capistrano where to find the rails executable and in turn where to find Ruby installation. For me, the preferred way is to use the rbenv utility to manage Ruby installation and the Bundler for gems installation. If you use another Ruby manager please adjust your paths appropriately.

For the rbenv path, we use the deployment user local installation location of $HOME/.rbenv. Rbenv has a concept of shims, which are lightweight executables that map installed Ruby commands such as gem, irb or rails to rbenv exec command. The rbenv exec will then prepend a specific command with a correct Ruby installation path. To execute the rails command, we need to use bundle shim which will ensure all our dependencies are loaded. We can access the bundle executable in $HOME/.rbenv/shims directory. As a last step, we inform the rails console command about the environment we're using via the -e flag. The full command to open rails console looks like this:

console_cmd = "$HOME/.rbenv/shims/bundle exec rails console -e #{env}"

With the final piece of the puzzle assembled, we can ssh into the current Rails application directory and open the Rails console:

"ssh -t #{uri} 'cd #{path}/current && #{console_cmd}'"

Now let's tie everything together in the final script:

desc "Open Rails console on a server based on the deployment stage"
task :console do
  on roles(:app) do |server|
    env  = fetch(:rails_env)
    user = fetch(:user)
    path = fetch(:deploy_to)

    uri = [
      user,
      user && '@',
      server.hostname,
      server.port && ":",
      server.port
    ].compact.join

    console_cmd = "$HOME/.rbenv/shims/bundle exec rails console -e #{env}"

    exec("ssh -t #{uri} 'cd #{path}/current && #{console_cmd}'")
  end
end

Removing Duplication

Now all you eagle-eyed readers probably noticed that we have a bit of repetitive code between the tasks. Let’s tackle it so we can finish strong and improve maintainability. One thing I like to do before I remove any clutter is to repeat code verbatim, like I did in the second task. This is so that I can visually highlight the parts that are the same. We can see that apart from the rails environment variable and ssh command to run, the tasks are identical.

In the spirit of removing duplication, we will move the common part of setup and running ssh command into its own method called run_ssh_with. The method will accept as arguments the server configuration and command to run:

# lib/capistrano/tasks/login.rake

def run_ssh_with(server, cmd)
  user = fetch(:user)
  path = fetch(:deploy_to)

  uri = [
    user,
    user && "@",
    server.hostname,
    server.port && ":",
    server.port
  ].compact.join

  exec("ssh -t #{uri} 'cd #{path}/current && #{cmd}'")
end

Thanks to the run_ssh_with, we can simplify both tasks into:

# lib/capistrano/tasks/login.rake

desc "Login into a server based on the deployment stage"
task :login do
  on roles(:app) do |server|
    run_ssh_with(server, "exec $SHELL -l")
  end
end

desc "Open console on a remote server based on the deployment stage"
task :console do
  on roles(:app) do |server|
    env  = fetch(:rails_env)
    console_cmd = "$HOME/.rbenv/shims/bundle exec rails console -e #{env}"
    run_ssh_with(server, console_cmd)
  end
end

That's so much nicer! Before we finish though, there is one more thing we can do to appease our programmer nature that craves optimization in every keyboard keystroke. We can create aliases for our two tasks! Rake doesn't provide an alias feature per se but we can fake it. The way to do it is to define a new task whose execution will depend on invoking another task first. Let’s abbreviate both tasks to single letters:

# lib/capistrano/tasks/login.rake

task :c => :console
task :l => :login

Summary

We've finished a whirlwind tour of Capistrano. It's a lot to take in if you've never written a Capistrano script before. Even if you have, hopefully, this has clarified for you some of the Capistrano's features. This article should give you a general understanding of how a task works, how to configure variables, and how to run any command locally.

The login and console tasks provide a quick way of automating rudimentary jobs. One important side effect of the tasks is consistency across Rails projects. You don’t need to know the details to be able to quickly search the project files or query database on a remote server. These little efficiencies accumulate over time and create a smooth development workflow. If you have similar useful Capistrano tasks please share them!

This article was originally published on PiotrMurach.com.

Photo by Rock'n Roll Monkey on Unsplash

Streaming Large ZIP Files in Rails

Piotr Murach — Sat, 26 Oct 2019 12:37:58 +0000

Recently, I needed to add a "Download all" button in a Rails application for managing meeting assets. Specifically, this magic button would allow attendees to download all the meeting documents in a single zip file. Before I explain how I tackled streaming of large zip files, let’s first look at the files storage implementation.

How the Files are Stored

Since Rails 5.2, there is a baked-in solution for handling file uploads named Active Storage. What I like about this is that it doesn’t require you to alter any of your application existing models with extra columns to support file uploads. You can easily add file uploading to any model in your application. Active Storage achieves this flexibility via a polymorphic association in the ActiveStorage::Attachment model, which is a join model between your record and the ActiveStorage::Blob model:

# rails/active_storage/app/models/active_storage/attachment.rb

class ActiveStorage::Attachment < ActiveRecord::Base
  ...
  belongs_to :record, polymorphic: true, touch: true
  belongs_to :blob, class_name: "ActiveStorage::Blob"
  ...
end

The ActiveStorage::Blob record contains all the necessary file metadata. Among them a unique key to the storage location, filename, content type, byte size and more. Later, we will use the Blob model to access our storage files content bit by bit.

Despite using the Active Storage, the advice in this article is storage agnostic. There are many other great alternatives for handling file uploads such as Carrierwave, Dragonfly or Shrine. But we're going to stick with the default storage solution here.

We aren't going to go cover setting up active storage from scratch. Please follow the official Rails guides on how to do it in your project. Instead, our starting point will be a Meeting model. The only thing we need to do to be able to attach many documents to our meeting is to use has_many_attached method:

# app/models/meeting.rb

class Meeting < ApplicationRecord
  has_many_attached :documents
end

A Common Approach

The common solution to downloading zip archives is to create an entire zip file with all the files first. This means reading each file into memory first before writing it back to the disk as part of a single archived zip file. Once done, the web server will begin sending the zip file to the client.

Unfortunately, this approach has a few drawbacks. Depending on the sizes of files in the archive, you may need a lot of memory and disk space to generate a zip file. Even if you have ample resources, your application user may need to wait a long time before their browser starts downloading the archived file. The perceived lag and inactivity will negatively impact their experience.

So what's the alternative?

Tricks Up the Streaming Sleeve

The solution is to stream a zip archive immediately to the client as the very first file is being read from the disk. This way, we don’t even have to wait for the file to be fully read. We can start streaming in smaller byte chunks without creating a zip file upfront. This approach removes the need for large disk space and reduces memory allocations as our zip content is sent over the wire in small chunks. With decreased latency and faster download time, the user experience improves significantly.

To stream large files in a single zip archive, we're going to use the zip_tricks gem. The library boasts the ability to handle millions of zip files generated per day. So, we have our backs covered with the volume of archived files here as well.

Let's add the download button.

The Download Button

We start by creating a request path that will handle streaming of our download. To do so, we add a download route to the meeting resources that will use a custom controller:

# config/routes.rb

resources :meetings do
  member do
    post :download, to: "zip_streaming#download"
  end
end

It's worth noting that we chose the POST method in place of GET. It’s so that we can skip having to deal with templates rendering in our controller action.

Next, we add a custom controller zip_streaming_controller.rb with a download action that will handle streaming of the zip archive:

# app/controllers/zip_streaming_controller.rb

class ZipStreamingController < ApplicationController
  before_action :set_meeting

  def download
  end

  private

  def set_meeting
    @meeting = Meeting.find(params[:id])
  end
end

And finally, in our view, we add the "Download all" button that will trigger zip file download:

# app/views/meetings/show.html.erb

<%= button_to "Download all", download_meeting_path(@meeting),
    method: :post, data: { "no-turbolink" => true } %>

Now we're ready to discuss the implementation of the download action.

File Download Response Headers

When discussing downloading files of any kind, we need to touch on the subject of HTTP response headers and, in particular, the Content-Disposition header. The Content-Disposition response header tells the browser how to display the response content. If the browser knows how to handle the MIME type, the inline value displays the content as part of the web page. Otherwise, the content is immediately downloaded. We can also instruct the browser to always download the content and save it locally. To do this, we use an attachment disposition. When the "Save as" dialog is presented, by default, the filename is the last segment of the URL. To change this, we can use filename attribute to name the downloaded file:

response.headers["Content-Disposition"] = "attachment; filename=\"download.zip\""

We also want to inform the browser about the content type. To do so we use the Content-Type response header with the "application/zip" MIME type:

response.headers["Content-Type"] = "application/zip"

To help the user identify their download, we name our archived file by the meeting title using an easy to read slug identifier. Putting it all together, we add the Content-Disposition and Content-Type response headers to the download action:

def download
  zipname = "#{@meeting.slug}.zip".gsub('"', '\"') # escape quotes
  disposition = "attachment; filename=\"#{zipname}\""

  response.headers["Content-Disposition"] = disposition
  response.headers["Content-Type"] = "application/zip"
end

There are many quirks when dealing with the filename attribute of a Content-Disposition header. For starters, the filename may contain special characters that need escaping. To handle the edge cases in filenames and make the solution more robust, we can use ActionDispatch::HTTP::ContentDisposition and the format method:

def download
  zipname = "#{@meeting.slug}.zip"
  disposition = ActionDispatch::Http::ContentDisposition.format(disposition: "attachment", filename: zipname)

  response.headers["Content-Disposition"] = disposition
  response.headers["Content-Type"] = "application/zip"
end

However, we're not done with the response headers yet. Since we're dealing with streaming, we do not know the byte length of our content. When the Content-Length header is omitted, the browser will assume that the content will be streamed in chunks in a single request/response cycle. So we ensure that the header is removed:

response.delete_header("Content-Length")

If our meeting app becomes widely successful, we want to be kind to our server resources and send a cached copy when possible. To control cache settings, we use the Cache-Control header with "no-cache" directive. Contrary to popular belief, the "no-cache" doesn't imply that the server will perform no caching. It means that the server will perform validation before releasing a cached copy.

response.headers["Cache-Control"] = "no-cache"

For the server to perform cache validation, we need to provide a validator in our response as well. One choice is to use Last-Modified response header to validate the cached archive file. We use the Time class httpdate method to provide the date and time in the expected format for when the archive was last modified:

response.headers["Last-Modified"] = Time.now.httpdate.to_s

Before we finish headers declaration, we need to deal with the HTTP server buffering problem. Web servers like Nginx perform buffering to reduce overhead with writing and reading streamed content. The problem is that if you stream chunks of content, the Nginx's will store them in a buffer and send it back to the client only when the buffer fills up or the stream closes. Unfortunately, this will make the browser wait for content. To disable this behaviour, we can use the X-Accel-Buffering header to stop the Nginx from buffering:

response.headers["X-Accel-Buffering"] = "no"

Finally, the download action with all the response headers looks like this:

def download
  zipname = "#{@meeting.slug}.zip"
  disposition = ActionDispatch::Http::ContentDisposition.format(disposition: "attachment", filename: zipname)

  response.headers["Content-Disposition"] = disposition
  response.headers["Content-Type"] = "application/zip"
  response.headers.delete("Content-Length")
  response.headers["Cache-Control"] = "no-cache"
  response.headers["Last-Modified"] = Time.now.httpdate.to_s
  response.headers["X-Accel-Buffering"] = "no"
end

Streaming the Zip File

Now, we can turn our attention to actually streaming the zip file content. To do this, we use the ZipTricks::BlockWriter that will be responsible for streaming chunks of the zip archive back to the browser. Each time a writer receives a chunk of content, it will call a block and write the content directly onto the response stream:

def download
  ...
  writer = ZipTricks::BlockWrite.new do |chunk|
    response.stream.write(chunk)
  end
end

Having specified our writer, we're ready to open a stream for writing. We use ZipTricks::Streamer and call the open method with a previously created writer to begin writing the zip archive. As we do so, we ensure that we close the stream when the streaming is done, otherwise the socket could be left open forever:

def download
  ...
  writer = ZipTricks::BlockWrite.new do |chunk|
    response.stream.write(chunk)
  end
  ZipTricks::Streamer.open(writer) do |zip|
  end
ensure
  response.stream.close
end

Next, one by one, we begin to retrieve meeting documents for streaming. We use the write_deflated_file method to create an entry in the zip archive. This method takes the document filename as an argument and yields back the previously created writer IO object that will serve for writing the document content:

def download
  ...
  ZipTricks::Streamer.open(writer) do |zip|
    @meeting.documents.each do |doc|
      zip.write_deflated_file(doc.filename.to_s) do |file_writer|
        ...
      end
    end
  end
ensure
  response.stream.close
end

Thanks to ActiveStorage::Attachment association, we can access document metadata via the blob record. The ActiveStorage::Blob provides a download method which, when called with a block, will stream the file content in chunks. Be careful here though, as calling this method without a block would read the entire file into memory before returning its content - not what we want. Notice, since the file writer is an IO object it responds to the << message that we can use to write our chunks:

def download
  ...
  writer = ZipTricks::BlockWrite.new do |chunk|
    response.stream.write(chunk)
  end
  ZipTricks::Streamer.open(writer) do |zip|
    @meeting.documents.each do |doc|
      zip.write_deflated_file(doc.filename.to_s) do |file_writer|
        doc.blob.download do |chunk|
          file_writer << chunk
        end
      end
    end
  end
ensure
  response.stream.close
end

The Missing Piece

Unfortunately, calling response.stream.write isn't enough to make file streaming work. If you were to try running our code now, it would work but hold the browser from downloading until the full archived file is ready. Each chunk from the response.stream object would be added to the response buffer and sent to the client when the entire response body finishes.

There is one more piece missing from this puzzle - the ActionController::Live module. To make all your actions capable of streaming live data, all you need to do is to mix in this module into your controller:

# app/controllers/zip_streaming_controller.rb

class ZipStreamingController < ApplicationController
  include ActionController::Live
  ...
end

Once the ActionController::Live is included, the response.stream.write will stream data to the client in real-time without buffering. When downloading you will see archive file size growing as in this example:

Under the covers, the streaming is done by executing an action in a child thread. This lets Rails, and specifically Rack process response body in parallel with sending data to the client. Because of this, you need to make sure your action is thread-safe. It also means that a web server needs to be capable of multithreading and streaming. But, Rails default web server Puma has you covered here. The final caveat is that you need to specify response headers before writing data to the response stream.

Summing it all up, the entire zip streaming controller with download action looks like this:

# app/controllers/zip_streaming_controller.rb

class ZipStreamingController < ApplicationController
  include ActionController::Live

  before_action :set_meeting

  def download
    zipname = "#{@meeting.slug}.zip"
    disposition = ActionDispatch::Http::ContentDisposition.format(disposition: "attachment", filename: zipname)

    response.headers["Content-Disposition"] = disposition
    response.headers["Content-Type"] = "application/zip"
    response.headers.delete("Content-Length")
    response.headers["Cache-Control"] = "no-cache"
    response.headers["Last-Modified"] = Time.now.httpdate.to_s
    response.headers["X-Accel-Buffering"] = "no"

    writer = ZipTricks::BlockWrite.new do |chunk| 
      response.stream.write(chunk)
    end
    ZipTricks::Streamer.open(writer) do |zip|
      @meeting.documents.each do |doc|
        zip.write_deflated_file(doc.filename.to_s) do |file_writer|
          doc.blob.download do |chunk|
            file_writer << chunk
          end
        end
      end
    end
  ensure
    response.stream.close
  end

  private

  def set_meeting
    @meeting = Meeting.find(params[:id])
  end
end

Improving the Design

Even though we have a working implementation, there is still room for improvement. Apart from the method being verbose, it’s usually a bad practice to have so much code logic in a single controller action. Let's do something about it.

We will deal with the headers first. As it turns out Rails provides a convenient method send_file_headers! for specifying the Content-Disposition and Content-Type headers. This method will ensure the right format and escaping for the file attachment. This will reduce the header specification to this:

def download
  zipname = "#{@meeting.slug}.zip"
  send_file_headers!(
    type: "application/zip",
    disposition: "attachment",
    filename: zipname
  )
  response.delete_header("Content-Length")
  response.headers["Cache-Control"] = "no-cache"
  response.headers["Last-Modified"] = Time.now.httpdate.to_s
  response.headers["X-Accel-Buffering"] = "no"
  ...
end

However, we can cut down the above response headers setup even further. The ActionController::Live module response.stream.write method deletes the Content-Length and sets the Cache-Control to "no-cache" headers for us, so we can remove them as well:

def download
  zipname = "#{@meeting.slug}.zip"
  send_file_headers!(
    type: "application/zip",
    disposition: "attachment",
    filename: zipname
  )
  response.headers["Last-Modified"] = Time.now.httpdate.to_s
  response.headers["X-Accel-Buffering"] = "no"
  ...
end

Next, we extract the streaming behaviour into a separate class called DocumentsStreamer. In the constructor, it will accept documents collection and allow us to enumerate over all the streamed chunks with each method. Essentially, turning our class into an Enumerable object. As a convenience, we add a class level method stream to abstract the underlying plumbing and provide a verb that expresses the class purpose.

# app/services/documents_streamer.rb

class DocumentsStreamer
  include Enumerable

  def self.stream(documents, &chunks)
    streamer = new(documents)
    streamer.each(&chunks)
  end

  attr_reader :documents

  def initialize(documents)
    @documents = documents
  end

  def each(&chunks)
    writer = ZipTricks::BlockWrite.new(&chunks)

    ZipTricks::Streamer.open(writer) do |zip|
      documents.each do |doc|
        zip.write_deflated_file(doc.filename.to_s) do |file_writer|
          doc.blob.download do |chunk|
            file_writer << chunk
          end
        end
      end
    end
  end
end

Using the DocumentsStreamer, we can reduce our download action code down to this:

def download
  ...
  DocumentsStreamer.stream(@meeting.documents) do |chunk| 
    response.stream.write(chunk)
  end
ensure
  response.stream.close
end

In the end, our refactored action uses Rails to its full potential and tells a more succinct story of how the download works:

def download
  zipname = "#{@meeting.slug}.zip"
  send_file_headers!(
    type: "application/zip",
    disposition: "attachment",
    filename: zipname
  )
  response.headers["Last-Modified"] = Time.now.httpdate.to_s
  response.headers["X-Accel-Buffering"] = "no"

  DocumentsStreamer.stream(@meeting.documents) do |chunk|
    response.stream.write(chunk)
  end
ensure
  response.stream.close
end

Summary

This concludes our overview of streaming large zip files in Rails. We covered a lot of ground by lifting the lid on how Active Storage can facilitate streaming of files. We explored various types of HTTP response headers that instruct clients to download content. All this wouldn't be possible without a great zip_tricks gem and convenient Rails APIs. We finished by cleaning our code up and abstracting away the streaming, thus making the whole thing more maintainable.

I hope this was a useful article that showcased how you can implement any type of download feature and take advantage of Rails streaming API. Feel free to post a comment on social media.

This article was originally published on PiotrMurach.com.

Photo by Fabien Barral on Unsplash

Software Product - Fake It Until You Make It?

Piotr Murach — Fri, 04 Oct 2019 20:22:23 +0000

The "Bad Blood" book by John Carreyrou compelled me to write a review. It also left me with a nagging question of ethics in software development. Is it okay for businesses to sell software that doesn't exist? It seems like an easy option to promote an imaginary product. After all, you can say anything you want.

But why is it so hard to create software products in the first place?

There is no shortage of online articles filled with news about exhilarating product launches. Stories of software product promises that ultimately amounted to nothing. But what are these Loch Ness Monsters of our technology world anyway?

Before we attempt to answer, let's define vapourware first.

What is Vapourware?

It was at Microsoft in the early 1980s that reportedly the word "vapourware" was first used. One of the engineers said it to Ann L. Winblad in relation to a Xenix operating system failure. She later went onto popularise the term "vapourware" comparing it to "selling smoke".

More generally, we can describe vapourware as a software or hardware product that is announced to the world with great enthusiasm. But here's the rub, it either doesn't exist yet or is in the very early stages of development and may never succeed.

We, the public, like a deer in the headlights, get blinded by the headlines and embroiled in the act of promotion. But how do we end up here in the first place?

Getting Hooked

There is a common tactic deployed by companies to grab the public’s attention. During press releases, a company announces a noteworthy product by presenting early low fidelity images, vague presentations or basic prototypes. These predict a bright but vague future in the hope of attracting the interest of journalists and the media. The new product is often thinly veiled by secrecy, partly to hide the truth and partly to generate excitement. Customers are left crumbs of information here and there, and need to imagine how a product might work and hypothesise its possible uses.

The product is usually connected with creating something new, revolutionary in some sense. Only hard to achieve problems create excitement and frenzy among the media. It is so much more thrilling to chase after an elusive unicorn than paint the old horse pink and strap a pair of wings to each side.

Occasionally the vapourware condenses into an actual product. Hurray for these moments! For a long time, a virtual reality product called Magic Leap was seen by many as a classic case of vapourware. Since 2010, the public was promised a revolutionary device that could create a deeply immersive experience, a 3D feast for the eyes. The real unicorn. However, years of waiting lead to articles like this "Magic Leap blows our mind with its incredible technology...that still doesn't...exist".

There are some tell-tale signs of vapourware magic at work. Chief among them, lack of transparency about timelines, evasive and vague answers about product capabilities or lack of any technical specifics. Even after a long wait, when the product eventually materialises it fails to live up to the original hype.

So is there any point to this?

The Merits of Vapourware

The broad-brush mechanics behind vapourware usually work like this. For the software product to gain traction it requires marketing. In turn, this involves telling as many people as possible about the irresistible benefits of the software product you're trying to sell. To achieve this, all the salespeople in the company are mobilised and tasked with hyping and promotion. Marketing campaigns are prepared, target groups are analysed and catchy taglines are conjured. Anything that will raise public interest and expectations. The line between reality and make-believe is rather thin at this stage.

There is nothing inherently wrong or evil with marketing your software product. It is important to start gauging the market before your company potentially invests tons of money and many development hours. The biggest advantage of promoting something that doesn't exist yet is to canvas the feedback from early adopters. It may become clear that what you have in mind doesn't sell as it doesn't solve anyone's problem. This is the reality check your product needs. In case of failure, you can pivot your idea and start again. As a positive side effect, you will release company resources to build something different instead.

An antidote to empty promises is a prototype.

Prototyping Ideas

Prototypes are key to quickly assessing your idea and getting customer feedback. They are an essential tool for communicating ideas within the company and to the outside world. Technology enables us to develop software prototypes fast and inexpensively in any form we desire. Marty Cagan in his book "Inspired" explains techniques and processes for product discovery and delivery.

There is a significant difference between using a prototype to gather feedback and generate sales. Using it to help refine your idea for the next iteration makes sense. You want to save time and build what the market wants. It's not until the point salespeople start spreading embellished stories, that vapour gets created. Don't get me wrong. Marketing people are necessary for launching actual products. But once the wheels of the sales department are set in motion on a phoney solution, it will be hard to backtrack.

The danger is that the marketing phase will inflate product image beyond reality. People will see fancy presentations, great videos, and marketing materials that far outstretch the prototype capabilities. If it turns out in the near future that the product doesn't exist or is still only a prototype, that may tarnish your reputation significantly. You may win a few sales and thus generate revenue in the short term but lose customer loyalty in the long term.

The Development Struggle

It seems that creating a software product is actually quite hard for many companies. They have all the resources they need to create software products. Many programmers, piles of money and copious time. Yet, the innovation doesn't yield the expected results. Throwing money at sales and marketing is much easier. This way you can quickly get new customers and feel like you're growing revenue.

One explanation for why developing products is hard is to do with a lack of experience and expertise in the product territory. Incompetence makes people make bold claims and be bullish about the technology area they don't understand. They conceal their ineptitude with bravado and can-do attitudes. Their lack of awareness about possible issues automatically increases the risk of failure. The possibility of a dire outcome is real.

Innovation in the area outside of your core expertise brings many challenges. Hiring is one of them. It's hard to gauge the talent you need. For example, machine learning and artificial intelligence are hot areas for innovation. But apart from great coding chops, you need people with a deep understanding of other domains. Mathematical modelling, visual recognition or speech processing are few that come to mind. Having and hiring the right people will increase your chances for success. But, there is no guarantee that they will solve your problems quickly. Breakthroughs take time.

Take Pixar for example. In its founding years under Steve Jobs at the helm, Pixar for a long time focused on selling image rendering computers. As reported by David Price in "The Pixar Touch", these computers didn't sell well. Pixar wasn't a hardware company. Their core expertise were the algorithms and software for rendering 3D animated graphics. These were already proven to work well in the classic "Luxo Jr." animation. What is worth pointing out is that Pixar had the most talented graphics programmers in the world. With such employees, they had a strong chance to bring innovation to the world.

Even when a company has a strong vision or an idea it doesn't necessarily mean it has the innovation structure to pull it off. My theory on this is that companies that fail, undertake innovation as if they're building the tower of Babel. There is a grand vision that when completed will overshadow all competition and dominate the whole market. This vision permeates the top of the organisation. Only the highest-earning people engage in the innovation. The vision gets passed down the chain of command. Each step down the ladder, the vision gets more blurry. Communication between departments breeds confusion, and the final product, if it materialises at all, is merely held together with gaffa tape and glue. Like the tower of Babel when built, the product is undesirable and gets destroyed by the Capitalist God of no customers.

Can we do better?

The Path to Solidware

The path to "solidware" that I can see is much simpler and may be viewed by many as naive. It advocates building a real product first that attracts early customers and provides market feedback. This means competing for customers with the product from the beginning. It assumes that you have gone through the ideation process and chosen an idea that has potential customers. Surely you're not attempting to build a product that nobody even wants to have as a hypothetical offering.

You don't have to wait until a complete vision of the product is done either. A version that has the most necessary features and limits its functionality to the essential elements is what you're aiming for. This assumes that you went through many idea generation sessions beforehand and pulled together the strongest contender. Whatever you decide to be the initial product, it needs to work well and demonstrate the features that early adopters will use.

There are plenty of software development practices for converting ideas into products to choose from. Scrum, eXtreme Programming or Kanban will all help deliver a minimum viable product (MVP). If you're starting from zero then Lean Startup methodology is a great approach. Eric Rise explains in the "Lean Startup" book how you can unlock creative ideas, experiment quickly and develop a product people will love. On the technical side, frameworks such as Ruby on Rails are built with the development speed in mind and the premise of quick iteration. You can chop and change your application parts such as database schema easily.

Agreed, it is a slow process compared to spreading humbug. But, this gives you the opportunity to go to market with the actual product. Your solution will have good and bad points and will be subject to scrutiny from the consumers. But that's fine. You can iterate on your product and make it better. Granted, it's much harder to excite people about something that exists. Anyone can spin a good story and pay for more advertisements. Making a quality product that works requires diligence and hard work. But every customer you get will be there because they want your product.

Companies that choose to focus on the quality of their product sooner or later start to win in the market they intended to innovate. It's cliched but the iPhone is a very good example of this approach. The first iteration of the phone was revolutionary but lacking in many ways - buttons could freeze and stop you from answering a call. Only later iterations of the iPhone uplifted Apple to the global mobile market domination sidelining other competitors like Nokia to second-grade players.

Where does this leave us?

Fake-It Culture

Video gaming has a long history of vapourware and it's probably going to stay this way. With a low cost of starting a Kickstarter campaign, it is easy to see developers claiming immersive environments and juicy gameplays without producing anything.

Originally started in Silicon Valley, the fake-it-until-you-make-it culture spreads like a virus to infect different industries. Entertainment, retail, and healthcare are the most affected by innovators claiming improbable breakthroughs.

My parents are from the generation that if you cannot hold it in your hands, smell it, turn it around, look at it, try it out then they won’t even consider a purchase. There is a generational leap that has happened in the recent decade or two where people pay big money for things they've never even touched or seen.

It is inevitable that people will be seduced by convicted con artists selling impossible dreams. Fear of missing out on a great opportunity? Maybe. Feeling of greed intensified by the promise of riches? Possibly. Basic gullibility that dictates action before any due diligence. Certainly!

And as for companies, you can fake your product for a while, but eventually, the lack of a tangible solution will catch up with you. Is it worth tarnishing your brand for cheap money?

This article was originally published on PiotrMurach.com.

Photo by Pascal Meier on Unsplash

Nobody Cares About Your Software

Piotr Murach — Fri, 07 Jun 2019 20:59:47 +0000

Hate to break it to you but the chances are that nobody cares about your software. I'm not trying to depress you. It sounds ruthless but most programmers are busy and don't have time to read your open source project code to understand what it does, how to use it and what it’s meant for.

The majority of developers that stumble across your project are employed to build software for their companies. They do not wish to tear your project to pieces with harsh comments or announce to the world how bad it is. They're merely people on a mission to get stuff done. Whether or not you agree with me, understanding their point of view can help you appreciate the people using your software and elevate your project to another level.

The Creator Mindset

As the creator, the more time you invest in your project, the harder it is to be objective about it. You grow fond of your project and believe that others should too. How you use variables, functions and modules creates envy among your peers. Your engineering skills are unparalleled in the company you work. Your code design achieves a perfect balance of beauty and speed.

So what?

Many software programmers tend to develop with a belief that their artisan software is self-evident to the programming masses. I myself was guilty of this once upon a time. Why bother with a documentation? Humph! Tutorials are for the uninitiated anyway. Anything that takes them away from writing the software is a waste of time. Heaven forbid, promoting their project is a sure sign of selling out to the Capitalist God.

It’s not a very productive attitude to embrace. Why leave all this work afloat in the vast ocean of open source software? If you wrote great software, and nobody uses it, was there any point in writing it in the first place?

The Fresh Perspective

The plight of most open source projects follows a pattern. A developer discovers a software project like a four leaf clover. They find it useful but not exactly matching their needs. So they decide to report an issue asking for the software to suck less and do things better. None of the appreciation and cult-like devotion that you originally expected materialises. Talk about a kick in the ego!

Over the years I learned that no matter how exciting your project may seem to you, all that most programmers care about is whether it solves their problem or not. They want to see an immediate benefit. That’s it. If your software almost fixes their problem then they may contribute an idea for improvement. If you're very lucky, they may even find the time and contribute a patch. This realisation helped me change my attitude, moving from a solution-focused developer towards a more human-focused developer.

So what can you do to increase your odds and make people pay attention?

Explain and Connect

In your own head, the project you are pouring countless hours into all makes sense. But what about the others? If the project is to be useful to anybody else, taking the time to explain its utility in a Readme file seems like a good starting point. The leading question can be: Why would anyone care about this? Here you can explain the source of your idea. Has something frustrated you about the current state of the art? To me, frustration using other (often much more complex) solutions is the most potent reason to create a new project. Equally, some projects exist purely to satisfy my curiosity and are fun to develop. I wrote the lex utility in Ruby to learn how compiler lexing works. Whatever the reason, explain it.

What other stories could you tell? The tty-command Ruby project, for example, has a motivation section which tells a story of a startup founder that had to deal with hundreds of unruly and hard to manage Bash scripts and then rewrote them in Ruby, a more readable, elegant and fun language. A story is the quickest and the strongest communication link between your project and another developer. It differentiates your solution and demonstrates deliberate thought behind it.

Once you explain the background of your project, you can follow it up with a question: What is unique about my software that others should skip reading Hacker News? I often write down a list of features from the developer perspective and in their language, to help them understand the benefits. For example, if they are likely to be familiar with the active record pattern, I may use that as the analogy to explain the ease of using certain API calls.

Help Developers Solve their Problems

The very act of thinking about the answers to previous questions will put you in a new frame of mind and elevate your awareness about your project. The answers will potentially create a snowball effect and lead you further up the enlightenment tree. Once you explain why your project exists and how it differs from any other already present solution, it’s time to consider levels of engagement with the users of your software. Some good questions are:

How does my project explain functionality to a first time user?
How do I approach explaining things to programming beginners versus advanced developers?
How do I help developers to be successful using my project?
How do I demonstrate the perceived utility of my project?
How can I motivate contribution and a higher level of involvement in the project?
What is the vision? What will the project success look like once it is done?

These are example questions that helped and continue to provoke me to write documentation for many of my open source projects. For example, I like to add a content menu that makes it a breeze to find all available API calls. The tty-command project splits the menu between essential and advanced elements of the interface. A first time user can find what they need without feeling overwhelmed with all the features. Similarly, the advanced user can quickly learn about all the extra knobs they can regulate.

Irrespective of their experience level, tell developers how to install your project and show the simplest example of how to get started so that they can quickly bask in the glory of running code. Even better, record a demo. The tty-reader project displays animation of me line-editing text to demonstrate available capabilities. Therefore, the simpler your project is to setup and the more intuitive API that can be quickly understood, the better. I often create an examples folder of common use cases that quickly gives a taste of what the software can do. Of course, I would be lying if I said that I know how to answer these questions perfectly. The well of potential questions is deep and down to you to explore.

But there are questions that are not worth answering.

What Things to Leave out

Explaining why your software is much better by belittling other solutions is a loser’s game. You may protest. But wait - the other projects are so much worse. People surely should know. Not really, and here is why. Nobody seeks to build complex, buggy and unreliable software. A trail of design decisions that seemed great at the time ossifies the project. Against our best intentions, with time and many refactorings later, our software may change beyond recognition. Instead of looking and behaving like a majestic swan gliding on water, it acts more like a raccoon searching through the rubbish bin; it’s working but not very smoothly.

Whatever documentation you write, it’s important not to play hide and seek with people and bury key information under marketing slogans. Programmers are perceptive and you don’t want to discourage them prematurely. Spare them the trouble. Instead, consider giving impartial information backed by benchmarks and examples. Once a programmer satisfies their need, they may consider using your software for other things and let their fellow developers know about it. Word of mouth is a powerful and underrated channel in the programmer circles.

If you want your open source project to be recognised and valued, you will need to work hard at explaining its purpose. Don't count on some famous developer to sprinkle some stardust and bless your project. It has to stand on its own merit. I wrote and released many Ruby packages. Only in recent times, I notice that people start paying attention and use my libraries in their code. Sadly, winning the adulation and the hearts of many programmers is often easier said than done. The truth is, even the best software doesn’t always attract the attention it deserves.

Stick with the Process

The good part of nobody knowing about your project is that it leaves you the liberty to do the right thing. So what could doing the right thing look like? You engage in refactoring your code with a long-term view. You fix all the broken windows. This may mean replacing awkward object inheritance with a chain of simple method calls. You spend time polishing existing features to handle corner cases or invalid inputs. You look at removing features that don't match your goals or clash with the project vision. You improve your test coverage, ensuring tests are fast and comprehensive. Basically, you do all the stuff you should do to have a well-maintained project. The act of maintenance with time creates a good software project. This surprisingly may lead to people starting to care about your software.

This article was originally published on PiotrMurach.com.

Photo by Patrick Robert Doyle on Unsplash

Bad Blood - a Tale of a Modern Vampire

Piotr Murach — Thu, 06 Jun 2019 22:21:55 +0000

I was completely consumed by a recent book "Bad Blood: Secrets and Lies in Silicon Valley" by John Carreyrou, an investigative journalist that uncovers the hidden truths behind one of the biggest health care scams in modern history.

The book touches on some important ethical issues and paints a disturbing picture of innovation in a modern technology company called Theranos. The saga of the company will be familiar to anyone who follows the technology news. What you get from the book, however, is the backstage entrance to all the drama that happened. And trust me on that, there are plenty of events and juicy details to make you gasp with disbelief.

Carreyrou wrote a real page-turner. It reads better than fiction but is grounded solidly in reality. While reading it, I continually checked the main characters on the Internet to see how closely my imagination matched their real personas. Feelings of incredulity, anger, and sadness kept me glued to the pages until the very end when I experienced an Aristotelian catharsis.

From the very first chapter, I was immediately captivated by the character of Elizabeth Holmes, a young drop-out from Stanford University who decided to pursue a grand vision. The vision of a device that could perform a multitude of blood tests for different conditions based only on a small sample from a fingertip.

The premise of the device appealed to me since it had a potential to revolutionise the industry by shortening wait times for various blood tests from weeks to hours and costs from heaps of money to a few hundred dollars. No wonder Elizabeth found willing investors and managed to grow Theranos at a dazzling speed. The company quickly reached a valuation exceeding $9 billion dollars making Elizabeth one of the youngest and wealthiest female entrepreneurs yet.

However, my initial admiration for Elizabeth was quickly dispelled by the author. Carreyrou chapter by chapter reveals more disturbing behaviours from the visionary that make your jaw drop with incredulity. The incredibly high valuation of Theranos starts to feel like Pyrrhic victory. The reader’s discomfort is further intensified by the numerous details of the company’s deceptions. The most important of them - the device itself. The first iteration is just a robotic arm inside a box that was previously used in factories to help mix colours in paints. Hardly the innovation the world awaits!

Theranos bubble burst when the Wall Street Journal published an article written by Carreyrou himself, in which he revealed the naked truth about the company’s bold claims. The article triggered a domino effect of unbelievable events, most of which wouldn’t have looked out of place in an espionage fiction thriller, ultimately leading to Theranos demise. This was by far the most intriguing part. The company shut down in August last year, with Elizabeth and her business partner facing multiple fraud charges for which they will be lucky to escape prison time.

The health industry grows at an amazing pace but we rarely get to know what happens behind the curtains. We blindly believe in health care companies and trust that they have our best interests at heart. However, this book quickly taught me some harsh truths and dampened my enthusiasm for medical innovation.

Theranos performed only a limited set of blood tests, despite claiming the capability to perform many more. But even the tests that could be done produced highly unreliable results due to too small blood sample size. The success rate of correct diagnosis was very low. Carreyrou explains the risk:

A false positive might cause a patient to have an unnecessary medical procedure. But a false negative was worse: a patient with a serious condition that went undiagnosed could die.

There is a fine line between promising a product that will revolutionise, for example, the way it measures your body functions and a device that analyses blood for medical procedures. Wrong results from a device that measures how many steps you take during a day or how long you sleep won’t put you at risk. But a medical device that is used to decide a serious treatment or life-saving operation may do so. Misdiagnosis can have serious consequences and put patients at risk of death.

The author presents Elizabeth as an uncompromising, determined and visionary. A mixture of values that she takes to an extreme. She mimics her clothing and leadership style after Steve Jobs and demands cult-like devotion from her employees. Carreyrou writes:

Like her idol Steve Jobs, she emitted a reality distortion field that forced people to momentarily suspend disbelief.

It became painfully clear to me that more than changing the world and saving lives, Elizabeth craved the limelight and media attention. She appeared on the covers of Fortune, New Yorker and Forbes magazines wearing a black turtleneck and holding ‘nanotainer’, a pill-sized container with a sample of blood inside. She also gave presentations such as a TED talk during which she tells a story of her uncle that died of cancer and how it impacted her as a human to give creed to her vision. The actual human bond between them never existed and the story was fabricated.

All this made me wonder what it would be like to work in Theranos and experience all the deceptions and manipulations. How would I feel about Elizabeth as a charismatic but self-serving leader who would use anything and anybody to reach her goal? Would I have the stamina to stand up or just coil in my chair and work obediently?

Whether or not Elizabeth is a classic case of a psychopath at work, I don’t know. But what I know for sure is that I wouldn’t want to cross her path.

Given the size of pharmaceutical and biotech industries, it is inevitable that we’re going to see products invented by con artists and Ponzi schemers. They will stop at nothing and provide bold statements about the future. Health industry seems especially ripe for the ‘innovation’. Everyone wishes to live longer and look younger. These are strong desires that can be easily exploited. Only due-diligence and familiarity with the subject matter can safeguard against malevolent entrepreneurs.

“Bad Blood” is an electrifying read about technological innovation, full of cliffhangers at the end of each chapter. It is also a powerful study of human psychology. The book throws all characters into a pressure cooker of unbelievable events. Learning about Elizabeth, a modern-day vampire that tries to suck blood from investors and feeds on the loved ones will make you question human nature, the notion of leadership and ponder ethical issues involved in running a technology company.

This article was originally published on PiotrMurach.com.

Photo by thought-catalog-505378 on Unsplash

The Path to Learning a Programming Language

Piotr Murach — Thu, 07 Mar 2019 22:42:42 +0000

"I've learnt Ruby in a weekend/ It's a simple language" boasted one developer who had had no prior experience of the Ruby programming language.

This particular developer had many years of C# programming experience and could probably draw a few parallels between the languages. However, their statement prompted three questions for me.

Can you spend 48 hours and be effective in a programming language?
When can you claim to be proficient in a programming language?
What does it mean to 'know' a programming language?

Research on learning patterns discussed in this Developer Learning Curve article shows that the learning to code path ends up being a curve with a predictable shape - a steep initial incline, a steady rise, and a plateau along the top.

    |                      xxxxxxxxxxxxx
  E |               xxxxxxx 
  X |           xxxx
  P |        xxx
  E |       x
  R |     xx
  T |    x
  I |   x
  S |  x
  E | x
    |x
    +----------------------------------->
             E X P E R I E N C E

The learning curve, charted between two axes of an unknowable scale, is not a diagonal line and it mirrors my own experience. I have programmed in many languages including Delphi, C, Java, and Ruby, to name a few and the learning curve was always steep and bumpy at the start and tapered off as my experience grew.

So what do you need to do to quickly climb to the top?

There are some essential steps that everybody learning a language needs to take such as studying syntax, semantics and error handling. But is this enough or is there more to learn?

As you gain an understanding of the more basic ideas, you need to start to adapt to a language’s paradigm, its idiomatic usage, and then begin exploring all the available standard libraries. Only then can you start to appreciate the unique advantages and capabilities of a language over others. However, this is still not the end of your journey.

Languages are not created in a vacuum. They are created by people for the people. The language tooling ecosystem and community provide the resources for you to be the most effective in a language but these take time to master.

All together, learning a programming language is a complex process.

Let’s break down each step of this process.

The Beginning of The Ascent

One thing is certain, a new language requires learning the syntax. This is a rudimentary but essential step. Every programming language has a set of rules that describe which combinations of symbols can be used to create a valid program. If you have an invalid syntax, the compiler will scream at you by spewing a stack of offending lines of code and your program will die in agony. So without knowing the language valid statements and constructs, you won't go very far.

After you've learned the syntax, the next concept to wrap your head around is semantics. Do you know what the code you wrote actually means? For example, let’s take a look at a classic first program in Java that many books throw at a newcomer to the language:

public class HelloWorld {
  public static void main(String[] args) {
    System.out.println("Hello, World");
  }
}

These few lines pack a powerful punch. I've typed the public static void main line thousands of times, without fully understanding what I was doing initially and then I slowly learned about the meaning of each word. You don’t have to understand every line of this program to understand what it does. But with time, you internalise all the language’s magical incantations and start to see the purpose of all the statements.

On a superficial level, if you know the basic data structures like an array or a list, flow control expressions such as an if or case statement, and understand the concept of a function, you can accomplish a lot in many different languages. However, this may prove to be more of a hindrance than a help. For example, how an array is viewed and used really depends on the paradigm the language uses.

At this point, having studied the syntax and semantics, the language starts to feel familiar. But, of course, in the real world, programs have bugs.

When Things Turn Sour

A lot of developer time is spent debugging software and recovering from errors. What is an error anyway? Generally, an error can occur under these conditions:

Syntax: a wrong construct used.
Type checking: the types don’t match.
Evaluation (as Runtime): program runs but produces a wrong result.

The syntax and type checking errors can be verified by the interpreter or compiler. The syntax errors are usually relatively easy to fix. Depending on the language the type system may be complex and resolving type errors may prove hard. However, modern languages such as Elm provide a powerful help system to resolve even the thorniest type errors.

When the compiler is helpful enough to instruct you on how to fix syntax and type errors, the bane of existence for most programmers is runtime errors - when the code does something the programmer didn't intend it to do. Such unpredictable errors may be caused by network timeouts, out of memory issues but most often they are the result of logical errors, where a programmer’s mental model of a problem fails to match actual code behaviour. These are the worst errors! It’s embarrassing to admit how many times I used the assignment operator when what I needed was a comparison operator. One extra character made all the difference!

Hand in hand with errors goes a programming language’s safety. What does it mean to be a safe language? The Rust programming language provides a succinct definition which means safe from undefined behaviour. Safe language will prevent the most common mistakes and significantly reduce chances of runtime errors during your program's lifetime.

For example, Rust improves the situation over C programming language by automatically handling memory allocation and deallocation helping inexperienced programmers avoid memory leaks. On the other hand, in Ruby you don’t even have to worry about memory, it takes care of memory allocation for you, but it does it at the cost of less safety. The most infamous error is the NoMethodError which is raised when you try to call a method on nil value.

# => NoMethodError: undefined method `client' for nil:NilClass

These type of errors are common in Ruby programs and can be extremely difficult to fix. The developer’s skill in creating robust software, the strict testing habits and the experience are the only way to prevent and fix such generic errors.

All the fundamental constructs, errors and safety are the direct result of the underlying ideas and concepts that lead to a programming language creation.

What the Language Wants

There are myriad ways to think about a programming language based on the features it supports. However, I will focus on two major and most common ways for classifying modern programming languages.

Based on how information is accessed and transformed, a language can support Object Oriented or Functional paradigm. The other way to look at the language can be its type system, whether it is a Static or Dynamic. Here are a few modern languages that best fit each of the paradigm and type system combinations.

	Object Oriented	Functional
Static	Java, C++	Haskell, ML
Dynamic	Ruby, Python	Racket, Elixir

It is important to note that languages often don’t belong strictly to just one category. The Ruby language, though classified as Object Oriented due to everything in it being an object, includes many features that enable the Functional style.

Each paradigm creates a unique view of the world and encourages a specific approach to programming. Understanding this approach is essential to fully benefiting from the language design. For example, Java and Ruby are fundamentally different in how they handle interfaces. Ruby doesn't provide any compile-time safety for its interfaces. However, in Java, if an object implements an interface it has to fulfill the contract by implementing all of its methods.

To make it more concrete, let's look at Comparable interface in Java. In our example, we make the Car class adhere to Comparable contract which gives the class the ability to create objects that can be compared with one another based on the productionYear attribute.

public class Car implements Comparable<Car> {
  public String name;
  public int productionYear;  

  public Car(String name, Integer productionYear) {
    this.name = name;
    this.productionYear = productionYear;
  }

  @Override
  public int compareTo(final car Car) {
    return Integer.compare(this.productionYear, car.productionYear) 
  }
}

The Java implementation will force you to implement the compareTo method and ensure that only Car type can be compared to. The Ruby implementation is very similar bar the syntax. A Comparable module is included that will automatically trigger a comparison call <=> when needed:

class Car
  include Comparable

  attr_reader :name, :production_year

  def initialize(name, production_year)
    @name = name
    @production_year = production_year
  end

  def <=>(car)
    self.production_year <=> car.production_year
  end
end

Due to Ruby's duck type system, the above implementation will allow Car instance to be compared with any object as long as it implements the production_year method. Ruby doesn't even enforce implementation of <=> method! If absent, it will very happily use one in Object class that is pretty much useless in our scenario as it compares the object's identity.

But what about the style of programming?

More Than One Way to Code

As you climb up the programming language learning curve, you start to pick up the idioms of the language. The idioms allow you to express the intention behind your code to fellow language programmers in the most intuitive way.

Everyone who has coded for some time in more than one language is probably familiar with a phenomenon where a newcomer to a language writes code that is correct but just doesn't feel right to people experienced in that language and often reminds you of another language.

In Ruby, if you want to iterate over a collection of 10 numbers you can use a for loop that is common in many other languages.

for num in 0..10 do
  print num
end

You can also use a while loop to perform a similar computation.

num = 0
while num < 10 do
  print num
  num += 1
end

However, a more natural fit would be to use the times method.

10.times do |num|
  print num
end

In general, you will be hard pressed to find Ruby code that uses procedural for loop. It is more common to send each message to a collection and return computation back to a block that will evaluate the expressions included in it.

array.each do |el|
  print el
end

The need to understand how to use the language's concepts intuitively will often require you to explore the whole spectrum of what a modern language can offer.

The Universe Is Much Bigger Than Expected

Most programming languages are packaged and distributed with a standard library. Standard packages often provide support for networking operations, files processing or building graphical user interfaces. The Java language standard library even provides many alternatives to the core data structures to facilitate, for example, concurrent programming. So to become adept at writing concurrent code in Java you need to familiarise yourself with the java.util.concurrent package and its numerous thread-safe collections.

Knowing the fundamental components that come with the language is an important prerequisite to fully benefiting from the language. Failing to do so, you’re prone to invent solutions to solved problems. Familiarity with the standard packages is often not enough though. You will frequently need to install additional libraries and tools to be effective in a new programming language. Here are a few questions you may need to consider:

When you have an error, do you know how to effectively debug it?
Does the language come with REPL(Read and Eval Print Loop) and do you know how to use it?
Do you know how to set up automatic code formatting, linting, and testing?

The Ruby community created Bundler to automate management of project dependencies. Understanding how Bundler works is the key to knowing how to develop new libraries and work in large projects, for example, web applications written in Ruby on Rails framework. The knowledge of such tools is a must-have.

There are also IDEs(Integrated Development Environments) that can hugely enhance programmers' productivity with powerful shortcuts, code autocompletion and interactive debugging. One such editor is the Visual Studio Code that integrates easily via extensions with many modern languages.

Learning about the libraries will bring you a lot closer to knowing the language. But everyone comes on this learning path with a different bag of experiences.

Does the Past Matter?

Learning a new language has a lot do with what you are already familiar with. If you spent many years, for example, writing imperative code and suddenly had to switch to a new object-oriented paradigm, it may feel like crossing a chasm.

I remember when travelling in Germany, I met a developer who had spent nearly 20 years programming in Fortran, an imperative language, to suddenly find himself unemployed and in need of a new job. He needed to quickly pick up Java and learn many object-oriented concepts which took a long time to master due to many years of different habits and conceptual models.

It is much easier to move to a language that displays many similarities with what you already know. When I started learning regular expressions in Ruby, I found that they exactly matched my prior experience coding in Perl. Similarly, I could see many parallels between Ruby and Python approach to using dynamic type system.

Your past circumstances will differ and so will the motivation.

What Prompts You to Learn?

Why are you actually learning a new programming language? It can be because your boss asked you to do a job in an unfamiliar language and you have to quickly execute on a project. You’re just a person on a mission to get the job done.

Perhaps you like programming and have a growth mindset. The classic book The Pragmatic Programmer by Andy Hunt and Dave Thomas encourages readers to study a new programming language every year to open themselves up to new ideas and approaches. Following this advice will surely expose you to different views on the art of programming.

There is no escaping the fact that motivation in learning a language plays a big role in how well you’re going to master it. Is this a long-term or short-term investment? If you wish to be using the language for a long time then it benefits to learn it deeply.

But languages are not created equal in the minds of programmers.

Is It Hard or Is It Easy?

Urban legends, mythical tales, and crude jokes give rise to all sorts of perceptions about the programming languages. These perceptions often colour your attitude towards the language and the speed with which you learn it.

If you see syntax, naming conventions or paradigm that match your own experience, you're more likely to see the language as familiar and what follows easy to learn. At the extreme, a programmer may skip learning a language because of its label as highly difficult to master. Or on the contrary, studying a hard language may help convince others of your intellectual prowess and gain adulation of your peers after they hear that you have finally mastered the impossible!

Over the years listening to the stories from ‘experienced’ programmers and reading blog articles, I came to visualise a simplified scale of a programming languages difficulty. The scale starts with JavaScript viewed as ‘easy’ to learn and ends at Haskell whose scary sounding monads and applicative functors contribute to its perception as ‘hard’.

EASY  |----------x----------x----------x-----------x-----------| HARD
  JavaScript    PHP        Lisp       Ruby      C++/Java     Haskell

Any programming language has enough dark corners to trip a newcomer and can strike horror in the hearts of many 'experienced' programmers. For example, the JavaScript's overall size is small compared to many other languages and the lack of standard library creates a perception of an easy language to learn. However, the language is riddled with strange and unexpected behaviours. It takes time to master the good parts and stay clear of bad ones.

Dealing with perceptions is not easy, they are the byproduct of the programming communities.

Who Are My People?

A programming language has a culture that influences people within its ranks. Almost cult-like social constructs, where all ‘believers’ see the language as the true one. We as humans are social animals and like to feel that we belong somewhere, and if the language community meshes with our ideas and makes us feel accepted then it is much easier to sustain motivation and learn.

Though not often thought as an important ingredient of learning a new programming language, the culture has a significant impact on how quickly you will progress and find enjoyment in a new language. Finding mentors is key in order to be effective in any programming language, especially if you don’t want to spend decades rediscovering all the challenges anew.

Access to community and experienced programmers can foster the learning process. User groups, conferences and open source communities play a vital role in onboarding new programmers. Not only can you directly ask questions about the advantages and disadvantage of the language you're hoping to learn but also what is the best path to take to become proficient.

And this brings me back to the original question.

How Long Do I Need to Learn a Language?

So yes, you can probably learn the syntax of a language in 48 hours. However, I hope that I have convinced you that it takes a lot more than the syntax to learn a programming language. Among many things, you have to get good at idiomatic usage, understand the language paradigm, know built-in libraries and get familiar with the tooling and the language ecosystem. Your past experience has a large say in how fast will you go and so does the community of the like-minded programmers.

You didn't learn your language of choice overnight. It takes time. But at some point, things just click! You start to feel and think about problems in that language. You learn how to effectively use your IDE and the tools. The language becomes the extension of you that translates ideas into working software.

I would go as far as to say that programming in a language is another view of the world. The author of the language wants you to approach software creation using language features and paradigms. Spending time in understanding a language ‘philosophy’ leads to proficiency.

You can learn a language by attending a class, watching a video, or reading a book. If the community resources are ample, it makes the process so much easier. Granted community members and mentors will show you the way, but nothing matches the coding experience you gain working on actual projects that someone needs. This is where you really learn! Alone at your keyboard bashing your head against problems! A mentor can instruct you but they cannot implant the concepts in your head. This can only be done by yourself over many tries and failures.

There will never be a level when you are done learning a language. You may feel you know it but there is always something new to learn. The language itself goes through evolution and requires constant attention — practices change and tools get updated or replaced.

It seems natural to want to say how long something will take. Knowing this we can plan ahead. Of course, nobody can even on a superficial level expect to learn a new language in a short period of 48 hours. Learning a language is very context and environment dependent. This may take six months, one year, or a decade according to Peter Norvig’s Teach Yourself Programming in Ten Years.

Back to our developer and his exclamation. I think it is fair to say that he hasn't learnt the language over the weekend! All he did is translate his prior experience and understanding on a very superficial level to a new language. He doesn't yet know how to design software in a new language or anything about its idiomatic usage, standard library, the available tooling and ecosystem. He is bound to have a hard time. As I recall he did struggle with understanding the error messages and debugging his code.

Alexander Pope expressed this article sentiment aptly: “A little learning is a dangerous thing”.

Next time when you hear similar statements of someone learning a new programming language in a weekend, please be kind to that person, and if you want to help, you can always send them this article.

With thanks to Felienne Hermans, Nadia Odunayo and Thorsten Ball for their comments, reviews and contributions.

This article was originally published on PiotrMurach.com.

Photo by Ilija Boshkov on Unsplash