The Unix way... or why you actually want to use Vim

I like all the Vim tutorials on Dev.to. They're great! They're good starting points. Please go and read them all!

But...

I'd like to tell you all the secret teaching of Vim. Bit of a sidetrack. You know, the stuff you get in the 36th chamber of Shaolin or whatever, because for some reason we love mixing up all our software development metaphors with martial arts and/or Zen Buddhism.¹

Let me tell you about the Unix way...

Unix Philosophy

No, I'm not going to explain what Unix is. You have Google for that. Here's a link to the Wikipedia page on 'Unix philosophy', which is pretty long so let's get to the point, eh?

In the words of the Ancient Grand Master Douglas McIlroy:

Write programs that do one thing and do it well.

Which is easy to say but not easy to do. What if you want to do more than one thing?

Pipes

Pipes are the jewel of a Unix system. To me, its crowning glory. You send the output of one program as the input to another program - just like putting bits of hose pipe together and passing a stream of water through.

Say I wanted a random word with 'bob' in it somewhere. On a Unix like machine you will find a file with a lot of words in it - a dictionary. Mac and BSD systems have it at /usr/share/dict/words.

We can use the program cat to show it in a terminal:²

cat /usr/share/dict/words

But now we need to find all the words with 'bob' in ... well, we could open the file with grep

grep bob /usr/share/dict/words

but I find it more intuitive to pass the output of cat into grep using a pipe, which we write with the pipe symbol: |³

cat /usr/share/dict/words | grep bob

Well that gives me all the words with 'bob' in - how will I get a random one? There are a few ways of doing this, but here's one: I have the program shuf on my computer, which shuffles the lines you send in:

cat /usr/share/dict/words | grep bob | shuf

and now I just want one of the lines - it doesn't matter which one. Why not just the first one then? head will do - it can give us just the first line from the input:

cat /usr/share/dict/words | grep bob | shuf | head -1

and behold

> cat /usr/share/dict/words | grep bob | shuf | head -1
bobolink
> cat /usr/share/dict/words | grep bob | shuf | head -1
nabobish

What does that mean? I can use some other programs to help - I could open (at least on OSX) a URL with that word in it, using a handy program called parallel:

cat /usr/share/dict/words | grep bob | shuf | head -1 | parallel open https://en.wiktionary.org/w/index.php?title={}

and then... and then... and then...

The important lesson is this: we didn't write a new program. We didn't download a new random bob word generator. What we did was use a series of programs that do one thing (and one thing well) and glued them all together to get what we wanted.

But you promised us Vim!

Yes, yes. Vim. I've used Vim since I started developing - I used to write marketing emails using it to make writing marketing emails just a little bit more interesting (and therefore bearable). We always bang on about the keybindings in Vim - the magic of hjkl - but you can add those to almost any text editor these days.

So what do I think is great about Vim?

This:

:%!

For the Vim novitiates among us: : indicates we're entering a command. % is a reference to all the lines in the current buffer. And ! indicates we're going to run a command in the shell.

The above will pipe your current buffer through a shell command and replace the contents. Try it!

:%! shuf

Well that was fun, if highly destructive. But it can be a lot more useful. Want to reformat all the JSON in your buffer? jq is your friend:

:%! jq '.'

or maybe you'd like Python to do the same thing:

:%! python -m json.tool

view some Markdown as HTML? Try pandoc:

:%! pandoc 

You can do it with a range of lines, or a visual selection too - as always with Vim, help is available:

:h !

Now you can leverage other people's programs to extend your editor without installing extra modifications, or plugins, or using a new editor as you would with things like VSCode, Atom or Sublime. And if you find something useful, and use it over and over again, you can write a bit of VimL and to make the command easier to run.

Another advantage - by learning some of the simple, do-one-thing oriented programs that are available in a Unix environment, you'll be able to use them over and over again in more and more programs that you write.

Over to you

This becomes very real when you hit a problem that you've not got a tool for. For me, it was when I needed to translate some large JavaScript objects into JSON. A problem I couldn't see an easy way to complete in the editor⁴.

So I wrote a short(ish) NodeJS script that takes a stream of JavaScript and outputs a stream of JSON. Not beautiful, but effective.

Having written this once, with a pipe-friendly interface, I could now do this to a JavaScript object in a Vim buffer:

:%! js2json

or in the shell:

cat some.js | js2json | curl -X POST -d @- http://gimme-json.com

or maybe I'll never use it again.

But it's there. I've extended my editor by leveraging its ability to integrate with the programs I write. Thanks Vim! Thanks Unix!

Vi-Nature is Unix-Nature

Vim is not great because it impresses your friends with your knowledge of esoteric keystrokes,⁵ or because your elite touch typing skills allow you to churn out x-ity-x-billion characters per second by never taking your hands off the keyboard. These are fun things but they're shallow things. We should not obsess over them, and we should not be smug about knowing them.

Vim is great because it's everywhere - if you know Vim/vi then you've got an editor on most Unix systems you can use. It's great because you can integrate it into Unix tooling - and that's great because you can write those tools yourself because they're not that complicated and you're a programmer!

You'll see the same idea in other editors too. In the words of Master Wq:

Vim is not permanent. nvi is not permanent. vi itself is not permanent, only vi-nature. Emacs has vi-nature, nano has vi-nature, even Notepad has vi-nature. You narrow your sights, you grow attached [...]. You must leave. Come back when you have mastered Emacs.⁶

Where you can do the same with the simple Emacs command:

C-x h C-u M-| js2json

Happy hacking!

---

1. It's all ninjas and koans, or its woodworking and 'craftmanship'. ↩

2. Strictly speaking this is an abuse of cat. ↩

3. The idea was McIlroy, but the | symbol was all Ken Thompson. Imagine that first day of using it - pipemania! ↩

4. A hard way to solve it would have been to write a macro. That would have been throwaway and probably as time consuming.  ↩

5. You must all try ggg?G right now. ↩

6. Only two editors missing, and you should try them both: ed and acme. ↩

Comments

[Thomas H Jones II]

cat /usr/share/dict/words | grep bob | shuf | head -1 | \
  parallel open https://en.wiktionary.org/w/index.php?title={}

The (notionally) more portable — and less abusive of felines — method would be something like:

grep bob /usr/share/dict/words | shuf -n 1 | \
  xargs -I {} curl https://en.wiktionary.org/w/index.php?title={}

Thankfully, shuf lets you limit the amount of lines output ...meaning you can save yourself the scads of resources of calling head! =)

Or, if you want to avoid helpers like xargs or parallel altogether:

curl "https://en.wiktionary.org/w/index.php?title=$( grep bob /usr/share/dict/words | shuf -n 1 )" 

If you're going to abuse a cat, do so by trying out multiple skinning-methods. =)

[David Wickes]

All very true, and very good!

If you're going to abuse a cat, do so by trying out multiple skinning-methods.

🤣

[Sean Kelleher]

What I tend to use more often than :%! is to actually highlight a number of lines using V (visual line) and then apply an operation by simply typing : and then ! (vim automatically inserts the range specifier '<,'> for you), followed by your command. I typically do this with sort to arrange import sections in particular.

[David Wickes]

Yup - me too. And a quick look at :h ! will show you how to pass through motions and line ranges to an external program.

The possibilities are ... well, not endless. But there are a lot of them.

If you enjoy doing this sort of thing, you should really take Acme out for a spin.

[Ben Sinclair]

Saying "the possibilities are finite!" is more accurate but less fun.

[David Wickes]

Another thought: although find implementations can be bloated with extra functionality, they always play nicely with pipes. That's the one rule that nobody wants to break, because it's so damn useful!

[DrBearhands]

I think you might be looking for the second part of the Unix philosophy:

Write programs to work together

[David Wickes]

Composition is important... but reading

Raunak Ramakrishnan

Nov 10 '18

I always google before using find. But it is a powerful tool. For simpler tasks, I just use ls -1 GLOB or ls -1 | grep PAT.

One task where find really helped was finding and removing broken symlinks in a directory with over 10 million files.

I start to wonder whether the whole 'do one thing' idea is good because it's easier to remember the 'one thing' program's simple and focused interface, and so compose it with other 'one thing' programs, than it is to remember the large and complicated interface of things like find.

Because I also have to look up how to use find every time I have to use it - but the piping of ls to grep is intuitive.

In other words, in order to get all the benefits of composition, don't you need the simplicity of 'do one thing'?

[Raunak Ramakrishnan]

I always google before using find. But it is a powerful tool. For simpler tasks, I just use ls -1 GLOB or ls -1 | grep PAT.

One task where find really helped was finding and removing broken symlinks in a directory with over 10 million files.

[Jacek Złydach]

Great article, showing an entire extra dimension for capabilities of a good editor.

RE the Emacs example, let me explain for posterity. This may look scary:

C-x h C-u M-| js2json

But it's actually built out of commands Emacs regulars would know well.

• C-x h means "select all" (more accurately, the mark-whole-buffer command).
• C-u is a "prefix argument", applying to the next command you issue; it's something used frequently.
• M-| is shell-command-on-region; it'll take what you selected and pipe it into a shell command. Without a prefix argument, the output would end up in another Emacs buffer; with a prefix argument, it replaces the selection.

(And, of course, C-x is "control + x", M-| is "alt + |" in Emacs-speak.)

Cheers!

[David Wickes]

Many Unix programs have deviated from the 'Unix philosophy', the best example being cat which is rarely if ever used to concatenate files any more.

Take a look at cat -v considered harmful where Rob Pike writes very well about this.

[Nurbol Alpysbayev]

Finally someone

[David Wickes]

What do you mean by this?

[Nurbol Alpysbayev]

Unix Philosophy is very similar to Functional Programming philosophy. They overlap in many ways e.g. small programs vs small functions, do one thing vs pure (no side-effect) functions etc. However sometimes overall productivity both short and long term as well as efficiency is better if we not primitively and blindingly follow the utopic idea (Unix-way or FP). Example: vim follows Unix-philosophy and VSCode (or WebStorm etc.) breaks it. The former while it is a cool concept, generally is inefficient because it has a very, very steep learning curve. Regarding FP: Haskell and friends are generally less efficient than JS because you have to become a f***ing (sorry, I lack vocabulary to express myself) mathematician to start gain something from it.

While I generally love the Unix (and FP) Philosophy, it should be taken with a grain of salt (i.e. applied to appropriate problems e.g. low-level programs), just like any idea.

[Renan Cidale Assumpcao]

You are talking about a purely functional language. The paradigm itself is way easier to get into, is just a matter of wanting to learn something. Simple as that. Functional programming doesn't need to be understood just by mathematical references.
And then you go and say "I love" ? Really?

[Kalin Chernev]

Thanks for the tutorial! Made me seek whether there's an extension (atom.io/packages/pipe) for Atom which could enable the same way of thinking, and I think I found one which I plan to try soon.

[Jean-Michel 🕵🏻‍♂️ Fayard]

Vim is great because it's everywhere - if you know Vim/vi then you've got an editor on most Unix systems you can use.

Yes, this is like saying that Javascript is the best programming language that runs in the browser. While it's true, it tells us nothing about the quality of Javascript's design (or lack thereof in many aspects).

I would argue against using Vim to preach the Unix philosophy for non-believers. It could backfire, and rightly so.

Vim gets a special pass because it's a fascinating part of computer history. But if it was invented today, nobody would be insane enough to use it.

Really, users should not have to try hard to understand a piece of software. We understand today that it's the software that has to try hard to understand its users.

The Unix philosophy is great though! But it is about having simple core concepts that you can combine ad nauseam. Your paragraphs on pipes is a much better illustration of it!

[David Wickes]

Not a bad point, but I think you're a little mean to call the most popular editor for sysadmins/devops a 'part of computer history'. It's still pretty popular for web developers.

So... I lay down a challenge to you: can you name another editor which has a similar command to call out to the shell and replace the current buffer/file with the contents? i.e. I can extend the editor using external programs.

[Jean-Michel 🕵🏻‍♂️ Fayard]

I have various ways to do that in Jetbrains IntelliJ, like this one
jetbrains.com/help/idea/settings-t...

Reading again my comment, it's true that my tone was a bit mean. I used vim a lot and found it fascinating. I think we have better tools now, but that's not vim's fault, that's progress.