Emulating Xmonad in JavaScript - Part 2

#xmonad #javascript #developerexperience

If you squint at the above GIF, you can see that I've made quite a bit of progress since my last post - a great introduction, pls read for context.

Shmonad.js can now spawn multiple child processes and handle their output, buffering it into an appropriately-sized viewport. For now, formatting control sequences are stripped out - these would normally produce a vast array of pretty colours. There are also many, many control sequences that are not yet handled, and I'm still figuring out which ones to prioritise.

My ultimate goal would be to handle enough control sequences to be able to run, say, vim. At the moment, the best it can do is to handle top, which is hugely satisfying to watch, but not very useful on it's own.

bash seems to run pretty well, with auto-complete and reverse-i-search kind of working, but they seem to rely on some control sequences that aren't yet supported.

All in good time!

So how have I made this progress? Well, I took a another deep dive after my last dig into terminals and control sequences. This time, I learned all about pseudo terminals.

Let's take a look!

Pseudo Terminals

Having achieved most of the functionality of Xmonad itself in my previous efforts, I switched focus to the other important part of this project - spawning processes.

In itself, this is not a complicated task. There are well-defined APIs provided by the OS for this and Node.js exposes them in a fairly straight-forward way.

However, programs and their creators have made things complicated.

I'd like to start this journey by looking at the well-known less command. Commonly used for manually scanning and jumping through a file or program output, the incredibly useful and powerful less has been around since 1984 (according to man less).

Piping hot output

When I ran less package.json in my project directory in an earlier version of shmonad, an unexpected thing happened. less appeared to output the entire file from stdout. The whole point of less is that it only outputs a portion of the file to the terminal! So what gives?

I began to look for clues that might indicate what was going on here. A useful tool for debugging the output of a program designed to run in a terminal is cat -v. Piping program output into cat -v will display all control sequences escaped so that they can be displayed in a terminal without being interpreted.

For example, running nvim | cat -v produces the control sequences and text that Neovim uses to render its UI. If you try this yourself, you'll notice the characteristic "[No Name]" somewhere in the mess of "["s. When nvim is running in a terminal, the text "[No Name]" is inserted at a position determined by the current cursor position of the terminal. This in turn depends on the control sequences (and characters) that were previously sent to the terminal.

So you might think - as did I - that running less package.json | cat -v would produce a similar output. However, this is not the case.

The more seasoned POSIX-based developer would be familiar with the fact that piping output from less into another command basically does nothing (without specifying any options to less).

The key difference between Neovim and less in this case is their behaviour when running in a terminal as opposed to streaming output to something that is not attached to a TTY (a legacy acronym that used to mean one of these things):

credit: http://www.classiccmp.org/

Anyway. Vintage terminals aside.

When faced with this difference in behaviour, my first thought was:

There must be a control sequence that tells less that it's running in a terminal.

And I was wrong, again.

Going back to Node.js, I started wondering about the API for accessing stdout and stdin. Part of this API indicates whether stdout/stdin is attached to a terminal, called .isTTY.

This flag is always true for a process that is started within a terminal - useful for when it makes sense to behave differently in a terminal.

Having realised that Node.js provides an interface for such a thing, I naturally assumed that there would likewise be an interface provided by Node.js that allows the program to spawn a process in a way that makes it think it was spawned within a terminal. But these two concepts are very much disassociated: one is an API provided to the current program, another requires creation of processes attached to stdin/stdout streams that are in turn (kind of) attached to a TTY.

Initially, I confused this for "raw" mode of terminals, which is a whole other thing. If you're interested in the 2.5h I spent digging deep into the depths of all this, see this twitter thread. As part of my deep dive, I came across a term "Pseudo Terminals" and decided to take a closer look at these strange creatures. https://en.wikipedia.org/wiki/Pseudoterminal

In some operating systems, including Unix, a pseudoterminal, pseudotty, or PTY is a pair of pseudo-devices, one of which, the slave, emulates a hardware text terminal device, the other of which, the master, provides the means by which a terminal emulator process controls the slave.

At this point, I was like "why didn't search for that before??"

A quick search in my fave search engine provided exactly what I was looking for: a set of Node.js bindings for "Getting certain programs to think you're a terminal, such as when you need a program to send you control sequences - node-pty.

node-pty is an Open Source library for spawning processes in the context of a pseudo terminal. It also supports a lot of different operating systems, which is pretty cool! It also happens to power VSCode, Hyper and a bunch of other really cool terminal emulators.

Having struck this gold mine (yay for Open Source <3), I proceeded to add node-pty to shmonad and with very little modification; it worked like a charm!

The next steps will be to continue to add support for more and more control sequences. The results should be quite amazing, especially when I get around to implementing formatting (which would warrant it's own blog post!)

Prior to supporting all the control sequences in the world, shmonad needs a bit of a refactor.

For now, I'll leave you with more TTYs.

vc303: credit http://www.classiccmp.org