If you squint at the above GIF, you can see that I've made quite a bit of progress since my last post - a great introduction, pls read for context.
Shmonad.js can now spawn multiple child processes and handle their output, buffering it into an appropriately-sized viewport. For now, formatting control sequences are stripped out - these would normally produce a vast array of pretty colours. There are also many, many control sequences that are not yet handled, and I'm still figuring out which ones to prioritise.
My ultimate goal would be to handle enough control sequences to be able to run, say,
vim. At the moment, the best it can do is to handle
top, which is hugely satisfying to watch, but not very useful on it's own.
bash seems to run pretty well, with auto-complete and
reverse-i-search kind of working, but they seem to rely on some control sequences that aren't yet supported.
All in good time!
So how have I made this progress? Well, I took a another deep dive after my last dig into terminals and control sequences. This time, I learned all about pseudo terminals.
Let's take a look!
Having achieved most of the functionality of Xmonad itself in my previous efforts, I switched focus to the other important part of this project - spawning processes.
In itself, this is not a complicated task. There are well-defined APIs provided by the OS for this and Node.js exposes them in a fairly straight-forward way.
However, programs and their creators have made things complicated.
I'd like to start this journey by looking at the well-known
less command. Commonly used for manually scanning and jumping through a file or program output, the incredibly useful and powerful
less has been around since 1984 (according to
When I ran
less package.json in my project directory in an earlier version of shmonad, an unexpected thing happened.
less appeared to output the entire file from
stdout. The whole point of
less is that it only outputs a portion of the file to the terminal! So what gives?
I began to look for clues that might indicate what was going on here. A useful tool for debugging the output of a program designed to run in a terminal is
cat -v. Piping program output into
cat -v will display all control sequences escaped so that they can be displayed in a terminal without being interpreted.
For example, running
nvim | cat -v produces the control sequences and text that Neovim uses to render its UI. If you try this yourself, you'll notice the characteristic "[No Name]" somewhere in the mess of "["s. When
nvim is running in a terminal, the text "[No Name]" is inserted at a position determined by the current cursor position of the terminal. This in turn depends on the control sequences (and characters) that were previously sent to the terminal.
So you might think - as did I - that running
less package.json | cat -v would produce a similar output. However, this is not the case.
The more seasoned POSIX-based developer would be familiar with the fact that piping output from
less into another command basically does nothing (without specifying any options to
The key difference between Neovim and
less in this case is their behaviour when running in a terminal as opposed to streaming output to something that is not attached to a TTY (a legacy acronym that used to mean one of these things):
Anyway. Vintage terminals aside.
When faced with this difference in behaviour, my first thought was:
There must be a control sequence that tells
lessthat it's running in a terminal.
And I was wrong, again.
Going back to Node.js, I started wondering about the API for accessing
stdin. Part of this API indicates whether
stdin is attached to a terminal, called
This flag is always true for a process that is started within a terminal - useful for when it makes sense to behave differently in a terminal.
Having realised that Node.js provides an interface for such a thing, I naturally assumed that there would likewise be an interface provided by Node.js that allows the program to spawn a process in a way that makes it think it was spawned within a terminal. But these two concepts are very much disassociated: one is an API provided to the current program, another requires creation of processes attached to
stdout streams that are in turn (kind of) attached to a TTY.
Initially, I confused this for "raw" mode of terminals, which is a whole other thing. If you're interested in the 2.5h I spent digging deep into the depths of all this, see this twitter thread. As part of my deep dive, I came across a term "Pseudo Terminals" and decided to take a closer look at these strange creatures. https://en.wikipedia.org/wiki/Pseudoterminal
In some operating systems, including Unix, a pseudoterminal, pseudotty, or PTY is a pair of pseudo-devices, one of which, the slave, emulates a hardware text terminal device, the other of which, the master, provides the means by which a terminal emulator process controls the slave.
At this point, I was like "why didn't search for that before??"
A quick search in my fave search engine provided exactly what I was looking for:
a set of Node.js bindings for "Getting certain programs to think you're a terminal, such as when you need a program to send you control sequences -
node-pty is an Open Source library for spawning processes in the context of a pseudo terminal. It also supports a lot of different operating systems, which is pretty cool! It also happens to power VSCode, Hyper and a bunch of other really cool terminal emulators.
Having struck this gold mine (yay for Open Source <3), I proceeded to add
node-pty to shmonad and with very little modification; it worked like a charm!
The next steps will be to continue to add support for more and more control sequences. The results should be quite amazing, especially when I get around to implementing formatting (which would warrant it's own blog post!)
Prior to supporting all the control sequences in the world, shmonad needs a bit of a refactor.
For now, I'll leave you with more TTYs.
vc303: credit http://www.classiccmp.org
vc404: credit http://www.classiccmp.org
vt100*: credit http://www.classiccmp.org
*: the TTY to popularise ANSI control sequences!!
Please feel free to leave a comment and share your thoughts and if you're keen for more posts like this, follow me! :)
See you next time folks, thanks for reading.