Moonicipal Explained

I've recently published a new task runner plugin for Neovim - Moonicipal. Several years ago I've wrote a similar post about Moonicipal's predecessor - Omnipytent. I've opened that post with an explanation why such a plugin is needed. The same reasoning apply for Moonicipal, so I'll not repeat them here.

Why a new plugin?

When Omnipytent was written in 2016, the Neovim ecosystem looked very different that how it looks today. Plugins were not written in Lua - they were written either in Vimscript or in some external language by using Neovim's RPC API, and many plugins still aimed to support both Vim and Neovim. Additionally, Python 2 was still alive and kicking, with many plugins written in it, and Vim had some issues supporting both versions of Python in the same instance.

So I decided to write Omnipytent using Python, in a way that'll support both Python 2 and Python 3. This means that the tasks themselves can be written in either Python 2 or Python 3. Since this was before Lua plugins, most plugin used Vim commands and/or keybinds as their interface, and these only required passing strings - which were pretty easy and straightforward to build in Python (though no quoting conventions was a bit of a pain)

Nowadays, most Neovim plugins are in Lua, and their interface is Lua functions that often accept tables and functions. These are a bit harder to generate from Python - especially the back-and-forth function calls - and it's starting to seem easier to just write everything in Lua.

Another problem was asynchronous tasks. Having to support Python 2 meant I was limited to Python 2's version of async (only using yield - I couldn't even use yield from!), but that was manageable. A bigger problem was GC - having to maintain a references to the Python generators so that Vim code can resume them, and having to clean that reference manually after usage. With Lua coroutines this is simply not an issue - Lua functions can just be stored as Vim functions.

So, even though Lua is less expressive than Python, its superior interoperability with Neovim and Neovim's plugin ecosystem makes the switch worthwhile.

Using Moonicipal

For the sake of demonstration, I'm going to use the same project I've used in my post about Omnipytent - the Java Sprint example project PetClinic.

After cloning the repository and cding into it, we fire up Neovim and:

So what just happened here?

I started by writing the command :MCedit test to edit the Moonicipal task named "test". This opened a Lua file named .idanarye.moonicipal.lua.

Why that name?

Because I've configured Moonicipal as:

require'moonicipal'.setup {
    file_prefix = '.idanarye',
}

But even if I left file_prefix out of the setup, Moonicipal would just take it from my $USER environment variable.

The purpose of the file prefix is to make the tasks file personal. Sharing task files causes all sorts of problems:

• You need to be careful about what you change. You can't, for example, just add a parameter to a command in a task - that will cause problems to other people who use that task.
• You can't just use Neovim plugins and locally installed Lua modules - that will break the tasks file for other developers who don't have them. You'll have to add a dependency management (or at least list the dependencies), which is kind of an overkill for a task runner...
• You won't be able to put personal information in the tasks file. I'm not talking about passwords or tokens - take extra care with those! - but things like paths on your local machine or URLs to servers assigned to you. Other developers will have their own paths and their own URLs, and if you share a tasks file you'll need to put a mechanism to avoid sharing these.
• A shared task file means you are writing code that other people will use, which generally requires higher quality code. Not that high quality code is a bad thing - but it does cost time and effort, and Moonicipal is supposed to allow quick and dirty rapidly changing tasks.

It's better not to commit the tasks file at all, but having a different filename for each developer also helps in ensuring they remain separate.

:MCedit test created a scaffold for the tasks file and for the task:

local moonicipal = require'moonicipal'
local T = moonicipal.tasks_file()

function T:test()

end

A task file is simple. It requires the module and uses its tasks_file() function to create a tasks registrar T. We can then add our tasks as functions on that registrar - like T:test(), which was automatically added because I've added it as an argument to the :MCedit command.

Then I write the body of the task:

vim.cmd'!mvn test' 

vim.cmd is a Neovim Lua API for running Vim commands. The parameter passed to it is !mvn test - which means it'll run the :!mvn test Vim command, which is just running mvn test in the non-interactive shell.

What if we want interactive shell? This requires some more commands, but it's still basic usage of Neovim's Lua scripting interface:

Or, if you're more comfortable with Vim commands, we can just use Lua's multiline quotes:

This simple example already demonstrates an advantage of Moonicipal over CLI task runners - we can control, from the test, on how Neovim will run the command. We can run it in the non-interactive shell, open an interactive shell buffer, use a terminal management plugin - whatever we want! If we can code it in Lua, we can put it in a task.

Getting input

Running all the tests is good for CI, but during development we often want to run a single test that focuses on the changes we are currently making. This takes less time to run and less effort to look at the results.

Maven lets us run a single tests class by setting the test property:

vim.cmd.terminal('mvn test -Dtest=ClinicServiceTests')

The Moonicipal tests file is built to be easy to change, so always editing the task when you want to run a different test is not that bad - but we can also make our task receive input:

moonicipal.input is a function for receiving input. By default it functions like Vim's builtin input.

So why can't we just use vim.fn.input()?

Because Neovim has vim.ui.input(), which can be overridden by plugins to offer nicer input UI.

Okay, wise guy, why can't we just use vim.ui.input() then?

vim.ui.input() works with a callback. You pass a function to it, and when the user enters their text the function gets called with that text. If you need more input after receiving the first input, you need another callback in the vim.ui.input() call that's nested inside the callback of the first vim.ui.input() call. This pattern is called "Callback Hell", and considered not very pleasant.

To avoid this callback hell, many programming languages introduce some form of asynchronous execution. Lua does it with coroutines, and Moonicipal runs all its tasks inside coroutines. moonicipal.input() can only run inside a coroutine, and instead of using a callback, it resumes the coroutine once the input was entered.

Entering the test class name each and every time seems like a step backwards though. moonicipal.input() is not really that useful. But just like Neovim has vim.ui.input() and vim.ui.select() - Moonicipal also has moonicipal.select. We can collect all possible test files inside our tasks, and then just select the test file to run from the list:

Note that just like moonicipal.input(), moonicipal.select() uses vim.ui. I get my nice interactive list from fzf-lua, but there are several other plugins which provide their own UIs.

Another new thing we see here is moonicipal.abort(). If I cancel my selection, I don't want the task to open a new window and then fail formatting a command to run in it and print a traceback while keeping the new window open. Instead I use or moonicipal.abort() so that if moonicipal.select() returns nil, it'll just abort the task without much fuss.

Caching choices

Even if we choose it from a list, it is still a waste to have to choose the test every single time we want to run it - especially during development, when we want to keep running the same test over and over. Luckily, Moonicipal has some caching facilities that let us cache our choice:

Caching in Moonicipal is task-bound, so we need to create a new task. They are simple enough to write by hand, but here I used :MCedit choose_test to scaffold a new task. The task's self has some caching facilities - one of them is self:cached_choice(), which lets us cache a selection from a list. Only the choice is cached - the list itself is generated from scratch every time we use the cache - so we must set a key for Moonicipal to recognize the choice by. In this case we use strings, so a simple tostring suffices.

Then we call the cached choice object as a function with all the cache choices. We already had a loop for generating them, so I just reused that.

Finally, we use the select() method to run the actual selection - either by asking the user or by using the cache.

Back in the original test task, we can access the cached choice by calling its task as a method on the registrar object - T:choose_test().

When I run :MC test, it prompts me to choose a test class - just like before. But when I run it a second time - it remembered my choice of test to run!

Eventually, though, we may want to change our choice. To do that, all we have to do is run the choose_test task directly with :MC choose_test. This will let us choose again, even if there already is a cached choice. Of course, it won't actually run the test until we use :MC test again.

Cached buffers and Channelot terminals

Let's leave the tests, and look at another aspect of development cycle - trying queries against a live server. For this, one of Moonicipal's supplemental plugins - Channelot.

Here we also use a new caching facility - cached_buf_in_tab. This method accepts a function, and expects that function to finish in a different buffer. After the function runs, cached_buf_in_tab will automatically jump back to the window where it started (if its still open in the current tab)

The caching part is that if the buffer from a previous run is open in the current tab - the function will not run, and instead cached_buf_in_tab will return the value from its previous run. This makes it ideal for processes like interactive shells that other tasks can use.

As for the plugin I mentioned, Channelot - I hope its API is clear enough:

local j = channelot.terminal_job('bash')
j:writeln('mvn spring-boot:run &')

This creates a bash job, and writes a text line to its STDIN to run the server in a background process. This seems a bit weird - why not simply use vim.cmd.terminal('mvn sprint-boot:run')?

Because we can use the same bash instance to run queries against the server:

By calling T:launch() - just like T:choose_test() from before - we get the returned value from that task, which is the job handler j returned from the launch job. We can than use that handler to send commands the the running bash process, and see the results live in the terminal.

Also, you may have noticed that instead of :MC launch and :MC start here I've used just :MC without arguments, which prompted me to select the task from a list. :MC <task-name> is useful for keybinding, but if you have a good selection UI :MC is more ergonomic for running tasks you don't have keybindings for.

Data cells and BuffLS

One of the most powerful types of input we can cache is a data cell - a buffer that can be edited with Neovim's full editing capabilities, including plugins and language servers. The buffered is stored in memory even when closed (as long as it's not deleted with something like :bdelete), and

(To save precious screen space (I'm recording these at 24 rows on 80 columns - much smaller than typical modern terminals) I'll abandon the bash terminal split and just use print(vim.fn.system(...)) instead (vim.cmd'!...' has problem with line breaks). We'll lose syntax highlighting for the JSON, but I can live with that)

cached_data_cell's argument is a table. The default parameter should be obvious - default content for the buffer, the first time we run the task to edit it. buf is a function (or a string with Vim commands) for preparing the buffer. It is called every time the task is invoked to edit the data cell.

Note that there is no need to create the split - cached_data_cell will do it automatically. This is because buf is called every time the task is invoked, and we want to reuse the split if it is open. You can change how the split is opened with the win parameter, but that's rarely necessary.

Merely setting the filetype to jq gives us syntax highlighting and everything else we have configured for editing jq query files. But we can do more with another one of Moonicipal's supplemental plugins - BuffLS:

require'buffls.TsLs' is BuffLS' Treesitter based language server. Its for_buffer method creates an instance, attaches it to a buffer (current by default - or you can pass a buffer number), and sets the Treesitter syntax based on the buffer's filetype. In this case - jq (so you need to have that syntax installed for it to work - easiest way is to install nvim-treesitter and run :TSInstall jq)

This gives us a BuffLS language server object we can use to configure BuffLS for that specific buffer. Which means we can use it to add very specialized LSP functionalities.

The simplest one is a code action. Not that LSP code actions are that simple in general - but BuffLS uses null-ls behind the scenes, and because null-ls runs inside the Neovim instance its code actions are simply Lua functions. We can add our code actions with ls:add_action - all it needs is a caption for the action and a function.

Actions that set the buffer content to some template are the simplest - all they have to do is use moonicipal.set_buf_content (which is a wrapper for nvim_buf_set_lines with a simpler signature). But of course they can also read the content, configure things in the buffer (or in Neovim in general - but don't get crazy please), or anything else you can do in a Lua function (which is basically everything). Additionally, they run in a Lua coroutine - which means they can use things like moonicipal.select() to get input from the user.

But... we didn't really need the jq Treesitter syntax for that, did we?

Let's add something that does use Treesitter - completion!

By using a Treesitter query, we can detect that we are inside the string part of .firstName == "", and in that case offer appropriate completions.

Treesitter queries can be a bit too complex for most Moonicipal usecases. Luckily, these usecases are often easy to wrap. One common usecase is bash - authoring commands with the specific flags our project uses.

BuffLS supports this with buffls.ForBash:

Conclusion

These tasks are all just Lua functions that you need to write yourself, but Moonicipal provides scaffolding for these tasks, a nice entry point for loading and running them in coroutines, and ways to collect input and cache it. Because of that, the task writing becomes accessible enough to allow automating even small things you would otherwise have done manually because adding them to your init.lua or creating a plugin for them would have been too much of a hassle.