DEV Community: Idan Arye

Moonicipal Explained

Idan Arye — Wed, 11 Jan 2023 20:32:58 +0000

I've recently published a new task runner plugin for Neovim - Moonicipal. Several years ago I've wrote a similar post about Moonicipal's predecessor - Omnipytent. I've opened that post with an explanation why such a plugin is needed. The same reasoning apply for Moonicipal, so I'll not repeat them here.

Why a new plugin?

When Omnipytent was written in 2016, the Neovim ecosystem looked very different that how it looks today. Plugins were not written in Lua - they were written either in Vimscript or in some external language by using Neovim's RPC API, and many plugins still aimed to support both Vim and Neovim. Additionally, Python 2 was still alive and kicking, with many plugins written in it, and Vim had some issues supporting both versions of Python in the same instance.

So I decided to write Omnipytent using Python, in a way that'll support both Python 2 and Python 3. This means that the tasks themselves can be written in either Python 2 or Python 3. Since this was before Lua plugins, most plugin used Vim commands and/or keybinds as their interface, and these only required passing strings - which were pretty easy and straightforward to build in Python (though no quoting conventions was a bit of a pain)

Nowadays, most Neovim plugins are in Lua, and their interface is Lua functions that often accept tables and functions. These are a bit harder to generate from Python - especially the back-and-forth function calls - and it's starting to seem easier to just write everything in Lua.

Another problem was asynchronous tasks. Having to support Python 2 meant I was limited to Python 2's version of async (only using yield - I couldn't even use yield from!), but that was manageable. A bigger problem was GC - having to maintain a references to the Python generators so that Vim code can resume them, and having to clean that reference manually after usage. With Lua coroutines this is simply not an issue - Lua functions can just be stored as Vim functions.

So, even though Lua is less expressive than Python, its superior interoperability with Neovim and Neovim's plugin ecosystem makes the switch worthwhile.

Using Moonicipal

For the sake of demonstration, I'm going to use the same project I've used in my post about Omnipytent - the Java Sprint example project PetClinic.

After cloning the repository and cding into it, we fire up Neovim and:

So what just happened here?

I started by writing the command :MCedit test to edit the Moonicipal task named "test". This opened a Lua file named .idanarye.moonicipal.lua.

Why that name?
Because I've configured Moonicipal as:

require'moonicipal'.setup {
    file_prefix = '.idanarye',
}

But even if I left file_prefix out of the setup, Moonicipal would just take it from my $USER environment variable.

The purpose of the file prefix is to make the tasks file personal. Sharing task files causes all sorts of problems:

You need to be careful about what you change. You can't, for example, just add a parameter to a command in a task - that will cause problems to other people who use that task.
You can't just use Neovim plugins and locally installed Lua modules - that will break the tasks file for other developers who don't have them. You'll have to add a dependency management (or at least list the dependencies), which is kind of an overkill for a task runner...
You won't be able to put personal information in the tasks file. I'm not talking about passwords or tokens - take extra care with those! - but things like paths on your local machine or URLs to servers assigned to you. Other developers will have their own paths and their own URLs, and if you share a tasks file you'll need to put a mechanism to avoid sharing these.
A shared task file means you are writing code that other people will use, which generally requires higher quality code. Not that high quality code is a bad thing - but it does cost time and effort, and Moonicipal is supposed to allow quick and dirty rapidly changing tasks.

It's better not to commit the tasks file at all, but having a different filename for each developer also helps in ensuring they remain separate.

:MCedit test created a scaffold for the tasks file and for the task:

local moonicipal = require'moonicipal'
local T = moonicipal.tasks_file()

function T:test()

end

A task file is simple. It requires the module and uses its tasks_file() function to create a tasks registrar T. We can then add our tasks as functions on that registrar - like T:test(), which was automatically added because I've added it as an argument to the :MCedit command.

Then I write the body of the task:

vim.cmd'!mvn test'

vim.cmd is a Neovim Lua API for running Vim commands. The parameter passed to it is !mvn test - which means it'll run the :!mvn test Vim command, which is just running mvn test in the non-interactive shell.

What if we want interactive shell? This requires some more commands, but it's still basic usage of Neovim's Lua scripting interface:

Or, if you're more comfortable with Vim commands, we can just use Lua's multiline quotes:

This simple example already demonstrates an advantage of Moonicipal over CLI task runners - we can control, from the test, on how Neovim will run the command. We can run it in the non-interactive shell, open an interactive shell buffer, use a terminal management plugin - whatever we want! If we can code it in Lua, we can put it in a task.

Getting input

Running all the tests is good for CI, but during development we often want to run a single test that focuses on the changes we are currently making. This takes less time to run and less effort to look at the results.

Maven lets us run a single tests class by setting the test property:

vim.cmd.terminal('mvn test -Dtest=ClinicServiceTests')

The Moonicipal tests file is built to be easy to change, so always editing the task when you want to run a different test is not that bad - but we can also make our task receive input:

moonicipal.input is a function for receiving input. By default it functions like Vim's builtin input.

So why can't we just use vim.fn.input()?

Because Neovim has vim.ui.input(), which can be overridden by plugins to offer nicer input UI.

Okay, wise guy, why can't we just use vim.ui.input() then?

vim.ui.input() works with a callback. You pass a function to it, and when the user enters their text the function gets called with that text. If you need more input after receiving the first input, you need another callback in the vim.ui.input() call that's nested inside the callback of the first vim.ui.input() call. This pattern is called "Callback Hell", and considered not very pleasant.

To avoid this callback hell, many programming languages introduce some form of asynchronous execution. Lua does it with coroutines, and Moonicipal runs all its tasks inside coroutines. moonicipal.input() can only run inside a coroutine, and instead of using a callback, it resumes the coroutine once the input was entered.

Entering the test class name each and every time seems like a step backwards though. moonicipal.input() is not really that useful. But just like Neovim has vim.ui.input() and vim.ui.select() - Moonicipal also has moonicipal.select. We can collect all possible test files inside our tasks, and then just select the test file to run from the list:

Note that just like moonicipal.input(), moonicipal.select() uses vim.ui. I get my nice interactive list from fzf-lua, but there are several other plugins which provide their own UIs.

Another new thing we see here is moonicipal.abort(). If I cancel my selection, I don't want the task to open a new window and then fail formatting a command to run in it and print a traceback while keeping the new window open. Instead I use or moonicipal.abort() so that if moonicipal.select() returns nil, it'll just abort the task without much fuss.

Caching choices

Even if we choose it from a list, it is still a waste to have to choose the test every single time we want to run it - especially during development, when we want to keep running the same test over and over. Luckily, Moonicipal has some caching facilities that let us cache our choice:

Caching in Moonicipal is task-bound, so we need to create a new task. They are simple enough to write by hand, but here I used :MCedit choose_test to scaffold a new task. The task's self has some caching facilities - one of them is self:cached_choice(), which lets us cache a selection from a list. Only the choice is cached - the list itself is generated from scratch every time we use the cache - so we must set a key for Moonicipal to recognize the choice by. In this case we use strings, so a simple tostring suffices.

Then we call the cached choice object as a function with all the cache choices. We already had a loop for generating them, so I just reused that.

Finally, we use the select() method to run the actual selection - either by asking the user or by using the cache.

Back in the original test task, we can access the cached choice by calling its task as a method on the registrar object - T:choose_test().

When I run :MC test, it prompts me to choose a test class - just like before. But when I run it a second time - it remembered my choice of test to run!

Eventually, though, we may want to change our choice. To do that, all we have to do is run the choose_test task directly with :MC choose_test. This will let us choose again, even if there already is a cached choice. Of course, it won't actually run the test until we use :MC test again.

Cached buffers and Channelot terminals

Let's leave the tests, and look at another aspect of development cycle - trying queries against a live server. For this, one of Moonicipal's supplemental plugins - Channelot.

Here we also use a new caching facility - cached_buf_in_tab. This method accepts a function, and expects that function to finish in a different buffer. After the function runs, cached_buf_in_tab will automatically jump back to the window where it started (if its still open in the current tab)

The caching part is that if the buffer from a previous run is open in the current tab - the function will not run, and instead cached_buf_in_tab will return the value from its previous run. This makes it ideal for processes like interactive shells that other tasks can use.

As for the plugin I mentioned, Channelot - I hope its API is clear enough:

local j = channelot.terminal_job('bash')
j:writeln('mvn spring-boot:run &')

This creates a bash job, and writes a text line to its STDIN to run the server in a background process. This seems a bit weird - why not simply use vim.cmd.terminal('mvn sprint-boot:run')?

Because we can use the same bash instance to run queries against the server:

By calling T:launch() - just like T:choose_test() from before - we get the returned value from that task, which is the job handler j returned from the launch job. We can than use that handler to send commands the the running bash process, and see the results live in the terminal.

Also, you may have noticed that instead of :MC launch and :MC start here I've used just :MC without arguments, which prompted me to select the task from a list. :MC <task-name> is useful for keybinding, but if you have a good selection UI :MC is more ergonomic for running tasks you don't have keybindings for.

Data cells and BuffLS

One of the most powerful types of input we can cache is a data cell - a buffer that can be edited with Neovim's full editing capabilities, including plugins and language servers. The buffered is stored in memory even when closed (as long as it's not deleted with something like :bdelete), and

(To save precious screen space (I'm recording these at 24 rows on 80 columns - much smaller than typical modern terminals) I'll abandon the bash terminal split and just use print(vim.fn.system(...)) instead (vim.cmd'!...' has problem with line breaks). We'll lose syntax highlighting for the JSON, but I can live with that)

cached_data_cell's argument is a table. The default parameter should be obvious - default content for the buffer, the first time we run the task to edit it. buf is a function (or a string with Vim commands) for preparing the buffer. It is called every time the task is invoked to edit the data cell.

Note that there is no need to create the split - cached_data_cell will do it automatically. This is because buf is called every time the task is invoked, and we want to reuse the split if it is open. You can change how the split is opened with the win parameter, but that's rarely necessary.

Merely setting the filetype to jq gives us syntax highlighting and everything else we have configured for editing jq query files. But we can do more with another one of Moonicipal's supplemental plugins - BuffLS:

require'buffls.TsLs' is BuffLS' Treesitter based language server. Its for_buffer method creates an instance, attaches it to a buffer (current by default - or you can pass a buffer number), and sets the Treesitter syntax based on the buffer's filetype. In this case - jq (so you need to have that syntax installed for it to work - easiest way is to install nvim-treesitter and run :TSInstall jq)

This gives us a BuffLS language server object we can use to configure BuffLS for that specific buffer. Which means we can use it to add very specialized LSP functionalities.

The simplest one is a code action. Not that LSP code actions are that simple in general - but BuffLS uses null-ls behind the scenes, and because null-ls runs inside the Neovim instance its code actions are simply Lua functions. We can add our code actions with ls:add_action - all it needs is a caption for the action and a function.

Actions that set the buffer content to some template are the simplest - all they have to do is use moonicipal.set_buf_content (which is a wrapper for nvim_buf_set_lines with a simpler signature). But of course they can also read the content, configure things in the buffer (or in Neovim in general - but don't get crazy please), or anything else you can do in a Lua function (which is basically everything). Additionally, they run in a Lua coroutine - which means they can use things like moonicipal.select() to get input from the user.

But... we didn't really need the jq Treesitter syntax for that, did we?

Let's add something that does use Treesitter - completion!

By using a Treesitter query, we can detect that we are inside the string part of .firstName == "", and in that case offer appropriate completions.

Treesitter queries can be a bit too complex for most Moonicipal usecases. Luckily, these usecases are often easy to wrap. One common usecase is bash - authoring commands with the specific flags our project uses.

BuffLS supports this with buffls.ForBash:

Conclusion

These tasks are all just Lua functions that you need to write yourself, but Moonicipal provides scaffolding for these tasks, a nice entry point for loading and running them in coroutines, and ways to collect input and cache it. Because of that, the task writing becomes accessible enough to allow automating even small things you would otherwise have done manually because adding them to your init.lua or creating a plugin for them would have been too much of a hassle.

Is using an executable to help client libraries connect to a server a good architecture?

Idan Arye — Mon, 25 Jul 2022 14:49:34 +0000

I have a project that has three components:

A server that exposes a WebSocket interface.
A GUI that connects to the server.
Client library that is used in user programs and also connect to the server.

The idea is that the host program controls the GUI through the server.

This is a pet project I barely work on, but I like to dream big. I want the three parts to run on different hosts (which mean I'll need credentials and user authentication, which I haven't implemented yet because I still run them all on the same machine and only connect via the loopback). I also want the client library to be implemented in many different languages, so that they can all use my project.

The Problem

Now, while I'm trying to keep the API simple, adding all the security to each implementation of the client library could be tedious, and if I ever need to do security changes (how often are these necessary anyway?) they'll have to be done in all the implementations.

Another problem is concurrency. My WebSocket API is asynchronous - you send a request with an ID and eventually you get a response with the same ID, but meanwhile other messages from other threads can flow in the channel. This works well for the server and for the GUI, because both use an actor system, but the clients are not going to use actors. This kind of concurrency can still be implemented without actors, but it takes work - I managed to do it in Python (my first and currently only implementation of the client library) but it was a bit tricky and I don't want to repeat this for every language...

The Solution

I'm thinking about adding a relay between the client and the server. The relay will be an executable that client library invokes as a child process and pass it a configuration file with all it needs to know to connect to the server. The relay process will login to the server, and also open a 127.0.0.1-only port and print its number (and maybe a security token?) to STDOUT. The client library will store this information - and that's all it has to implement authentication-wise.

For actual usage, the client library will simply connect to the local port. Since this is a loopback and there are no security constraints (except maybe passing a secret token to avoid other processes on the same machine discovering the port and using it), and because there is no need for a handshake (the relay process will be a single session), this should be relatively fast and cheap, and also easy to implement in any language. For concurrency, I'll just open multiple connections to the same port. Maybe even add an option to invoke the relay executable with different parameters to send a request and receive a response.

Another benefit of this architecture is that in the future I'll be able to implement more advanced features (dreaming big!) without having to implement them for each client library implementation. For example - HTTP tunneling between the GUI and the client via the server.

The Drawback

While the relay executable will make the client library implementations easier to write - it'll also make them harder to distribute. Most modern languages have their own package manager built into their build system, capable of easily adding dependencies written in that language, but adding binary dependencies is harder. Especially if you want to support multiple platforms.

It might be easier to install the relay separately, but that would make it harder for the end users. And while my project is going to be the greatest thing since sliced bread I still don't want to scare people away with complex installation instructions...

Are my concerns legit? What would be the best design decision here?

Need help with terminology for humans that follow instructions from a script

Idan Arye — Wed, 03 Jul 2019 14:32:32 +0000

I'm trying to name things in my project, and I've been struggling too long to find names for two roles. These names are important because they are going to appear as prefixes/suffixes all over the codebase.

One of these roles is for a script that connects to the server and transmits instructions. The other role is for humans that connect to the server with a client. The client shows the instructions from the script to the humans, and they need to follow them and click on a button to report when they have finished.

My initial thought was to call them Host and Client - but the host is also a client that connects to the server, so that doesn't feel right.

I also thought to call them Script and Human, but "script" already packs too much meaning in general. Same with Plan and Worker.

Does anyone have any ideas? Maybe common terms for these roles from fields where this pattern is common?

Analogical naming - yay or nay?

Idan Arye — Tue, 02 Jul 2019 16:28:45 +0000

I'm working on a project and struggling with one of the two hard things - naming things. I have names - but I don't really like them (the most descriptive names I could find for the two main roles only differ in a single letter)

When I picked a name for the project itself, I used an analogy to fantasy adventures. The software itself is not related to the fantasy genre, but I'm starting to think about using the same analogy to name some of the major components and processes in my architecture.

But I'm still not sure about this. It was fine for naming the project itself - projects need unique and memorable names - but the names inside the project should be as descriptive as possible, to aid readers who need to understand what does what. Not that I suggest arbitrary naming - I have a clear and concise analogy and I'm not going to just pick a random fantasy term for every software term that needs to be named - but I still fear it may look unprofessional to have classes named Adventurer or Wizard or GuildHall...

Should I go for it?

Are there any API specification formats for WebSockets?

Idan Arye — Wed, 01 May 2019 19:03:34 +0000

The WebSocket protocol allows sending messages back-and-forth between a server and a client, but it does not define the structure of these messages. They can be strings or blobs, but how these strings/blobs are structured is left to the user.

This is not a bad thing - you want this layer of abstraction to be simple so it'll be easy to implement and to build on - but with HTTP - which is similar in that way (you can only send plaintext with some headers) there are many protocols built on top of it that add this structure: OpenAPI, OData, XML-RPC and many more.

Some of these protocols offer schema generators - you write your protocol's specification once in the protocol's favorite format, and it generates types and/or functions for using your protocol in many different languages.

I tried looking for something similar for WebSocket, but all I could find is AsyncAPI. It has a generator, which can output using templates - but it only has templates for Java (Spring, actually), HTML and Markdown (the last two are for generating docs) and googling for third-party templates yielded nothing (except results from their issue tracker, which has some requests to add templates for more languages).

Is there anything more complete than AsyncAPI? If this is the best we have I'll use that - good documentation generation is already a bonus, and I can probably contribute myself templates for the languages I use - but if there is something better I'd rather just use that.

Omnipytent 1.3.0: Async Tasks and Selection UIs

Idan Arye — Thu, 15 Nov 2018 23:58:58 +0000

If you don't know what Omnipytent is, read this first:

Omnipytent Explained

Idan Arye ・ Dec 19 '17

#vim #workflow #automation

Version 1.3.0 of Omnipytent introduces a new concept - async tasks. In this post I'll try to explain what are async tasks and what are they good for.

The problem: single-threaded, event loop based UI

(you can skip this section if you already understand why we need async tasks)

Vim has a single-threaded architecture. When you perform an operation (by typing a key, running a run-mode command, using :autocmd etc.) that operation takes over the thread, and nothing else can get updated. The operation may receive input from the user (e.g. with input() or getchar()) or update the TUI (e.g. with :echo or by running a shell command with :!) but Vim's event loop itself is stuck until the operation is finished. You can't run other commands, which means that there is nothing to update the TUI, but even the jobs and terminals can't read anything from their streams while that operation is running.

Your everyday Vim operations are quick and "atomic" enough for this to not be a problem. Vim can't do anything else when you type w twe needo jump to the next word, but this happens so fast that it doesn't matter, and you don't need Vim do do anything else during. If you are running shell commands, on the other end, they can take quite long and you'd have to patiently wait for them to finish. Luckily, Neovim and Vim8 have jobs and terminals, so you can just launch it and it runs in the background.

But what if you need the "result" of the command you ran in the terminal? fzf, for example, is running fzf in a terminal, and use the result to open the chosen file. This can't be done in a single Vim function invocation - it needs in one function to start the terminal, then yield executing back to the event loop, and once fzf exist run another function to deal with the result. If it was done in the same function that launched the terminal, Vim would be unable to update the terminal buffer and the user wouldn't be able to use it.

And it's not just shell commands - sometimes you just need to user to use Vim's UI itself. A good example is fugitive.vim's :Gcommit command. It opens a new buffer for the user to type the commit message in, and when they save and close that buffer the plugin creates a new Git commit with that commit message. Until the function called by :Gcommit finishes it hogs the event loop and the user can't write the commit message nor can they save and close the window, so :Gcommit must terminate before that and another function must be called when the window is close to finish the process.

Plugins like fugitive or the one bundled with fzf are registering callbacks to implement that behavior. It's not that hard to do, but it is quite cumbersome (even more than in JS!) and requires some familiarity with Vimscript and Vim's architecture. Omnipytent tasks are supposed to be simple and streamlines, so this callback registration is too much - we need something simpler. And that's where async tasks come in.

The idea: generator based async tasks

(you can skip this section if you don't care how they work, and go directly to the next sections for examples on how to use them)

A generator function in Python is a function that has at least one yield expression. If you want to learn more about it read the Python docs - for our purpose it's enough to mention that the yield yields both a value and the execution itself - so if a callee yields, the execution goes back to the caller which can chose, at a later point, to resume the callee from where it yielded.

This was used for async IO before the async and await keywords were introduced in Python 3.5, and since Omnipytent wants to support older versions it uses yield for its async tasks.

An async task looks something like this:

@task
def my_async_task(ctx):
    do_something()
    result = yield ASYNC_COMMAND()
    followup(result)

When the task yields, ASYNC_COMMAND() does three things:

It prepares of the command (e.g. open a window or start a terminal)
It registers itself in Omnipytent.
It registers a callback/:autocmd in Vim to resume itself once done.

After that - :OP my_async_task terminates and control goes back to Vim's event loop. But my_async_task itself is not terminated yet - at some point it will be resumed and the task will continue and do the followup.

And of course - you can yield another async command later in the task.

Omnipytent comes bundled in with some useful async commands. If you need to create your own - refer to :help omnipytent-creating-AsyncCommand.

`INPUT_BUFFER` - basic async user input

For the example, I'll use the same example project from the first post - Spring's example pet clinic web application. Lets say we want a task for adding animal owners. The API is simple - a POST request with the details - but how will we prompt the user (which is us) to enter the fields?

Up until now, we'd have to use input() to allow the user to enter that data. With async tasks, we can do better:

import requests, yaml

OWNER_FIELDS = ('firstName', 'lastName', 'address', 'city', 'telephone')


@task
def add_owner(ctx):
    empty_form = '\n'.join('%s: ' % field for field in OWNER_FIELDS)
    filled_form = yield INPUT_BUFFER(text=empty_form, filetype='yaml')
    parsed_form = yaml.load('\n'.join(filled_form))
    requests.post('http://localhost:8080/owners/new', data=parsed_form)

The important line is the second line of the task function - the one where we do the yield. INPUT_BUFFER is one of the async commands bundled with Omnipytent, and it opens a new window with a buffer for the user to edit. We set the original text - a YAMLish form with blank fields for the user to fill - and also set the file type to YAML to get some nice coloring. Then we yield this command object - and the execution control returns to Vim's event loop. Now the user can fill the form, and when they close it Omnipytent kicks back in and the task resumes - with filled_form set to the lines of the buffer the user filled.

This is how it looks in action:

`CHOOSE` - the power of selection UIs

The primary motivation behind async tasks was to support selection UIs. These are usually fuzzy matchers like fzf or Unite, but I use the term Selection UIs because this mechanism is not limited to tools that allow fuzzy search - any tool that provides a TUI for selecting from a list can fit.

Tests can accept arguments, but it's not always convenient to type the arguments. When the arguments are long, or complex, or hard to remember, and when there is an easy way to programmatically generate the list of possibilities, it is far more convenient for the user to filter and pick what they want with a selection UI.

For my next trick I'll need to add a missing method to the example project:

// src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java

    @GetMapping("/owners.json")
    public @ResponseBody List<Map<String, Object>> showResourcesVetList() {
        return this.owners.findByLastName("").stream().map(owner -> {
            Map<String, Object> entry = new HashMap<>();
            entry.put("id", owner.getId());
            entry.put("firstName", owner.getFirstName());
            entry.put("lastName", owner.getLastName());
            entry.put("address", owner.getAddress());
            entry.put("city", owner.getCity());
            entry.put("telephone", owner.getTelephone());
            return entry;
        }).collect(Collectors.toList());
    }

All it does is add a new path - /owners.json - that generates a JSON of all the owners. Nothing fancy, but the orig example project only supported getting them as an HTML page, which is harder to parse (unless you use a regex)

We want to write an Omnipytent task that reads that list with a GET request, lets the user pick one of the owners, edit it, and update it with a POST request. To do the selection, we are going to use another async command - CHOOSE:

import json


@task
def edit_owner(ctx):
    entries = json.loads(requests.get('http://localhost:8080/owners.json').text)
    entry = yield CHOOSE(
        entries,
        fmt='{firstName} {lastName}'.format_map,
        preview=lambda entry: yaml.dump(entry, default_flow_style=False))

    owner_id = entry.pop('id')
    orig_form = yaml.dump(entry, default_flow_style=False)
    edited_form = yield INPUT_BUFFER(text=orig_form, filetype='yaml')
    parsed_form = yaml.load('\n'.join(edited_form))

    edit_url = 'http://localhost:8080/owners/%s/edit' % owner_id
    requests.post(edit_url, data=parsed_form)

CHOOSE runs whatever selection UI installed in your Vim. It first checks for fzf, then Denite, Unite, CtrlP, and finally - if none of the above is available, it uses an inputlist() based selection UI. Or - if you have other preferences - you can set the selection UI with g:omnipytent_selectionUI.

Other than the list of options, we pass two more arguments to CHOOSE:

fmt: The options are dicts but the selection UIs pick from lines. This argument is a function that formats each option into a line.
preview: fzf, Denite and Unite support a preview of the items, and this argument is a function for rendering that preview.

After we yield CHOOSE, we get back the picked option and use it to display an INPUT_BUFFER - which I already explained earlier.

Let's see it at work:

Multi selection and generator options tasks - an actually motivational example

The previous examples were a nice way to show the power of async tasks, but they are not really something you'd write Omnipytent tasks for. If you want to actually add and edit owners you'd use the web application, and if you need a quick way to run these while developing to test your code, editing the text buffer each and every time is a bit cumbersome.

So how about something you actually want to create a task for? How about... running tests?

Running tests is a great use case for CHOOSE. When I work on a piece of code I often want to run a test that checks it. I don't want to run all the tests, because it'll take too long and will generate too much output. I want to run just this specific test and see just its output - which will reflect the changes I did to the code.

Test names are often long - they need to encode the name of the class/module that contains the test, the name of the test function itself, and sometimes the parametrization. Writing the full name each time is cumbersome. Writing that name inside the test makes it easier to run, but now we have to edit the tasks file when we want another test - and figure the name of that test. Not that hard, but can now do better - we now have CHOOSE!

Decent project management tools usually have a command for listing all the tests, but this is Maven so we'll have to parse the files ourselves:

import re


@task
def run_tests(ctx):
    pattern = re.compile(r'@Test\n\s*public void (\w+)\(')
    test_names = []
    for path in local.path('src/test').walk(lambda p: p.basename.endswith('.java')):
        for match in pattern.finditer(path.read('utf8')):
            test_names.append('#'.join((path.stem, match.group(1))))

    chosen_tests = yield CHOOSE(test_names, multi=True)

    cmd = local['mvn']['test']
    cmd = cmd['-Dtest=' + ','.join(chosen_tests), 'test']
    cmd & TERMINAL_PANEL

Note that there is a new argument to CHOOSE: multi=True. As you may have guessed from the name, it allows the user to select multiple options. Only fzf, Denite and Unite support this, but even with CtrlP and inputlist() it'll still return a list to keep some uniformity in the task.

We then join the chosen tests, and voila!

But... we still need to pick the test we want to run each time. Picking it with fzf is definitely better than typing it, but since we usually want to run the same test(s) many times when we work on the same area of the code, it could be nice if Omnipytent could remember our last choice.

Well - it can. Omnipytent already had @task.options that remembers the user's choice, but you could only pick one option and the option keys had to be hard-coded as local variables. Omnipytent 1.3.0 solves both these problems:

@task.options is now based on CHOOSE - so it can use more elaborate selection UIs. A new variant - @task.options_multi - allows you to pick multiple options. If you are using CtrlP or inputlist() and still want multiple choices you'll have to pass them as arguments. Or just upgrade to fzf/Denite/Unite.
If the task function is a generator, instead of using the local variables as options it uses the yielded values as options.

This means we can split our run_tests into two tasks: pick_tests and run_tests. pick_tests will always prompt us to choose the tests, but run_tests will remember our last choice:

@task.options_multi
def pick_tests(ctx):
    ctx.key(str)
    pattern = re.compile(r'@Test\n\s*public void (\w+)\(')
    for path in local.path('src/test').walk(lambda p: p.basename.endswith('.java')):
        for match in pattern.finditer(path.read('utf8')):
            yield '#'.join((path.stem, match.group(1)))

@task(pick_tests)
def run_tests(ctx):
    cmd = local['mvn']['test']
    cmd = cmd['-Dtest=' + ','.join(ctx.dep.pick_tests), 'test']
    cmd & TERMINAL_PANEL

Note the first line of pick_tests: ctx.key(str). Because the yielded options can be objects, we need a string keys of them, and ctx.key sets the function for picking these keys. The key must be deterministic, because these keys will be used to cache the choice. There is also ctx.preview for setting a preview function, but we don't need one here.

And here is how it works:

Conclusion

Omnipytent's goal was to allow micro-automation of simple project tasks. Async tasks allow you to add better UI to that automation, farther enhancing the power at your fingertips.

Don't initialize your variables

Idan Arye — Sun, 05 Aug 2018 17:23:51 +0000

Developers coming from C know that variables should always be initialized. Not initializing your variables means they contain junk, and this can result in undefined behavior. For example:

#include<stdio.h>

int main(void) {
    char buffer[256];
    char answer;
    char* name;

    printf("Do you want to enter a name? [yn] ");
    answer = getchar();

    while (getchar() != '\n') { } // because we need CR for getchar but it doesn't read the CR...

    if (answer == 'y') {
        printf("Please enter name: ");
        name = fgets(buffer, 256, stdin);
        if (name == 0) {
            name = "<too long>";
        }
    } else if (answer == 'n') {
        name = "<user refused to enter name>";
    }

    printf("The name is %s\n", name);
    return 0;
}

If the user entered a character that is not y or n, not of the name = ...; statements will be executed, and name will still hold the same value it had when main started. What is that value? In release mode C, that would be whatever random data happened to be in that piece of memory name was assigned. And then we take that utterly random number and pass it to printf where it'll get printed as if it was a string pointer!

If we are lucky, we'll hit some illegal memory address and the OS will stop us. If we aren't it'll just go to some random place at memory and start printing whatever it encounters: passwords, credentials, application tokens...

And of course - this will not be reproducible. Because every time you run the program, there will be a different value at that place in memory and you'll get different results.

To avoid these problems, C developers have conditioned themselves to always initialize their variables. If you don't have something meaningful to put in the point of declaration - just put 0:

#include<stdio.h>

int main(void) {
    char buffer[256] = {};
    char answer = '\0';
    char* name = 0;

    printf("Do you want to enter a name? [yn] ");
    answer = getchar();

    while (getchar() != '\n') { } // because we need CR for getchar but it doesn't read the CR...

    if (answer == 'y') {
        printf("Please enter name: ");
        name = fgets(buffer, 256, stdin);
        if (name == 0) {
            name = "<too long>";
        }
    } else if (answer == 'n') {
        name = "<user refused to enter name>";
    }

    printf("The name is %s\n", name);
    return 0;
}

While null pointer dereference is still formally an undefined behavior, it is still much better than random pointer dereference because your operation system will probably make it s SEGFAULT - which is better than security leaks.

OK, but that's C. What about more modern languages?

There are two main reason this was so needed in C:

Uninitialized variables having junk data.
Inability to declare variables in the middle of a block.

More modern languages allow declaring variables in the middle of a block, so it is usually preferable to only declare the variable at the point where you have something meaningful to put in it.

This greatly reduces the cases where you have to initialize something with a default value - but does not prevent all of them. In our case, for example, name gets its value inside if branches - if we declared it there we wouldn't be able to use it after the if. Some languages (mostly the functional ones) have easy syntax solution, but in most mainstream languages you'd have to either extract it to a function or declare the variable outside the block.

When going with the latter solution, because C is such a common background, many developers will initialize the value. So if we convert our code to Java:

import java.util.Scanner;

public class Main {
    public static void main(String[] args) {
        Scanner scanner = new Scanner(System.in);

        System.out.print("Do you want to enter a name? [yn] ");
        String answer = scanner.nextLine();

        String name = null;
        if ("y".equals(answer)) {
            System.out.print("Please enter name: ");
            name = scanner.nextLine();
        } else if ("n".equals(answer)) {
            name = "<user refused to enter name>";
        }

        System.out.printf("The name is %s\n", name);
    }
}

Sure, this is Java, a language with managed memory that will never allow undefined behavior from uninitialized variables, so we don't really need to initialize name to null, but better safe than sorry, right?

WRONG!

Java analyses code paths to make sure no variable can be used without being initialized first. So if we remove the initialization:

import java.util.Scanner;

public class Main {
    public static void main(String[] args) {
        Scanner scanner = new Scanner(System.in);

        System.out.print("Do you want to enter a name? [yn] ");
        String answer = scanner.nextLine();

        String name;
        if ("y".equals(answer)) {
            System.out.print("Please enter name: ");
            name = scanner.nextLine();
        } else if ("n".equals(answer)) {
            name = "<user refused to enter name>";
        }

        System.out.printf("The name is %s\n", name);
    }
}

We'll get a compilation error:

$ javac Main.java 
Main.java:18: error: variable name might not have been initialized
        System.out.printf("The name is %s\n", name);
                                              ^
1 error

I just broke the compilation, but this is a good thing - the compiler found a bug! The same bug we had in the C version - what if the user enters something which isn't y or n. The Java compiler sees that there are three possible code paths that reach the last line but we are only initializing two of them.

To be able to compiler again, we must tell Java what to do in case the user gave an invalid answer. Failure is also an option - as long as we do it intentionally:

import java.util.Scanner;

public class Main {
    public static void main(String[] args) {
        Scanner scanner = new Scanner(System.in);

        System.out.print("Do you want to enter a name? [yn] ");
        String answer = scanner.nextLine();

        String name;
        if ("y".equals(answer)) {
            System.out.print("Please enter name: ");
            name = scanner.nextLine();
        } else if ("n".equals(answer)) {
            name = "<user refused to enter name>";
        } else {
            System.err.printf("Illegal answer \"%s\". The only legal answers are \"y\" and \"n\".", answer);
            return;
        }

        System.out.printf("The name is %s\n", name);
    }
}

Now there are still three code paths, but in the third we return from the function early, before printing name. The Java compiler can determine that there are no code paths where name is used without being assigned a value first - and thus the compilation succeeds.

This is still initialization

Despite the clickbaity title, we do actually initialize name. We don't do on declaration, but we are initializing it nevertheless. This compiles:

import java.util.Scanner;

public class Main {
    public static void main(String[] args) {
        Scanner scanner = new Scanner(System.in);

        System.out.print("Do you want to enter a name? [yn] ");
        String answer = scanner.nextLine();

        final String name;
        if ("y".equals(answer)) {
            System.out.print("Please enter name: ");
            name = scanner.nextLine();
        } else if ("n".equals(answer)) {
            name = "<user refused to enter name>";
        } else {
            System.err.printf("Illegal answer \"%s\". The only legal answers are \"y\" and \"n\".", answer);
            return;
        }

        System.out.printf("The name is %s\n", name);
    }
}

Wait - how? Didn't they teach us that you can't change the value of a final variable?

Well, yes, but we are not changing the value of any final variables here - we are just initializing it. Since name has never been assigned before in either of the paths that assign to it, these assignments are actually initializations - which are perfectly fine for final variables. It wouldn't have worked with final String name = null, but without the initialization on declaration it's fine, and even without the final name could be used in lambdas (provided they appeared after the first assignment).

Conclusion

Do initialize your variables - but don't always force a default value when you can't initialize them with a proper one. Know how your language behaves with uninitialized variables and pick the best strategy for uncovering bugs.

Abusing Type Erasure: Passing Back the Same Subclass the Object Yields

Idan Arye — Mon, 22 Jan 2018 16:24:16 +0000

Java's type erasure generics are generally limited compared to real generics/templates because you don't have access to the actual class object(unless you pass it as runtime parameter) so you can't create new instances of it, create arrays of it, or inspect it. But recently I've encountered a case where type erasure is actually helpful.

The problem

Lets say we have this interface:

public interface Foo {
    public Object createValue();
    public void handleValue(Object value);
    public void printValue(Object value);
}

Implemented by these classes:

import java.util.LinkedList;
import java.util.List;

public class Bar implements Foo {
    @Override
    public Object createValue() {
        List<String> list = new LinkedList<>();
        list.add("one");
        list.add("two");
        list.add("three");
        return list;
    }

    @Override
    @SuppressWarnings("unchecked")
    public void handleValue(Object value) {
        List<String> list = (List<String>) value;
        for (int i = 0; i < list.size(); ++i) {
            list.set(i, "(" + list.get(i) + ")");
        }
    }

    @Override
    @SuppressWarnings("unchecked")
    public void printValue(Object value) {
        List<String> list = (List<String>) value;
        for (String entry : list) {
            System.out.printf("- %s\n", entry);
        }
    }
}

public class Baz implements Foo {
    @Override
    public Object createValue() {
        return new int[]{ 1, 2, 3 };
    }

    @Override
    public void handleValue(Object value) {
        int[] array = (int[]) value;
        for (int i = 0; i < array.length; ++i) {
            array[i] = array[i] * 10;
        }
    }

    @Override
    public void printValue(Object value) {
        int[] array = (int[]) value;
        for (int entry : array) {
            System.out.printf("- %s\n", entry);
        }
    }
}

Foo is an API for using a value - it can create it, do something with it, and print it. Bar implements Foo with a list of strings that can be surrounded by parenthesizes, while Baz implements it with an array of integers that are multiplied by 10. There is a hidden assumption here that handleValue and printValue accept objects of the same type returned by createValue of the same Foo object:

public class App {
    public static void main(String[] args) {
        Foo[] foos = new Foo[]{ new Bar(), new Baz() };
        for (Foo foo : foos) {
            Object value = foo.createValue();
            foo.handleValue(value);
            foo.printValue(value);
        }
    }
}

So - that assumption is easy to fulfill in actual code, but nothing forces it.

Proper OOP practices dictate that handleValue and printValue should be methods of the value:

public interface Value {
    public void handle();
    public void print();
}

public interface Foo {
    public Value createValue();
}

Sadly, this is not always possible. Sometimes we don't control the types returned from createValue. Sometimes we do control them, but they belong in a different module that should not know about these particular operations.

Generics to the rescue?

This is what generic are for, aren't they? Just make Foo use a generic parameter instead of Object:

public interface Foo<T> {
    public T createValue();
    public void handleValue(T value);
    public void printValue(T value);
}

import java.util.LinkedList;
import java.util.List;

public class Bar implements Foo<List<String>> {
    @Override
    public List<String> createValue() {
        List<String> list = new LinkedList<>();
        list.add("one");
        list.add("two");
        list.add("three");
        return list;
    }

    @Override
    public void handleValue(List<String> value) {
        for (int i = 0; i < value.size(); ++i) {
            value.set(i, "(" + value.get(i) + ")");
        }
    }

    @Override
    public void printValue(List<String> value) {
        for (String entry : value) {
            System.out.printf("- %s\n", entry);
        }
    }
}

public class Baz implements Foo<int[]> {
    @Override
    public int[] createValue() {
        return new int[]{ 1, 2, 3 };
    }

    @Override
    public void handleValue(int[] value) {
        for (int i = 0; i < value.length; ++i) {
            value[i] = value[i] * 10;
        }
    }

    @Override
    public void printValue(int[] value) {
        for (int entry : value) {
            System.out.printf("- %s\n", entry);
        }
    }
}

So, we solved the problem of having to downcast in handleValue and printValue. So now we are typesafe, right?

WRONG!

We did not resolve the type issue - we have simply moved it from the declaration to the usage. When we loop over our array of Foos we are still keeping the value from createValue as Object, and still passing it as Object to handleValue and printValue.

Java is kind enough to warn us about it:

$ javac App.java -Xlint:unchecked
App.java:6: warning: [unchecked] unchecked call to handleValue(T) as a member of the raw type Foo
            foo.handleValue(value);
                           ^
  where T is a type-variable:
    T extends Object declared in interface Foo
App.java:7: warning: [unchecked] unchecked call to printValue(T) as a member of the raw type Foo
            foo.printValue(value);
                          ^
  where T is a type-variable:
    T extends Object declared in interface Foo
2 warnings

But... what can we do about it?

The solution

public class App {
    public static void main(String[] args) {
        Foo[] foos = new Foo[]{ new Bar(), new Baz() };
        for (Foo<?> foo : foos) {
            createHandlePrint(foo);
        }
    }

    private static <T> void createHandlePrint(Foo<T> foo) {
        T value = foo.createValue();
        foo.handleValue(value);
        foo.printValue(value);
    }
}

I created an helper method - createHandlePrint. This method has a generic parameter, but it's caller doesn't pass it - neither explicitly nor implicitly. We only need it inside, to be be able to declare that value is the same type returned by the createValue of the foo passed to us, so it is safe to pass it back to that foo.

This is only possible thanks to type erasure. With real generics createHandlePrint would have been a different method for each parametrization of T, and I wouldn't be able to call them both from the same line of main. Unless I used reflection - which kind of defeats the purpose of having the compiler verify the type... (and also doesn't work with all implementations of generics)

Omnipytent Explained

Idan Arye — Tue, 19 Dec 2017 21:48:51 +0000

My Omnipytent plugin for Vim is a central part of my workflow, but it seems to be a bit hard for other people to grok it. More specifically - to understand why would they need something like that. So - I'm making this post to explain the rationale behind Omnipytent and to demonstrate it's worth.

The problem: running commands

TL;DR - this section explains why I needed to create a plugin for something most developers just... do. If you don't care about justifications, just skip it and go directly to the next session - "Using Omnipytent".

This may look like a solved problem - you have ran commands before you ever heart of Omnipytent. Heck - you probably ran commands before you even heard of Vim! Why do you need a plugin for that? That's what the command line is for!

As Vimmers we tend to adhere the "Unix is my IDE" philosophy - every development task has a command line tool, and we just run it with the arguments we need. So - my project can be built and ran with simple commands. And I know these commands. But...

Do I really have to type the same command each and every time I want to run it?
And if I need to build with different arguments - do I add them every time I type the build command? Or do I change the actual build file to make them the default?
I want to run a specific test - do I need to type it's name every time? Or, I can paste it - but then I need to keep it in the clipboard, or copy it each time...
I need to run the tool with certain arguments(yes, this should be a test, I'll make this a test, I promise!) - do I type these arguments every time?

You get the idea - I'm lazy and I don't want to keep typing the same command line commands over and over again. What can I do?

So... map some keys?

An obvious choice - if there is a command you use a lot and want quick access to, just set a keymap. A simple solution - but for me, at least, it was not flexible enough:

I work on several projects in different languages and environments, and I need different commands to build each project. The usual Vim solutions is to use :nnoremap <buffer>s :autocmds or in filetype plugins - but what keys will I set, for example, for XML files? I may want to build while in one of those!
Even in the same environment, I need different ways to run(and sometimes build) different projects. Each project has it's own entry point, and unless you only work on single-file scripts that entry point will not be the file in your current buffer. Having different config for each project does not scale.
Even in the same project, I often want to change these commands. Build with a different flag, run with different arguments. Changing the keymap each time is too much trouble.

No... I want something I can easily change - without touching my .vimrc!

Just use the command line history like everyone else

A straightforward solution - if you use a command a lot, it's going to be in the history, so just Ctrl+r in Bash to find it.

That's good and all when you work a single project, but with many projects - they are just going to override each other's history. So... I'd still like something better. Also, I may be risking getting my Vim card revoked - but I really don't want to context-switch to a terminal emulator every time I want to build or run what I'm working on. And Vim's own history as not as easy to navigate - not to mention I need it for command-mode commands...

You spoiled brat! Put these commands in `.sh` files and get it over with!

That's another common solution(though apparently not as common as the first two) - creating simple scripts for the commands. From the shell, or from Vim's :!, it's easy to tab-complete and launch these scripts. And I can even set keymaps to the different scripts, and have the same keymap do project-specific stuff in different projects!

But... all these files scattered all over the project create a huge mess. So we need to:

Have a single file containing all these commands

Now we are getting somewhere! It can be a simple bash script with case on the first argument, and you could just set it to have as many commands as you want. Or - you could go fancy and abuse a build system, which usually have nicer syntax, and use their tasks as commands.

So that's what I did - I chose Rake, because it wasn't colliding with existing build systems(it's mainly used with RoR, and I didn't really need to have build and run commands there(at least not at first)), and I get to write my tasks in Ruby instead of Bash. Yey!

But I still wanted more. What if, I thought, I had a plugin to ease the creation of new tasks, autocomplete task names for me, help me easily use a different file than the one rake targets without additional args, etc. etc.

And then I realized - Rake is a Ruby library! If I can load it into Vim with the Ruby interface, I can run my tasks inside Vim, and they'll have access to my Vim instance! This opens a new world of possibilities - I can make a task run with regular :!, or with Erroneous to fill the quickfix list, or in a VimShell terminal(Vim did not have :terminal back then, and Neovim was not even conceived), or load a log file in a buffer, or... or anything I wanted!

And that's how Integrake was born. And when I had to abandon Ruby so I can move to Neovim - I rewrote the whole thing in Pyhton - and that's Omnipytent.

Using Omnipytent

Tohttps://thepracticaldev.s3.amazonaws.com/i/229bjqis5lejr6ma3rhc.gif demonstrate Omnipytent I will use this example Java Spring project.

Simple tasks and commands

So, after cloning the repository, let's say I want to run tests. This is a Maven project, so we run tests with mvn test. I'm going to do it with Omnipytent:

So... what happened here?

First thing first - I have set these in my .vimrc:

let g:omnipytent_filePrefix = '.idanarye'
let g:omnipytent_defaultPythonVersion = 3

This means I want my tasks file to begin with my name and be hidden(.idanarye) and I want to use Python 3(you can write your tasks files in Python 2 or Python 3). The tasks file is supposed to be personal, so we won't have to invest effort in making it portable to allow other developers to use it. We don't check it in to source control, and even if we do - other developers that happen to use Omnipytent will have their own tasks file with a different name.

So, with Vim opened in the repository's root, I run :OPedit test and it opens the task file for that project - .idanarye.omnipytent.3.py. Because the file did not exist before it added some imports, and because I wanted to edit a non-existing task it created a skeleton for that task - and all that's left is to write the task's body:

import vim
from omnipytent import *


@task
def test(ctx):
     BANG << 'mvn test'

So, what do we have here? Imports are imports - we have vim - the built-in interface for Vim from Pyhton - and a start import from omnipytent with the common things you are going to want to use in a tasks file. One of the is task - a decorator we use to - surprise surprise - create tasks. The other is BANG - a Shell Command Executor. Shell command executors are handles for running shell commands - BANG specifically is using Vim's bang command(:!). There are other shell command executors, and you can define your own - it's all in the docs. The << operator in shell command executors can be used to execute a string as a shell command. And once I save the tasks file and run :OP test - Omnipytent executes the code of the test task and runs all the tests.

The << operator executes the string as is. You can also use it as a function - and it'll quote each argument(which is better if you get them from a variable):

@task
def test(ctx):
     BANG('mvn', 'test')

The third way to use shell command executors is with Plumbum - a shell combinator library for composing shell commands with Pythonic syntax. If you have it installed, and import omnipytent.interation.plumbum in your tasks file, you can use the shell command executors like Plumbum's background and foreground modifiers BG and FG. Since you usually want Plumbum's local to start the commands, you can import that from omnipytent.interation.plumbum:

from omnipytent.integration.plumbum import local
local['mvn']['test'] & BANG

Since we are going to be using Maven quite a lot, might as well bind local['mvn'] globally:

import vim
from omnipytent import *
from omnipytent.integration.plumbum import local

mvn = local['mvn']


@task
def test(ctx):
    mvn['test'] & BANG

OK - but :! is not a very convenient way to run a test - certainly not with a tool that spans you with text like Maven. How about we run it with a terminal emulator instead? It's simple - all we have to do is use a different shell command executor - TERMINAL_PANEL - and it'll create a Vim 8 or Neovim's terminal emulator:

OK... but why?

So far, we didn't really need Omnipytent - couldn't we just run these commands, right from Vim's command mode? Well, yes, but running them with Omnipytent has two advantages:

This is a Maven project, so the command is :!mvn test. If it was a Gradle project, I'd need :!gradle test. And with Ant I'd need :!ant test. Or maybe :!ant junit? Ant is free-spirited like that, so it can be anything.

And what about flags you sometimes need to set? And that's just Java - other languages have their own various build systems...

With Omnipytent, it's always :OP test - because you don't depend on what build system the project's creator picked and how they chose to configure it - you always create your own Omnipytent task test to run it. You can even map a key to it, and it'll work with any project(after your created the task for it). Personally, I mapped many short generic verbs to an "Omnipytent leader" followed by their first letter:

They don't all do something on all my projects, but as projects get big I tend to have many different useful tasks and it's nice to have keymaps available for them.
Sometimes commands need arguments - like the test's name when you want to run a specific test. You can't bind that in your global .vimrc because it's constantly changing - but it's easy to edit an Omnipytent file:

import vim
from omnipytent import *
from omnipytent.integration.plumbum import local

mvn = local['mvn']


@task
def test(ctx):
    mvn['test']['-Dtest=ClinicServiceTests#shouldFindOwnersByLastName'] & TERMINAL_PANEL

So - we can use :OP test(or the key(s) we mapped to it) to run this test, and when we want to work on a different test - we can just edit the tasks file.

OK - but what if we want something more dynamic? Maybe we don't want to edit the tasks file each time, and prefer to give the test to the command? We can do that too - with task arguments:

Task arguments and completion

@task
def test_specific(ctx, testname):
    mvn['test']['-Dtest=' + testname] & TERMINAL_PANEL

And now, we just need to give that argument to our task with:

:OP test_specific ClinicServiceTests#shouldFindOwnersByLastName

Aaaaannnd... we are back to square one - because if we are going to type the test's name anyways we could have just used:

:terminal mvn test -Dtest=ClinicServiceTests#shouldFindOwnersByLastName

So, why use Omnipytent? As you may have guessed from the subsection's title - completion! Omnipytent already gives you command mode completion for task names, and if you want to you can easily define completions for the task arguments.

If I were to create a generic completion plugin for Java tests, I would need to make it super-robust to account for the different styles and conventions. Maybe even run Maven/Ant/Gradle with some injected target that emits them. But here I just need them for one specific project - so I don't need to put all that effort, and can just depend on the characteristics of the tests:

They are all inside src/test/java.
They are all void methods.
They all have @Test in the line above them.

So - all I have to do is grep for the @Test lines, get the lines after them, and extract the filename and the method name from those lines. I can quickly write something like this:

import re
pattern = re.compile(r'(\w+)\.java-.*void (\w+)')
for line in local['rg']['-e', '@Test']['--after-context', 1]['src/test/java']().splitlines():
    m = pattern.search(line)
    if m:
        class_name, test_name = m.groups()
        yield '%s#%s' % (class_name, test_name)

This is definitely not plugin-grade - but for a personal search snippet just for me and just for this project it's acceptable. And now all that's left is to make it the completion function for test_specific:

@task
def test_specific(ctx, testname):
    mvn['test']['-Dtest=' + testname] & TERMINAL_PANEL


@test_specific.complete
def test_specific__completion(ctx):
    import re
    pattern = re.compile(r'(\w+)\.java-.*void (\w+)')
    for line in local['rg']['-e', '@Test']['--after-context', 1]['src/test/java']().splitlines():
        m = pattern.search(line)
        if m:
            class_name, test_name = m.groups()
            yield '%s#%s' % (class_name, test_name)

Interacting with Vim

Running tests is nice - but we also want to build the project, don't we? "use :make" - a thousand Vim users would scream at once(if... they were reading this at the same time). OK, let's use :make:

Oh, right - my &makeprg is set to Gradle for Java files, and this is a Maven project. Well - I don't want to change my .vimrc to use Maven - so let's use an Omnipytent task!

I run :OPedit compile to scaffold the compile task, and write this:

@task
def compile(ctx):
    with OPT.changed(makeprg='mvn', errorformat=r'[ERROR] %f:[%l\,%v] %m'):
        CMD.make.bang('compile')

What's going on here?

OPT - the helper for setting Vim options. We could use OPT.makeprg or OPT['makeprg'] to get and set the&makeprg` option.
OPT.changed(...) a context manager for temporarily changing the values of some Vim option. In this case - &makeprg and &errorformat.
CMD - the helper for running Vim commands.
CMD.make - Vim's :make command - can be used like a function.
CMD.make.bang - this is :make!(because I don't like to get jumped to the first error)

All together - when we run the compile task, it'll temporarily set &makeprg and &errorformat, run :make! compile, and then set &makeprg and &errorformat back. This will result with executing mvn compile and running it's output through the proper error format to populate the quickfix list:

Of course - instead of CMD.make you can use :make alternatives - e.g. you can install dispatch.vim and use CMD.Make. Or you can use CMD to do other things, unrelated to building the project...

Extensions

Writing the error format in each tasks file makes little sense. Chances are I'll use the same error format in many different projects. Same thing may be true for other things we define in our tasks files.

To allow easy reuse of such things, Omnipytent supports an extension mechanism. A plugin can put a Python source file under it's omnipytent/ directory, and it'll become a child module of omnipytent.ext. For example, my MakeCFG plugin exposes such interface - a makecfg function for setting &makeprg and &errorformat for entries in it's database.

So - if I have MakeCFG installed, I can write my compile task like this:

@task
def compile(ctx):
    from omnipytent.ext.makecfg import makecfg
    with makecfg('mvn'):
        CMD.make.bang('compile')

Combining tasks together

During development we often want to interact with the application we are working on. The one we chose is a web application, so we will want to run it, send commands to it, and stop it. Omnipytent can automate that as well!

Let's start with running. The README says we need to use ./mvnw spring-boot:run - so let's write a task to run it in a Vim terminal:

@task.window
def launch(ctx):
    mvn['spring-boot:run'] & TERMINAL_PANEL

Noticed anything new? Instead of @task I've used @task.window. This creates a special type of test called window task. Inside window tasks you can create new Vim windows which can be used in other tasks(we'll see that later). On it's own, it acts like a normal task - expect:

If you go to a different window during that task(you are expected to create one), when the window task is over you will be moved back to the window where you started.
If you run the task when the window it created last time is still open - it will be closed before the task runs.

OK - so we can start the server whenever we want, and we will only have one running at a time. But what about when we don't need it? Do we have to kill it manually? No - we write a task:

@task(launch)
def kill(ctx):
    from omnipytent.util import other_windows
    with other_windows(ctx.dep.launch):
        CMD.bdelete.bang()

There are several new things here:

@task gets an argument - launch! This makes the launch task a dependency of the kill task - so it will be invoked whenever we call kill.
ctx.dep.launch - that weird ctx argument we always had in our tasks is the task's execution context - it provides methods for interacting with Omnipytent itself, and is useful when we want to combine tasks -like we do now. ctx.dep is the access point for Python objects passed to us from dependencies - in this case, because launch is a window task it automatically passes the window object(:help python-window) it created.
other_windows is a context manager which allows us to travel to other windows and promises to return us to where we started. It also accepts a window object argument, and when it does it brings us to that window - so we can do stuff in it.

Window tasks have a special behavior when used as dependencies - when the window they were supposed to create already exists, they don't execute and instead pass the same thing they passed before. So when we call kill, launch will pass to it the window that the previous launch task created. We then go to that window with other_windows and delete that buffer to terminate the program and close the window:

OK - so we have our server running - how do we use it? This server accepts JSON requests for finding a vet - let't create something to query it:

@task.window
def queries_terminal(ctx):
    shell = local['sh'] & TERMINAL_PANEL.vert.size(50)
    ctx.pass_data(shell)


@task(queries_terminal)
def find_vet(ctx, name):
    import json
    name = json.dumps(name)
    cmd = local['curl']['-s']
    cmd = cmd['localhost:8080/vets.json']
    cmd = cmd | local['jq']['.[] | map(select(.firstName == %s))' % name]
    cmd & ctx.dep.queries_terminal

OK... this is starting to get complex. queries_terminal creates a terminal we can use to run our queries with curl. To make the results easier to read, it makes it a vertical terminal this time(.vert) and sets it to 50 columns(.size(50)). The it calls ctx.pass_data with the result of the terminal-opening command? What's going on here?

The result of a terminal opening is a shell command executor you can use for interacting with the terminal. We then use ctx.pass_data to pass it to dependent tasks. A window task will automatically pass the window - but in this case we want to pass the terminal handler so that dependent tasks will be able to run things in it.

Which leads us to find_vet, that constructs a curl&jq command to find a vet with a given name, and executes this command using ctx.dep.queries_terminal - the shell command executor we got from queries_terminal.

Let's see it in action(I've moved the server's terminal to a tab because screen real estate):

Easy!

JSON is nice, but apparently PetClinic also supports XML. What if we want to tinker with both? We can make it an argument, or duplicate the task, or... use an options task!

@task.options
def query_format(ctx):
    json = dict(suffix='.json',
                filter=lambda name: local['jq']['.[] | map(select(.firstName == %s))' % name])
    xml = dict(suffix='.xml',
               filter=lambda name: local['xmllint']['--xpath', '//vetList[firstName=%s]' % name, '-']
               | local['xmllint']['--format', '-'])


@task(queries_terminal, query_format)
def find_vet(ctx, name):
    import json
    name = json.dumps(name)
    cmd = local['curl']['-s']
    cmd = cmd['localhost:8080/vets' + ctx.dep.query_format['suffix']]
    cmd = cmd | ctx.dep.query_format['filter'](name)
    cmd & ctx.dep.queries_terminal

WHOA! What's that? Don't be alarmed - most of it are just shell pipes stuff for filtering the data. Let's focus on the main new thing - @task.options. This creates an options task - a task used for choosing an option. This task uses a weird syntax - every local variable it creates is an option. In this case - xml and json.

If you run find_vet without picking an option first, Omnipytent will prompt you to pick one. After that it'll remember your choice - but you may change it by invoking query_format directly(with an argument to pick the option or one to get prompted).

Conclusion

If you know some basic Vimscript, you could have created commands for all the things I demonstrated. But... you probably wouldn't. Too much hassle for things you can just type in the terminal. And even if you would, you wouldn't go the extra mile to add completion and choice-cache. Too much work for something you can only use in one project...

Omnipytent's power is not in allowing you to do things - it's in making these things more accessible. When adding tasks is so easy(just :OPedit <task-name> and code it in Python), they suddenly worth the effort - even if you are only going to run a task a few times.

DEV Community: Idan Arye

Moonicipal Explained

Why a new plugin?

Using Moonicipal

Getting input

Caching choices

Cached buffers and Channelot terminals

Data cells and BuffLS

Conclusion

Is using an executable to help client libraries connect to a server a good architecture?

The Problem

The Solution

The Drawback

Need help with terminology for humans that follow instructions from a script

Analogical naming - yay or nay?

Are there any API specification formats for WebSockets?

Omnipytent 1.3.0: Async Tasks and Selection UIs

Omnipytent Explained

Idan Arye ・ Dec 19 '17

The problem: single-threaded, event loop based UI

The idea: generator based async tasks

INPUT_BUFFER - basic async user input

CHOOSE - the power of selection UIs

Multi selection and generator options tasks - an actually motivational example

Conclusion

Don't initialize your variables

OK, but that's C. What about more modern languages?

WRONG!

This is still initialization

Conclusion

Abusing Type Erasure: Passing Back the Same Subclass the Object Yields

The problem

Generics to the rescue?

The solution

Omnipytent Explained

The problem: running commands

So... map some keys?

Just use the command line history like everyone else

You spoiled brat! Put these commands in .sh files and get it over with!

Have a single file containing all these commands

Using Omnipytent

Simple tasks and commands

OK... but why?

Task arguments and completion

Interacting with Vim

Extensions

Combining tasks together

Conclusion

Followup:

Omnipytent 1.3.0: Async Tasks and Selection UIs

Idan Arye ・ Nov 15 '18

`INPUT_BUFFER` - basic async user input

`CHOOSE` - the power of selection UIs

You spoiled brat! Put these commands in `.sh` files and get it over with!