Omnipytent 1.3.0: Async Tasks and Selection UIs

If you don't know what Omnipytent is, read this first:

Omnipytent Explained

Idan Arye ・ Dec 19 '17

#vim #workflow #automation

Version 1.3.0 of Omnipytent introduces a new concept - async tasks. In this post I'll try to explain what are async tasks and what are they good for.

The problem: single-threaded, event loop based UI

(you can skip this section if you already understand why we need async tasks)

Vim has a single-threaded architecture. When you perform an operation (by typing a key, running a run-mode command, using :autocmd etc.) that operation takes over the thread, and nothing else can get updated. The operation may receive input from the user (e.g. with input() or getchar()) or update the TUI (e.g. with :echo or by running a shell command with :!) but Vim's event loop itself is stuck until the operation is finished. You can't run other commands, which means that there is nothing to update the TUI, but even the jobs and terminals can't read anything from their streams while that operation is running.

Your everyday Vim operations are quick and "atomic" enough for this to not be a problem. Vim can't do anything else when you type w twe needo jump to the next word, but this happens so fast that it doesn't matter, and you don't need Vim do do anything else during. If you are running shell commands, on the other end, they can take quite long and you'd have to patiently wait for them to finish. Luckily, Neovim and Vim8 have jobs and terminals, so you can just launch it and it runs in the background.

But what if you need the "result" of the command you ran in the terminal? fzf, for example, is running fzf in a terminal, and use the result to open the chosen file. This can't be done in a single Vim function invocation - it needs in one function to start the terminal, then yield executing back to the event loop, and once fzf exist run another function to deal with the result. If it was done in the same function that launched the terminal, Vim would be unable to update the terminal buffer and the user wouldn't be able to use it.

And it's not just shell commands - sometimes you just need to user to use Vim's UI itself. A good example is fugitive.vim's :Gcommit command. It opens a new buffer for the user to type the commit message in, and when they save and close that buffer the plugin creates a new Git commit with that commit message. Until the function called by :Gcommit finishes it hogs the event loop and the user can't write the commit message nor can they save and close the window, so :Gcommit must terminate before that and another function must be called when the window is close to finish the process.

Plugins like fugitive or the one bundled with fzf are registering callbacks to implement that behavior. It's not that hard to do, but it is quite cumbersome (even more than in JS!) and requires some familiarity with Vimscript and Vim's architecture. Omnipytent tasks are supposed to be simple and streamlines, so this callback registration is too much - we need something simpler. And that's where async tasks come in.

The idea: generator based async tasks

(you can skip this section if you don't care how they work, and go directly to the next sections for examples on how to use them)

A generator function in Python is a function that has at least one yield expression. If you want to learn more about it read the Python docs - for our purpose it's enough to mention that the yield yields both a value and the execution itself - so if a callee yields, the execution goes back to the caller which can chose, at a later point, to resume the callee from where it yielded.

This was used for async IO before the async and await keywords were introduced in Python 3.5, and since Omnipytent wants to support older versions it uses yield for its async tasks.

An async task looks something like this:

@task
def my_async_task(ctx):
    do_something()
    result = yield ASYNC_COMMAND()
    followup(result)

When the task yields, ASYNC_COMMAND() does three things:

• It prepares of the command (e.g. open a window or start a terminal)
• It registers itself in Omnipytent.
• It registers a callback/:autocmd in Vim to resume itself once done.

After that - :OP my_async_task terminates and control goes back to Vim's event loop. But my_async_task itself is not terminated yet - at some point it will be resumed and the task will continue and do the followup.

And of course - you can yield another async command later in the task.

Omnipytent comes bundled in with some useful async commands. If you need to create your own - refer to :help omnipytent-creating-AsyncCommand.

INPUT_BUFFER - basic async user input

For the example, I'll use the same example project from the first post - Spring's example pet clinic web application. Lets say we want a task for adding animal owners. The API is simple - a POST request with the details - but how will we prompt the user (which is us) to enter the fields?

Up until now, we'd have to use input() to allow the user to enter that data. With async tasks, we can do better:

import requests, yaml

OWNER_FIELDS = ('firstName', 'lastName', 'address', 'city', 'telephone')


@task
def add_owner(ctx):
    empty_form = '\n'.join('%s: ' % field for field in OWNER_FIELDS)
    filled_form = yield INPUT_BUFFER(text=empty_form, filetype='yaml')
    parsed_form = yaml.load('\n'.join(filled_form))
    requests.post('http://localhost:8080/owners/new', data=parsed_form)

The important line is the second line of the task function - the one where we do the yield. INPUT_BUFFER is one of the async commands bundled with Omnipytent, and it opens a new window with a buffer for the user to edit. We set the original text - a YAMLish form with blank fields for the user to fill - and also set the file type to YAML to get some nice coloring. Then we yield this command object - and the execution control returns to Vim's event loop. Now the user can fill the form, and when they close it Omnipytent kicks back in and the task resumes - with filled_form set to the lines of the buffer the user filled.

This is how it looks in action:

CHOOSE - the power of selection UIs

The primary motivation behind async tasks was to support selection UIs. These are usually fuzzy matchers like fzf or Unite, but I use the term Selection UIs because this mechanism is not limited to tools that allow fuzzy search - any tool that provides a TUI for selecting from a list can fit.

Tests can accept arguments, but it's not always convenient to type the arguments. When the arguments are long, or complex, or hard to remember, and when there is an easy way to programmatically generate the list of possibilities, it is far more convenient for the user to filter and pick what they want with a selection UI.

For my next trick I'll need to add a missing method to the example project:

// src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java

    @GetMapping("/owners.json")
    public @ResponseBody List<Map<String, Object>> showResourcesVetList() {
        return this.owners.findByLastName("").stream().map(owner -> {
            Map<String, Object> entry = new HashMap<>();
            entry.put("id", owner.getId());
            entry.put("firstName", owner.getFirstName());
            entry.put("lastName", owner.getLastName());
            entry.put("address", owner.getAddress());
            entry.put("city", owner.getCity());
            entry.put("telephone", owner.getTelephone());
            return entry;
        }).collect(Collectors.toList());
    }

All it does is add a new path - /owners.json - that generates a JSON of all the owners. Nothing fancy, but the orig example project only supported getting them as an HTML page, which is harder to parse (unless you use a regex)

We want to write an Omnipytent task that reads that list with a GET request, lets the user pick one of the owners, edit it, and update it with a POST request. To do the selection, we are going to use another async command - CHOOSE:

import json


@task
def edit_owner(ctx):
    entries = json.loads(requests.get('http://localhost:8080/owners.json').text)
    entry = yield CHOOSE(
        entries,
        fmt='{firstName} {lastName}'.format_map,
        preview=lambda entry: yaml.dump(entry, default_flow_style=False))

    owner_id = entry.pop('id')
    orig_form = yaml.dump(entry, default_flow_style=False)
    edited_form = yield INPUT_BUFFER(text=orig_form, filetype='yaml')
    parsed_form = yaml.load('\n'.join(edited_form))

    edit_url = 'http://localhost:8080/owners/%s/edit' % owner_id
    requests.post(edit_url, data=parsed_form)

CHOOSE runs whatever selection UI installed in your Vim. It first checks for fzf, then Denite, Unite, CtrlP, and finally - if none of the above is available, it uses an inputlist() based selection UI. Or - if you have other preferences - you can set the selection UI with g:omnipytent_selectionUI.

Other than the list of options, we pass two more arguments to CHOOSE:

• fmt: The options are dicts but the selection UIs pick from lines. This argument is a function that formats each option into a line.
• preview: fzf, Denite and Unite support a preview of the items, and this argument is a function for rendering that preview.

After we yield CHOOSE, we get back the picked option and use it to display an INPUT_BUFFER - which I already explained earlier.

Let's see it at work:

Multi selection and generator options tasks - an actually motivational example

The previous examples were a nice way to show the power of async tasks, but they are not really something you'd write Omnipytent tasks for. If you want to actually add and edit owners you'd use the web application, and if you need a quick way to run these while developing to test your code, editing the text buffer each and every time is a bit cumbersome.

So how about something you actually want to create a task for? How about... running tests?

Running tests is a great use case for CHOOSE. When I work on a piece of code I often want to run a test that checks it. I don't want to run all the tests, because it'll take too long and will generate too much output. I want to run just this specific test and see just its output - which will reflect the changes I did to the code.

Test names are often long - they need to encode the name of the class/module that contains the test, the name of the test function itself, and sometimes the parametrization. Writing the full name each time is cumbersome. Writing that name inside the test makes it easier to run, but now we have to edit the tasks file when we want another test - and figure the name of that test. Not that hard, but can now do better - we now have CHOOSE!

Decent project management tools usually have a command for listing all the tests, but this is Maven so we'll have to parse the files ourselves:

import re


@task
def run_tests(ctx):
    pattern = re.compile(r'@Test\n\s*public void (\w+)\(')
    test_names = []
    for path in local.path('src/test').walk(lambda p: p.basename.endswith('.java')):
        for match in pattern.finditer(path.read('utf8')):
            test_names.append('#'.join((path.stem, match.group(1))))

    chosen_tests = yield CHOOSE(test_names, multi=True)

    cmd = local['mvn']['test']
    cmd = cmd['-Dtest=' + ','.join(chosen_tests), 'test']
    cmd & TERMINAL_PANEL

Note that there is a new argument to CHOOSE: multi=True. As you may have guessed from the name, it allows the user to select multiple options. Only fzf, Denite and Unite support this, but even with CtrlP and inputlist() it'll still return a list to keep some uniformity in the task.

We then join the chosen tests, and voila!

But... we still need to pick the test we want to run each time. Picking it with fzf is definitely better than typing it, but since we usually want to run the same test(s) many times when we work on the same area of the code, it could be nice if Omnipytent could remember our last choice.

Well - it can. Omnipytent already had @task.options that remembers the user's choice, but you could only pick one option and the option keys had to be hard-coded as local variables. Omnipytent 1.3.0 solves both these problems:

1. @task.options is now based on CHOOSE - so it can use more elaborate selection UIs. A new variant - @task.options_multi - allows you to pick multiple options. If you are using CtrlP or inputlist() and still want multiple choices you'll have to pass them as arguments. Or just upgrade to fzf/Denite/Unite.
2. If the task function is a generator, instead of using the local variables as options it uses the yielded values as options.

This means we can split our run_tests into two tasks: pick_tests and run_tests. pick_tests will always prompt us to choose the tests, but run_tests will remember our last choice:

@task.options_multi
def pick_tests(ctx):
    ctx.key(str)
    pattern = re.compile(r'@Test\n\s*public void (\w+)\(')
    for path in local.path('src/test').walk(lambda p: p.basename.endswith('.java')):
        for match in pattern.finditer(path.read('utf8')):
            yield '#'.join((path.stem, match.group(1)))

@task(pick_tests)
def run_tests(ctx):
    cmd = local['mvn']['test']
    cmd = cmd['-Dtest=' + ','.join(ctx.dep.pick_tests), 'test']
    cmd & TERMINAL_PANEL

Note the first line of pick_tests: ctx.key(str). Because the yielded options can be objects, we need a string keys of them, and ctx.key sets the function for picking these keys. The key must be deterministic, because these keys will be used to cache the choice. There is also ctx.preview for setting a preview function, but we don't need one here.

And here is how it works:

Conclusion

Omnipytent's goal was to allow micro-automation of simple project tasks. Async tasks allow you to add better UI to that automation, farther enhancing the power at your fingertips.