Testing a Vim Plugin I Can't Test: Using Vim as Its Own Oracle

We built ttt, a terminal text editor, as an alternative to VS Code and Zed for the terminal. Not Neovim. Not Helix. A GUI-style editor that happens to live in your terminal. Arrow keys, Ctrl+S, click to place your cursor. The whole idea was that you shouldn't need to learn a new paradigm to use it.

Then a community member opened issue #386:

"Please add an optional Vim keybinding mode. I love TTT's UI and review experience but would like hjkl, gg, G, motions, and modal editing."

And later:

"I am a big nvim user so different keybindings is just weird for me. After this is implemented, from using TTT occasionally, now I will use it every day as my daily driver."

Fair enough. We had a plugin system. We had key interception. We could build a Vim layer as a Lua plugin without touching the core editor. So we did.

The Problem: I Don't Use Vim

Here is the thing about building a Vim emulation layer: Vim's behavior is the spec, and the spec is enormous. Motions compose with operators. Operators compose with text objects. Counts multiply. Registers prefix everything. Each combination has edge cases that only surface in specific cursor positions on specific buffer shapes.

We started with ibhagwan's vim-cheatsheet as a coverage checklist and wrote unit tests for every motion, operator, and text object listed. Cursor at position X, press keys Y, assert cursor lands at position Z. Buffer contains A, apply operator B, assert buffer becomes C.

That covered the obvious cases. But Vim has thousands of interactions between features, and I don't have the muscle memory to know which ones matter. I can't sit down and manually verify that d; after fa deletes the right range, because I've never used that sequence in my life. I don't know what "right" looks like.

The Oracle

If you can't test by hand, and you can't enumerate every case, you randomize. But random testing needs an oracle: something that tells you what the correct answer is.

We already had one. Vim itself.

The fuzzer works like this:

1. Generate a random sequence of 6 to 10 complete Vim actions using a seeded PRNG. Each action starts and ends in normal mode. The generator mixes bare motions (26%), operator+motion combinations (30%), simple edits like x, D, ~, J (15%), insert mode entries (3%), and undo/redo (4%). Counts and register prefixes get sprinkled in probabilistically.

2. Replay the exact same sequence against two targets: real Vim (vim -u NONE -N) and our plugin running inside ttt. Both start from the same fixture text.

3. Compare the saved files byte-for-byte. If they diverge, the fuzzer dumps a JSON report with the seed, the exact key sequence, and a diff.

The batch runner sweeps 200 seeds by default. Each run takes a few seconds. You get a list of seeds that produce divergences, and each seed is a reproducible test case.

Some commands are deliberately excluded from the alphabet. G, H, M, L depend on viewport size, which differs between the two runners. Bracket-matching text objects (i[, a{) are excluded because Vim's forward-scanning behavior is hard to replicate exactly. Search and ex commands have their own test paths. The fuzzer focuses on the core motion/operator/text-object composition matrix, which is where the surprising bugs live.

What the Fuzzer Found

The first run across 200 seeds caught six bugs. Every one of them was a compositional edge case that our hand-written unit tests had missed.

3D truncated instead of deleting lines. D with no count means "delete to end of line." 3D means "delete to end of line, then delete the next 2 lines." Our plugin was joining lines instead. The semantics of D change between count=1 (truncate) and count>1 (linewise delete), and we had only tested the count=1 case.

d; inherited the wrong inclusivity. ; repeats the last f/F/t/T motion. When used inside an operator (d;), it must carry the original find's direction and whether the motion is inclusive or exclusive. We were losing that state when threading through the operator-pending parser. A unit test for d; would have passed with a carefully chosen cursor position, but the fuzzer found a position where the off-by-one mattered.

dl at end of line was a no-op. dl at the last character of a line needs to use an inclusive delete to remove that character. The inclusive/exclusive distinction for character motions at line boundaries is the kind of Vim subtlety that only surfaces in specific positions. Our unit tests all had the cursor in the middle of the line.

dj on the last line did nothing. dj is a linewise motion: delete the current line and the one below. When there's no line below, Vim still deletes the current line. We were treating the out-of-bounds motion as a no-op and skipping the operator entirely. Same pattern for dk on the first line.

cw at end of line hung. cw at the last character should delete to end of line and enter insert mode. Our word-forward motion was returning the same position when already at EOL, so the operator had nothing to act on.

aw on whitespace extended wrong. When the cursor is on whitespace between words, aw (a-word) should select the whitespace plus the next word. We were extending in the wrong direction.

None of these bugs would have been caught by a developer who uses Vim daily, because no one types 3D on line 4 of a 5-line buffer with the cursor at column 7. But the fuzzer does. And when it finds a divergence, the seed is a permanent regression test.

The Loop

The workflow became:

1. Run the fuzzer.
2. Pick the first failing seed.
3. Replay it manually to understand the divergence.
4. Fix the bug in the plugin.
5. Run the fuzzer again to confirm the fix and check for regressions.
6. Repeat until all 200 seeds pass.

After fixing the initial six bugs, the fuzzer ran clean. We bumped it to 500 seeds, still clean. The surviving edge cases are all in excluded categories (viewport-dependent motions, forward-scanning text objects) that we handle through separate, targeted tests.

The Takeaway

If you're building an emulation layer for anything with complex interactive behavior, the original software is your best test oracle. You don't need to understand every edge case yourself. You need to generate inputs, run them through both systems, and diff the outputs.

The fuzzer for ttt-vim is about 400 lines of JavaScript. It found bugs that would have taken months to surface through user reports. And every bug it finds comes with a reproducer for free.

The Vim plugin ships as a single 4500-line Lua file. The editor doesn't know or care that Vim mode exists. It just sees a plugin that intercepts keys and calls the editor API. And thanks to the fuzzer, we have reasonable confidence that when a Vim user types a command, the result matches what they expect, even though the person who wrote the code has never used Vim as their primary editor.