Doctave CLI 0.2.0: A Benchmarking Story

The Doctave CLI is a free to use open source documentation generator. It takes your Markdown files and converts them into a beautiful documentation site. Today we have released version 0.2.0, which brings some cosmetic improvements and a fully in-memory development server for local development. We'll also talk about how this surprisingly did not make the CLI faster like we expected.

Serving your docs from memory

Before Doctave would rewrite the whole documentation site to disk every time you made a change to your Markdown files. It would pretty much rm -r the site directory, used to house the generated HTML, and regenerate the docs site from scratch. Because Doctave (which is written in Rust) is able to generate the whole site in ~tens of milliseconds, we hadn't looked into optimizations here yet.

But there were two non-performance related issues that bothered us about this approach:

1. You would end up with a site directory that didn't really serve any purpose while developing your docs
2. While very minor in scale in our case, constantly writing and deleting small files from disk isn't something that modern SSDs like to do

This is why Doctave 0.2.0 never writes the HTML documentation site to disk until you run
doctave build --release. In serve mode when you write your documentation, the embedded web server serves them directly from memory.

$ doctave serve Doctave | Serve Starting development server...

Server running on http://0.0.0.0:4001/

    File docs/README.md updated.
    Site rebuilt in 31.698781ms

Wait a minute - shouldn't this be faster now that we are not writing files to disk?

Why is this not faster?

After completing this feature, I was surprised to see that Doctave wasn't generating sites noticeably faster than before. I was expecting a speedup due to us not spending as much time on IO. This was not happening.

After double checking to make sure the code was doing what I expected, it was time to benchmark things to see where the time was being spent. After some digging I found the answer: generating the search index.

Doctave comes with offline search built in. We use the elasticlunr-rs crate to generate a search index that is compatible with the elasticlunr.js library. You can see it in action by going to our docs (built with the CLI, naturally) and hitting the letter s on your keyboard to focus on the search bar. The searching happens entirely client-side.

It turned out that we are spending ~70% of the site generation time creating the search index. This is completely reasonable, as this is a somewhat CPU intensive task that parses the input files and generates the index. The time spent doing IO writing files to disk is completely negligible in comparison, and thus did not move the needle much at all. On top of that, the CLI was already writing files to disk in parallel. Moving things in-memory did not save us much time.

So while Doctave 0.2.0 is technically faster, this was not a big performance win. There is some more work we can do to parallelize the build process further - maybe in the next release. In the meantime, Doctave will be more friendly on your SSD and not pollute your workspace as much.

Still, this was a change for the better. It just wasn't the big performance win I suspected.

Conclusions

I think this was an instructive little story about performance and benchmarking that was worth sharing. It's often the case that when we make assumptions about the performance of a system that turn out to be wrong once you measure things. This is just another reminder: always measure.

Finally, do try out Doctave 0.2.0! If you're looking for a batteries-included documentation generator that doesn't require plugins, doesn't pollute your repository with loads of files, and doesn't need a specialized environment to run, Doctave may be a good choice for you. It also comes with Mermaid JS diagram support and dark mode!

You can host sites generated by Doctave on GitHub pages, or your favorite static site hosting provider. We are also building a specialized host for teams using Doctave on multiple projects - check it out if you're using Doctave at your organization.

Let me know if you're using it to document your project. I can be reached at [e-mail]. I'd be excited to hear what you think.

Want to try out Doctave?

If you're on Mac using Homebrew, you can install Doctave with the following command:

brew install doctave/doctave/doctave

If you're a Rust fan and want to use Cargo, you can do so too:

cargo install --git https://github.com/Doctave/doctave --tag 0.2.0

We also have prebuilt static binaries for Mac, Linux, and Windows.