Doctests, the shy giant of testing modules

Ned Boromir Stark agrees

Do you use python, even to wash your clothes? Do you find unit testing boring, but still have to do it, because you find value in automated testing? Then this article is for you.

The idea

I believe you have used the python console, from time to time. Lets assume you are writing a few inline functions like below, to experiment with stuff:

$ python
Python 3.6.4 |Anaconda custom (64-bit)| (default, Jan 16 2018, 18:10:19) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def addme(a):
...     return a + a
... 
>>> addme(2)
4
>>> addme(1.9)
3.8
>>> addme(0)
0
>>> 

So, you have written an inline function, and you have made a couple test runs on it, to do some basic verifications. What if python could read the above output and do the reasoning for you, at runtime? That’s the idea behind doctests, my friend.

Seriously?

Calm down Patrick

Totally, python has introduced a lot of intuitive and…heartbreaking features from time to time, which are now mainstream to a few main languages. Why not this one too?

Benefits of doctests

The benefits are quite a few and not limited to:

• They are ridiculously easy to write. I mean, you can even outsource it to your younger cousin, who studies about the anatomy of soil atoms, because he owes you, for fixing his computer.
• Unless you find copy paste difficult, they are more joyful to write. This time you have to familiar with a terminal environment, though.
• There is no need to open another file(even though you can put all doctests of your app to a single file) to read the test code for any function, as they lie right under the signature of each function.
• They are executable with the doctest module and readable without knowing a bit of python.

What about unit tests?

Unit tests, still offer great value and advanced capabilities of testing. While doctests are great on validating simple and pure functions, they are not very good, when you have to do complex validation. For example, if a sequence of commands was called during the test.

What about mocking?

Doctests can handle mocking gracefully, with a great library called Minimock. I encourage you, to give it a try and let me know your thoughts.

Even though I like the initiative, I prefer my tests to have separate roles. I don’t want my doctests to be heavily loaded and one-size-fits-them-all. But that’s really a matter of taste, there is nothing wrong if you disagree and I am more than happy to hear your rationale , if so.

A working example

Talk is cheap, so let’s write some code. Below is a working example, using the function of the prompt above.

addme(a):
    """
    This is a docstring, usually to explain the use of a function. Please do not confuse it with doctests. They both mean to provide forms of documentation, but doctests are executable too.
    >>> addme(4)
    8
    >>> addme('a')
    'aa'
    >>> addme(set())
    Traceback (most recent call last):
        ...
    TypeError: unsupported operand type(s) for +: 'set' and 'set'
    """
    return a + a

if __name__ == '__main__':
    import doctest
    doctest.testmod()
    print(addme(1))

There are a couple of points, worth noticing here:

• The doctests, need to live inside a docstring. In there, just ‘copy-paste’ the output of your python console quick tests.
• If you need to simulate an exception case, you only need to add the first and the last lines of the exception message. Did you expect to hardcode paths or do any weird logic to get the file’s full path and print it in full? C’mon, python wont do that :)
• You need the doctest library to run the tests. Nose doesn’t need it though.
• You can run the above, as usual, with python mydoctests.py

I need my tests to run as part of CI/CD/CT cycle. What’s in for me?

I am not here to disappoint you, am I ? :)

The nose test runner, supports running all your doctests additionally to your unit tests. Just add the flag --with-doctest and you are good to go.

Nice stuff, where to go from here?

• Read the python documentation for more twitches and cases.
• Read this fantastic book and especially chapter 4, which covers doctests.
• Check implementations of other languages. For example, the one for NodeJS
• Doctests and nose

Thank you for reading this article, I hope you enjoyed it and that the article triggered your interest in writing more and less painful tests. Please spread the love, with the buttons below, if so. Feel free to add your own thoughts and experiences with doctests too.

Originally published at https://perigk.github.io

Comments

[Karl Gutwin]

I love doctests! Here’s my extra few tips:

• Treat doctests more as executable documentation rather than as just more unit tests. Sometimes, with all the mocks and setup needed to do a good unit test, the doctest can look really ugly and unreadable; you should prioritize readability and clarity over comprehensiveness. A new developer should be able to read the doctest and have a good idea about what the function does.

• Watch out for dictionary outputs, which are not guaranteed to be printed in any particular order. There are a couple ways to avoid this; what I typically do is to test that the output dictionary is equal to a hard coded dictionary.
  >>> func() == {‘one’: 1, ‘two’: 2}

• Not every function warrants a doctest. Don’t bend over backwards trying to achieve 100% code coverage with your doctests. If that’s your project goal, use traditional unit tests to achieve most of your code coverage.

[Periklis Gkolias]

Thanks for the tips Karl, especially for the dicts, I havent thought to put a case for that.

Regarding coverage and mocks, you are right, they are meant to be mostly for pure functions and not to replace unit tests. But again, this is up to everyone, as with all tools.

[Adam Brandizzi]

Hey people, nice post and nice tips :)

Not every function warrants a doctest. Don’t bend over backwards trying to achieve 100% code coverage with your doctests. If that’s your project goal, use traditional unit tests to achieve most of your code coverage.

This is indeed the recommended approach. Yet, I'm doing something different and quite satisfying.

When I have a function not worth a test case, I'm writing doctests for it. This is perfect for internal functions: I do not pollute a module test suite with implementation details. It can also guide me on how to write the function (like TDD for smaller units) and the resulting function tends to be way better designed.

We all know those quite local pieces of code and thus never documented or tested. I always thought Doctest would be overkill there, but after experimenting, I feel it is right the opposite: doctests are the most lightweight way to document and test these units! And it proved worth the hassle: the docstrings have been very useful to explain how to use old code, rationales behind it, catch a lot of regressions and even improve the design. I wrote a post of how it works for me some time ago.

Of course, I'm not talking about the "right" way to doctest—quite the contrary, most doctesters work the other way around—but this is a possibility that a) helped me a lot and b) I'd like to see tried more.

[Periklis Gkolias]

When you say'a function not worth a test case', what makes it need a doctest case?

[Adam Brandizzi]

Well, no function needs a doctest but, in my experience, it is useful to put a doctest in every function. Even the smallest private functions. For three reasons:

1. We get documentation. With a doctest, we have a basic explanation of how we expect the function to be used, naturally. Even if the function is not published, it will be helpful when you're maintaining code that uses the function. And it helps a lot to reuse the code.

2. We get some tests. The most basic behaviors of the function are now protected against unintended changes. If we have to handle a new corner case, we can add it to the doctest as well and ensure it will be preserved.

3. And finally, the least obvious one: we get a better function. If we force ourselves to doctest a function, we find ourselves working to make it more documentable. Some lazy decisions we could take (relying on globals, mixing I/O and processing etc.) are not that convenient anymore. Our function got more and better seams, is easier to call and is more predictable.

Again, this is my experience after trying an unorthodox approach. Would you try it to be if it is true? :)

[Daniel Schep]

A huge benefit of doctests is it allows your examples in documentation to be tested so that you know it's always update and correct.