DEV Community

Jazzy JJ
Jazzy JJ

Posted on

I audited marshmallow's docs: 184 missing docstrings, 65 stale. Here's the tool I built to fix that

#python #documentation #devtools #legacycode

Most teams have one repo nobody wants to touch. It works, it matters, and the people who understood it have moved on. Half the functions have no docstring, and the ones that do often describe a signature that changed years ago.

I built Legacy Doc-AI to deal with that repo. It's a small CLI that does two jobs: it tells you exactly where the documentation is missing or out of date, and it drafts docstrings and a README for you to review.

What it found in marshmallow

To test it on real code, I pointed it at the source of marshmallow, a well-maintained, widely used Python library. legacy-doc-ai check reported:

Metric Count
Files analysed 14
Functions and methods 236
Classes 63
Functions with a docstring 59 of 236
Missing docstrings (functions + classes) 184
Stale docstrings 65

"Stale" here means something specific and checkable: the function takes a parameter that its docstring never mentions. All 65 flags are of that kind. If a well-kept library has gaps like this, your 2014 internal service probably has more.

The full report is here: sample audit of marshmallow.

How it works

  1. Parse, don't guess. It walks the code's syntax tree (Python's ast) to list every function, class, parameter and existing docstring. That's what powers the coverage numbers and the stale-parameter check: no AI involved, so those numbers are exact.
  2. Draft with a language model. For each undocumented function it sends the function's code to a model and asks for a docstring in a consistent style. It works with OpenAI or Anthropic models, or any OpenAI-compatible endpoint, including a model running on your own machine, so the code doesn't have to leave it.
  3. Review before anything changes. --dry-run prints every proposed docstring without touching a file.

A real before and after

This is from the marshmallow run, using a small local model (Qwen2.5-3B), copied verbatim from the log.

Before:

def from_timestamp(value: typing.Any) -> dt.datetime:
    if value is True or value is False:
        raise ValueError("Not a valid POSIX timestamp")
    value = float(value)
    if value < 0:
        raise ValueError("Not a valid POSIX timestamp")
    ...
Enter fullscreen mode Exit fullscreen mode

After:

def from_timestamp(value: typing.Any) -> dt.datetime:
    """
    Convert a value to a datetime object based on a POSIX timestamp.

    Args:
        value (Any): The value to convert. If a boolean, raises a ValueError.

    Returns:
        datetime.datetime: The datetime object corresponding to the timestamp.

    Raises:
        ValueError: If the value is a boolean, or if the value is less than 0.
        ValueError: If there is an overflow error during conversion.
        ValueError: If there is an OSError during conversion.
    """
Enter fullscreen mode Exit fullscreen mode

It picked up the edge cases from the code itself: booleans, negatives, overflow. One honest caveat: a 3B model occasionally rewrites a type name (dt.datetime became datetime.datetime in the text), which is exactly why every change goes through review.

The commands 

legacy-doc-ai check path/to/repo                  # coverage + stale docstrings
legacy-doc-ai docstrings --dry-run path/to/repo   # proposed docstrings, nothing written
legacy-doc-ai readme --dry-run path/to/repo       # a README drafted from the code
Enter fullscreen mode Exit fullscreen mode

Why not Sphinx, Doxygen or Copilot?

  • Sphinx and Doxygen publish the docstrings you already have. With 184 missing, there isn't much to publish.
  • Copilot will write a docstring for the function in front of you. It won't tell you which 184 are missing or which 65 have drifted.

Legacy Doc-AI is the step before those tools: find the gaps across the whole repo, fill them, and keep them from drifting.

Try it on your repo

I'm running free audits while I get this in front of people. Send me a public repo URL and I'll send back the same report you saw for marshmallow, plus sample docstrings for a few of its worst-documented functions.

After the free first run it's £39 per repo per month. I'd especially like to hear from anyone looking after a codebase older than its current team.

Top comments (0)