Cleaning up metadata in a large Audiobookshelf library, one night at a time

Audiobookshelf is a great self-hosted audiobook and e-book server, as long as the files it indexes come with decent metadata. Mine mostly didn't. Years of collecting had left a large library (tens of thousands of items) where a good share of entries looked like this:

Title:   Anna.Author-The.Winter.House.SWEDISH.AUDIOBOOK.WEB-GRP
Author:  (empty)
Genres:  Speech
Description: Ripped by someone, enjoy.

That is the folder name as title, no author, a genre that is really an ID3 content type, and a description that is release-group graffiti. Multiply by thousands and the library is unbrowsable: no author pages, no series, search that only works if you remember the folder name.

This post is about the nightly job that fixed it, the rules that kept it from making things worse, and the four bugs it took to get right. Everything below is Python standard library plus public APIs; nothing exotic.

Rule one: fill only what is empty

The single most important design decision: never overwrite a field that has a value. Title and author are the exception for items that have no author at all (the release-name cases above, where the "title" is garbage by definition). Everything else (subtitle, ISBN, language, series and sequence, genres, narrators, publisher, year, description) is filled only when empty, or when it is certainly junk:

• descriptions matching known release-group signatures or shorter than a few words,
• genres that are ID3 leftovers (Speech, Other, F, (12)),
• the same "description" appearing on books by three or more different authors (a copy-pasted blurb).

Junk is cleared even when there is no replacement. Weak values (a one-word genre like Fiction) are replaced only when a matched record has something better.

Where the metadata comes from

Three sources, in this order:

1. The national library catalogue. In Sweden that is Libris, whose xsearch API is open, needs no key, and returns JSON or MARCXML. It knows every published edition with ISBN, publisher, year and, crucially for later, the original title of a translation (MARC field 240).
2. A store API with rich blurbs, series information, narrators and covers. Storytel's search endpoint turned out to be usable as-is for Swedish titles.
3. Audible through Audiobookshelf's own provider endpoint (/api/search/books?provider=audible) for English titles, and Google Books as a last resort: for descriptions and ISBNs, and for identifying a book nothing else found (more on its stricter rule below).

A query is just author + title. For items with a release name, the name is parsed into candidates: the Firstname.Lastname.Words.Of.Title form has no separator, so every split into 1-4 author words is tried, and the catalogue decides which one is right.

Rule two: a match must be clear

A record is accepted only when the author matches (same normalised name, or at least the same surname) and the title is close (normalised equality, prefix, or a similarity ratio above 0.85). Anything less is logged as a miss and left alone. Misses are retried after 30 days, because stores add titles all the time.

That rule rejected a lot of plausible-looking matches, which is exactly the point: an audiobook library with a thousand confident wrong matches is worse than one with a thousand gaps.

def close(a, b, need=0.85):
    a, b = norm_title(a), norm_title(b)
    if not a or not b:
        return False
    return a == b or (len(a) >= 5 and (a.startswith(b + ' ') or b.startswith(a + ' '))) \
        or difflib.SequenceMatcher(None, a, b).ratio() >= need

The prefix clause is there for subtitles: "The Winter House" should match "The Winter House: A Novel". It is also the loosest part of the rule. When Google Books became the last identification attempt, it happily matched one book in a children's series, say Harbour Town: The Dragon Summer, to another, Harbour Town: The End, because both titles start with the series name. The other sources rarely have that problem; Google's broad catalogue does. So a Google match must pass a stricter test: whole-title similarity of 0.85 or more, no prefix shortcut, and the same language as the release's own tag when Google states one. The looser the source, the stricter the rule.

Google Books has two more quirks worth knowing about. The free quota is 1,000 requests per day, counted per Pacific day, so a per-run limit is not enough: my daytime tests plus the night job went over it, and the rest of the night got nothing but "quota exceeded". The counter now lives in a small file shared by every run. And roughly half of all requests get a 503 Service temporarily unavailable at random, whatever the pacing, the headers or the query; the same request a few seconds later usually works. So each lookup gets two retries, and a book that only ever got 503s is not counted as "tried". It comes back the next day instead of in 30. It now runs as its own short daily job right after the quota resets, and identifies about one in ten of the books nothing else could. (The first day also showed why the language check matters: Google happily returns the Norwegian original of a Swedish translation, and my table of Google's language codes only knew Swedish and English, so "Norwegian" looked like "no language stated". A handful of books got Norwegian or Danish titles before I noticed; the plan file put them back.)

The night job and its parachute

The job runs from late evening to early morning, before the server's own scheduled scan. Four things make it safe to let loose on a library this size:

• A fresh native backup first. It triggers Audiobookshelf's own backup through the API, verifies that a new backup of plausible size exists, and refuses to start otherwise.
• A plan file per night. Every change is appended, with the previous values, to that night's JSON file before it is applied. --rollback <file> puts everything back, item by item. I have used it. More than once.
• A deadline fixed at start. The first version computed the deadline after the multi-minute backup; one night that rolled the deadline into the next day and the job ran into the morning scan.
• A weekly spot-check. Rollback only helps if someone notices. Once a week a separate script reads every change from the last seven nights and flags the patterns that have bitten me before: an author the folder name does not mention at all, a publisher in the author field, two people glued into one "name", a series name used as the author, a language that contradicts the release's own tag. Each flag is re-checked against the item as it is now, so problems that were already repaired don't count. The result is one push notification with counts and a few examples. The first run flagged a dozen or so out of well over ten thousand changes.

Throughput was a few thousand items a night, which meant the backlog took about a week. Since then it only touches new items and retries.

Bug one: the language tag that read "En" as English

Release names carry a language tag somewhere after the title: ...-SE-2020-GRP, ...SWEDISH.AUDIOBOOK..., ...-EN-.... The first regex took the first match. Swedish has a very common word spelled "En" ("a", "one"). So Firstname_Lastname_-_En_Gang_I_Byn-...-SE-2020 was tagged English, got an English ISBN, lost its Swedish one, and was looked up on Audible for good measure.

The fix is one word: take the last tag, because tags follow the title.

tags = re.findall(r'(?i)(?<=[._-])(swedish|english|danish|norwegian|se|en)(?=[._-])', name)
lang = {'se': 'Swedish', 'en': 'English'}.get(tags[-1].lower(), tags[-1].capitalize()) if tags else None

Bug two: the original title finds the original edition

Some Swedish releases are named after the book's original title: Author.Name.The.Winter.House.SWEDISH... for a translation sold as Vinterhuset. Searching the stores by that title finds the English edition: English ISBN, English blurb, English narrator, language "English". The library said Swedish, the metadata said otherwise, and a good four per cent of one night's matches were like that.

Two changes:

1. The release's own language tag wins. A matched edition in another language contributes only title, author and series; its ISBN, publisher, description and narrators are dropped. A record with no language but an English-range ISBN (978-0 / 978-1) counts as foreign too.
2. Then find the Swedish edition: a Libris MARCXML query restricted to Swedish records (spr:swe) where field 240, the original title, matches the title we have. That returns Vinterhuset with its Swedish ISBN and publisher; the store search with the Swedish title then finds the Swedish blurb and narrator.

Danish and Finnish ISBNs on Swedish audiobooks, by the way, are normal: several big audiobook publishers are registered there. The first version of my check flagged hundreds of those as errors. Check your assumptions against the data before you "fix" it.

Bug three: the author that was never right

Fill-only-when-empty has a blind spot: metadata that is wrong rather than missing. A scan found a surprising number of items whose stored author shared no name word with the author in the folder name. Some were the right book with a translator or narrator credited as author. Others were a different book entirely: a Swedish crime novel tagged as a German dictionary, a children's series carrying a biography's description.

So a second pass re-identifies those from the folder name alone, with two outcomes:

• Same book (old title close to the found one): the found author goes first, empty fields are filled, nothing is cleared. What happens to the old author name turned out to be bug four.
• Different book: the new identity replaces the old one, and fields the new record cannot fill are cleared, because they belonged to the other book.

Plus a guard: if the volume numbers disagree (Series 1 matched to Series 3), skip. The dry run caught that one.

The first batch of a few hundred was reviewed by hand before applying; it then joined the nightly job with its own state, so an item fixed by the first pass is not "done" for the second.

Bug four: "lossless" kept the wrong name

My first version of the same-book case kept the old author next to the found one. The reasoning sounded careful: the old name might be a co-author, so don't throw anything away. Lossless.

The first full night of it said otherwise. Of the few hundred books re-identified that night, roughly four in ten now listed two authors, and in most of them the second name was simply the leftover of the original wrong match: a Swedish crime novel credited to its real author and an American romance writer. A few dozen had a publisher as co-author. And Audiobookshelf keeps an item's existing author first when you add one, so the wrong name was the one you saw.

"Keep it, it might be right" is not lossless. It is keeping an error with a note attached. The rule now is that an old name stays only when something confirms it:

• the matched record lists that person as a creator (library records include illustrators and translators),
• the folder name contains the name, or
• a source finds this title under that name.

Publisher-like names never stay, and a found "author" that is really the series name (yes, that happened: a children's series whose title character was taken for the writer) is rejected. A one-off repair script re-planned every affected item from its pre-night values with the fixed rule and rewrote only the author field, with its own rollback file. Picture books came through with writer and illustrator intact, because the library record names both.

The same week, the opposite case: folder names that list two authors, where only one was found. A name like Anne_Author_Bea_Writer_-_Title cannot be split reliably, but the stores list every author of a book. So after a match, any other author from the record whose full name also appears in the folder name is added. A narrator or translator the folder doesn't mention is never added.

What it looks like now

Author pages exist. Series are grouped and numbered. Swedish books have Swedish titles and blurbs. Searching by a title you half remember works. The job still runs every night, mostly to catch new arrivals, and reports a one-line summary with the rollback command for that night.

If you take one thing from this: plan, then apply, and keep the previous values next to every change. The rest is regex and patience.

Tools: Audiobookshelf, Python 3 (standard library only), the Libris xsearch API, Storytel's search endpoint, Audible via Audiobookshelf, Google Books. Scripts are plain cron jobs on a small Linux box. They are tailored to this library, so I have not published them; if a step is unclear or you want more detail on one, ask in the comments.