<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: fullast.nu</title>
    <description>The latest articles on DEV Community by fullast.nu (@fullast).</description>
    <link>https://dev.to/fullast</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4213704%2Fa7d4ee66-2381-4bc5-ac2b-057424bed315.png</url>
      <title>DEV Community: fullast.nu</title>
      <link>https://dev.to/fullast</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/fullast"/>
    <language>en</language>
    <item>
      <title>Fediverse comments, a week in: who did it first, and the replies that went missing</title>
      <dc:creator>fullast.nu</dc:creator>
      <pubDate>Tue, 10 Nov 2026 14:00:20 +0000</pubDate>
      <link>https://dev.to/fullast/fediverse-comments-a-week-in-who-did-it-first-and-the-replies-that-went-missing-inn</link>
      <guid>https://dev.to/fullast/fediverse-comments-a-week-in-who-did-it-first-and-the-replies-that-went-missing-inn</guid>
      <description>&lt;p&gt;A week after &lt;a href="https://blog.fullast.nu/posts/2026-10-29-static-blog-fediverse-comments/" rel="noopener noreferrer"&gt;the post about using the fediverse as this site's comment system&lt;/a&gt; the thread under it had taught me more than the building did. One link a reader posted changed the widget twice. One thing a reader did showed me a hole I thought I couldn't close. And I owe some people a credit.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffhbomuqfsxktzddoe3i4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffhbomuqfsxktzddoe3i4.png" alt="A mock of the comments section: a question with a reply and a counter-reply nested under it, a Lemmy comment, a Bluesky comment, and below them a dashed box for a second Bluesky reply the thread never returns" width="800" height="427"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Not a new idea
&lt;/h2&gt;

&lt;p&gt;Among the comments was a link, with "that's been around for a while" attached. Andreas Scherbaum &lt;a href="https://andreas.scherbaum.la/post/2024-05-23_client-side-comments-with-mastodon-on-a-static-hugo-website/" rel="noopener noreferrer"&gt;wrote this up in May 2024&lt;/a&gt; for his Hugo site, and he opens by crediting the people before him: Carl Schwan's &lt;a href="https://carlschwan.eu/2020/12/29/adding-comments-to-your-static-blog-with-mastodon/" rel="noopener noreferrer"&gt;post from December 2020&lt;/a&gt;, which is the one most of the later ones trace back to, then Jan Wildeboer's for Jekyll and Daniel Pecos Martínez's. Four years of people arriving at the same shape: publish, announce on Mastodon, let the page fetch the replies to the announcement. I had arrived at it too, and written it up as if it were mine. It isn't, and this is the correction. What I added was Bluesky as a second source and a Lemmy community mention to get the post in front of people, and neither of those is much.&lt;/p&gt;

&lt;p&gt;I read Scherbaum's post properly and took two things from it.&lt;/p&gt;

&lt;p&gt;His comments are a thread. Mine were a flat list sorted by time, which read fine with four replies and fell apart at eighteen, when a reply to a reply landed six entries below the thing it answered. Each reply now carries the id of the one it answers, children render under their parent, oldest first, and anything deeper than three levels sits at three, because the indent on a phone had run out of room by then. A reply whose parent I can't see, because it was deleted or isn't public, goes to the top level rather than vanishing.&lt;/p&gt;

&lt;p&gt;The other was when to load. The page used to ask Mastodon and Bluesky for the replies as soon as it opened, for every reader, including the ones who never scrolled past the second paragraph. The privacy page says the site contacts no third party. That was only true if you didn't count the comments. Now an observer watches for the comments heading to come within about a screen of the viewport and fetches then. Someone who reads the first half and leaves causes no request to anyone. I checked that with a browser script against the live page: zero API calls before scrolling, three after.&lt;/p&gt;

&lt;p&gt;Then I looked at the live post and saw a flat list. The new script was there, the page even said so, and my browser was running the old one. The web server tells browsers to keep scripts for a week, which is sensible for a file that never changes and the opposite of that for one I had just changed under the same name. Fresh browsers got the threads; anyone who had read the post before, which is the audience of a follow-up, got the old file for up to a week. The stylesheet already went through the build with a hash in its filename, so a change means a new name and the old one simply isn't asked for again. The script now does the same. I had tested the change in a browser that had never seen the page, which is the one kind of browser a returning reader doesn't have.&lt;/p&gt;

&lt;p&gt;He also keeps a per-post list of comments to hide. I haven't copied that. Moderation stays on the Mastodon side, where blocking an account should keep its replies out of the API too, though I haven't had to find out, and the day a Bluesky reply needs hiding I'll add a list for that.&lt;/p&gt;

&lt;h2&gt;
  
  
  The slow network blanks the fast one
&lt;/h2&gt;

&lt;p&gt;While testing the threads I caught one page load where the section stayed empty for seconds with no error, and the requests had all gone out. It didn't happen again and I never proved what it was. But the shape was clear enough: the code waited for both networks before drawing anything, with no limit on the wait. I held the Bluesky request back on purpose to see it, and sixteen Mastodon replies that had arrived in under a second sat invisible behind it.&lt;/p&gt;

&lt;p&gt;Each network's replies now draw as they come in, and a request gives up after ten seconds. The status line under the heading says which one failed, "Bluesky replies unavailable right now", instead of claiming there are no replies, which is a different statement and a false one. Small change. The first post argued that you only meet this kind of thing on real networks with real readers, and here it took one slow request to prove it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The replies that weren't there
&lt;/h2&gt;

&lt;p&gt;One reader on Bluesky replied to the announcement, I answered, and a little later they added a second reply under their first one. I got the notification. The blog never showed it, and the thing that watches the threads for me and pushes new replies to my phone didn't see it either. My first guess was that the widget was broken.&lt;/p&gt;

&lt;p&gt;A Bluesky reply stores two references: the post it answers, and the top of the thread it belongs to. Their first reply had both pointing at my announcement, which is right. The second had both pointing at their own first reply. The parent was correct, the root was not, and when you ask the API for the thread under my post, a reply with the wrong root simply isn't in it. I tried from three directions and it came back from none of them.&lt;/p&gt;

&lt;p&gt;My first draft of this post said I wouldn't work around it. The only way I could see was to read every commenter's recent posts from every reader's browser and look for stragglers, which is slow and turns a comment widget into something that follows people. Then the same account did it again, on the eighth, this time under my answer, and I finally looked at the rest of its replies. Of its last fifty-three replies to other people, fifty-two set the root to the post they answer. That is only correct for a direct reply to a top-level post, which is why its first comment worked and anything deeper went missing.&lt;/p&gt;

&lt;p&gt;The way out had been in my notifications the whole time. Bluesky knew where those replies belonged well enough to tell the blog's account about them. So a small job on the server now reads that account's notifications every ten minutes, keeps the replies whose parent is somewhere in one of the post threads but which the thread itself doesn't contain, and writes their ids into the same small file the page already reads to find the announcements. The page fetches those posts directly and hangs each one under its parent. No reader's browser looks at anyone's other posts, and the reply watch and the stats read the same list. Both missing replies were back under the post within a few minutes of the first run.&lt;/p&gt;

&lt;p&gt;It only catches what Bluesky tells me about. If a third person answered one of those lost replies with the same kind of software, nothing would notify me, and that reply would stay missing. I haven't seen it happen yet.&lt;/p&gt;

&lt;p&gt;There's a smaller version of the same thing on the Mastodon side that I haven't tested either. The public endpoint that returns a thread's replies is said to stop at sixty for unauthenticated requests. My busiest thread has twenty. If one ever gets there, the last comments will drop off the page and I'll find out from a reader.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd say now
&lt;/h2&gt;

&lt;p&gt;The first post said comments outliving the site was the point. It still is. But "the comments" turned out to mean "what two public APIs return for one post id", and that's a narrower thing than a comment thread. Most of the time they're the same. Twice in a week they weren't, both times for the one person who had bothered to write more than once, and it took the notification on my phone to see that the network knew better than its own thread.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Tools: Hugo, PaperMod, Mastodon's public API, Bluesky's AT Protocol API, Playwright with Chrome. The widget is one file of plain JavaScript; the scripts around it are specific to this setup and not published.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>selfhosted</category>
      <category>homelab</category>
      <category>blog</category>
      <category>mastodon</category>
    </item>
    <item>
      <title>A static blog with the fediverse as its comment system</title>
      <dc:creator>fullast.nu</dc:creator>
      <pubDate>Tue, 03 Nov 2026 13:30:10 +0000</pubDate>
      <link>https://dev.to/fullast/a-static-blog-with-the-fediverse-as-its-comment-system-732</link>
      <guid>https://dev.to/fullast/a-static-blog-with-the-fediverse-as-its-comment-system-732</guid>
      <description>&lt;p&gt;I wanted a blog with comments and without a comment system. Comment forms attract spam the moment a page is indexed; a comment database is one more thing to back up and patch; third-party widgets bring tracking. Meanwhile I already run a single-user Mastodon instance where the conversation would happen anyway.&lt;/p&gt;

&lt;p&gt;So the design: &lt;strong&gt;publish on my own site, announce on the fediverse, and let the announcement be the comment thread.&lt;/strong&gt; The post page fetches the replies to its own toot (and its Bluesky post) and shows them underneath. Readers reply where they already have an account; I moderate with the tools those networks already have. This post is how it is built and what went wrong on the way.&lt;/p&gt;

&lt;h2&gt;
  
  
  The site
&lt;/h2&gt;

&lt;p&gt;Hugo with the PaperMod theme, markdown posts in a git repository on a box at home. The output is plain files served by nginx in a small, isolated container that only the reverse proxy can reach. Nothing dynamic. &lt;code&gt;GET&lt;/code&gt; and &lt;code&gt;HEAD&lt;/code&gt; only; everything else is a 405.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Publishing is a git push.&lt;/strong&gt; A cron job on a build box fetches the repository every five minutes and, when the commit changed, builds twice: the public site, and a preview that includes drafts and scheduled posts behind basic auth. The public build goes through a &lt;strong&gt;redaction lint&lt;/strong&gt; (below) and is then copied into the web container atomically: unpack next to the live directory, swap, delete the old one.&lt;/p&gt;

&lt;p&gt;Scheduling is free: a post with a future date and &lt;code&gt;draft: false&lt;/code&gt; is skipped by Hugo until the time has passed. The build job keys its "anything changed?" check on the commit &lt;em&gt;plus the list of still-future posts&lt;/em&gt;, so the first run after the time rebuilds, publishes and announces.&lt;/p&gt;

&lt;p&gt;One more gate, added after a near miss: a scheduled post only goes out if its &lt;strong&gt;reviewed content has been approved&lt;/strong&gt;, recorded as a hash of the file. Edit the post after approving it and the build holds it, with a line in the log, until someone approves the new version. The first post missed its own slot exactly like that: three small edits after the morning approval, and 15:00 came and went. Annoying for ten minutes, and exactly what the gate is for.&lt;/p&gt;

&lt;h2&gt;
  
  
  The announcement
&lt;/h2&gt;

&lt;p&gt;After each deploy, a script reads the site's RSS feed. For every post it has not announced yet:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Mastodon:&lt;/strong&gt; one public toot from my account: title, summary, link, up to eight hashtags from the post's tags. The toot's id is saved against the post's path. An &lt;code&gt;Idempotency-Key&lt;/code&gt; makes a retry safe.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bluesky:&lt;/strong&gt; a post from my account there, with a link card, the social-card image uploaded as the thumbnail, and tag facets.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lemmy:&lt;/strong&gt; if the post's front matter names a community, the toot mentions it, and Lemmy turns that into a post there. Replies federate back as replies to the toot. One catch: the mention only resolves if my instance has &lt;em&gt;seen&lt;/em&gt; that community before. The first time it had not, the mention stayed plain text and no Lemmy post appeared, until I looked the community up once in the web UI and edited the toot; Lemmy then created the post from the edit within seconds. The publisher now asks the instance to look the community up itself before it posts, which needs a token that is allowed to search. Since then it just works.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Search engines:&lt;/strong&gt; an IndexNow ping with the new URL.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;dev.to:&lt;/strong&gt; the full article via its API with a canonical link back here. Three things its API does not tell you: it refuses the default Python user agent outright ("Forbidden Bots"), it renders every single newline as a line break, so hard-wrapped markdown arrives as a poem, and it drops SVG images. The publisher now sends a descriptive user agent, unwraps paragraphs, swaps an SVG for its PNG twin when one exists, and picks the four tags from the post that dev.to readers actually follow.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The saved ids go into a &lt;code&gt;syndication.json&lt;/code&gt; on the site. That file is the only state the comments need.&lt;/p&gt;

&lt;h2&gt;
  
  
  The comments
&lt;/h2&gt;

&lt;p&gt;A small script on the post page reads &lt;code&gt;syndication.json&lt;/code&gt;, finds the toot id for the current path, and calls the Mastodon instance's &lt;strong&gt;public&lt;/strong&gt; context endpoint for that status. No token: public statuses and their replies are public. Bluesky's public API gives the same for the post thread.&lt;/p&gt;

&lt;p&gt;The rendering rules are the security story:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Everything is inserted as text.&lt;/strong&gt; Reply HTML from Mastodon is converted to plain text (paragraph breaks kept); Bluesky gives text already. Nothing from either API is ever parsed as HTML on my page.&lt;/li&gt;
&lt;li&gt;Links in replies become plain anchors with &lt;code&gt;rel="nofollow ugc noopener noreferrer"&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Avatars load only from the two known hosts, as small thumbnails; a missing one is dropped instead of showing a broken image.&lt;/li&gt;
&lt;li&gt;Only public and unlisted replies; at most 200.&lt;/li&gt;
&lt;li&gt;The page sends nothing anywhere. Reading is the only direction.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Moderation is zero extra work: block or mute an account on Mastodon and its replies disappear from the API response, and therefore from the page.&lt;/p&gt;

&lt;h2&gt;
  
  
  The headers
&lt;/h2&gt;

&lt;p&gt;Because nothing on the page is dynamic, the Content Security Policy can be strict: &lt;code&gt;default-src 'none'&lt;/code&gt;, scripts and styles from the site only (the theme needs inline for its dark-mode toggle), &lt;code&gt;connect-src&lt;/code&gt; limited to the Mastodon instance and Bluesky's public API, &lt;code&gt;img-src&lt;/code&gt; the same two hosts plus self, no framing, no forms, no base tag. Add &lt;code&gt;nosniff&lt;/code&gt;, a referrer policy, cross-origin isolation headers, rate limits per client, and an access log with the last part of every address removed. No cookies, so no banner.&lt;/p&gt;

&lt;h2&gt;
  
  
  The redaction lint
&lt;/h2&gt;

&lt;p&gt;The biggest risk on a blog about your own network is you. Project notes are full of internal addresses, hostnames, identifiers and the odd token. So the public build is grepped before every deploy for private IP ranges, the public IP, MAC addresses, the street and the town, phone numbers, e-mail aliases, anything that looks like a bearer token or API key, serial numbers, internal hostnames, and names of things I have decided never to mention. One hit and the deploy stops with a notification. An allow-list handles deliberate mentions. It has already caught a home-page title.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two bugs a real browser found
&lt;/h2&gt;

&lt;p&gt;Before the first post I ran the preview through headless Chrome on a VM, with the comments pointed at a test thread. Unit-level checks had all passed. The browser found two things that would have shipped:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The preview never showed comments.&lt;/strong&gt; The page's lookup key kept the &lt;code&gt;/preview&lt;/code&gt; prefix, so it never matched the keys in &lt;code&gt;syndication.json&lt;/code&gt;. Trivial, invisible without a browser.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nobody at home would ever see comments.&lt;/strong&gt; My other public hostnames resolve to the reverse proxy's LAN address when you are on the LAN (split DNS); the new blog name did not yet. To Chrome the blog was therefore a &lt;em&gt;public&lt;/em&gt; page trying to read a &lt;em&gt;private&lt;/em&gt; address, and its private-network-access rule blocked the request. Visitors on the internet were fine; the author was not. One DNS override fixed it, and "every public name gets the override" is now a rule.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That check is now part of publishing: load the draft in Chrome, collect console errors, failed requests, broken images and links, verify the comments widget, take phone and desktop screenshots, and put it all on a review page behind the preview login.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug a browser could not find
&lt;/h2&gt;

&lt;p&gt;The announcement toot for the first post went out fine: text, hashtags, link. No link card. Bluesky's card was there, every other crawler had fetched the page, and the Mastodon instance (mine) had not even tried. Nothing in its logs, no failed job, no retry.&lt;/p&gt;

&lt;p&gt;The cause was the fix from bug 2. Split DNS makes the blog's name resolve to the reverse proxy's &lt;em&gt;LAN&lt;/em&gt; address inside the house, and the Mastodon server lives inside the house. Mastodon's HTTP client refuses to fetch private addresses (a sensible guard against server-side request forgery), and the link-card service swallows that refusal silently. So every link to one of my own sites, posted from my own instance, would never get a card. The clean fix was not to loosen the guard but to give the Mastodon containers public DNS: my names then resolve to the public address and the fetch loops back in through the firewall, like any outside visitor. Running &lt;code&gt;FetchLinkCardService&lt;/code&gt; by hand on the toot produced the card a second later.&lt;/p&gt;

&lt;p&gt;Lesson 2 therefore comes with a footnote: split DNS is right for the clients, and wrong for any service on the LAN that is supposed to behave like a stranger.&lt;/p&gt;

&lt;h2&gt;
  
  
  After the first post
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Replies&lt;/strong&gt; are fetched every ten minutes and each new one arrives as a push notification with its text and a link to answer it. Answering within the hour instead of the next morning makes the fediverse feel like a comment system.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Where the readers came from:&lt;/strong&gt; the Lemmy community mention did more than everything else combined. Within an hour of the post, a few hundred fediverse servers had fetched the page for their link previews, almost all of them subscribers of that community. Page views were a fraction of that, in the low hundreds on the first day and two thirds of them within two hours of the announcement; previews are reach, not readers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The comments&lt;/strong&gt; came in the way I had hoped. The first one was from an account that describes itself as an AI agent; it asked a precise question about the matching rule and followed up with a sharper one. It is a public thread, so I answered it like any other reader. A few days later a human reader on another Mastodon server asked what the weekly spot-check actually catches, and in passing described their own convention for records they are not sure about yet, which I am borrowing. Two networks, two conversations, one page showing both.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Identity:&lt;/strong&gt; every page carries a &lt;code&gt;rel="me"&lt;/code&gt; link to my Mastodon profile, which gives the profile its verified tick, and each post names my account as its author (&lt;code&gt;fediverse:creator&lt;/code&gt;), so shared links show "More from" on servers that support it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What I would tell someone starting the same thing
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Static plus "comments live elsewhere" is a very comfortable place to be. The attack surface is a web server serving files.&lt;/li&gt;
&lt;li&gt;The toot id is the only state. Save it before anything else happens; never re-announce.&lt;/li&gt;
&lt;li&gt;Render third-party content as text. Always.&lt;/li&gt;
&lt;li&gt;Put your own address names behind split DNS consistently, or Chrome will teach you about private network access. Then give the services that must look like outsiders (your fediverse server) public DNS, or they will silently refuse to fetch your own pages.&lt;/li&gt;
&lt;li&gt;Lint the output for your own secrets. You will leak something otherwise.&lt;/li&gt;
&lt;li&gt;Test in a real browser from the network your readers are not on: your own.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Tools: Hugo, PaperMod, nginx, Nginx Proxy Manager, Mastodon, Lemmy, Bluesky's AT Protocol API, dev.to's API, IndexNow, Playwright with Chrome, Python 3, cron. The scripts are specific to this setup and not published; ask in the comments if you want more detail on a step.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>selfhosted</category>
      <category>homelab</category>
      <category>security</category>
      <category>blog</category>
    </item>
    <item>
      <title>Cleaning up metadata in a large Audiobookshelf library, one night at a time</title>
      <dc:creator>fullast.nu</dc:creator>
      <pubDate>Thu, 29 Oct 2026 14:34:32 +0000</pubDate>
      <link>https://dev.to/fullast/cleaning-up-metadata-in-a-large-audiobookshelf-library-one-night-at-a-time-331a</link>
      <guid>https://dev.to/fullast/cleaning-up-metadata-in-a-large-audiobookshelf-library-one-night-at-a-time-331a</guid>
      <description>&lt;p&gt;&lt;a href="https://www.audiobookshelf.org/" rel="noopener noreferrer"&gt;Audiobookshelf&lt;/a&gt; is a great self-hosted audiobook and e-book server, as long as the files it indexes come with decent metadata. Mine mostly didn't. Years of collecting had left a large library (tens of thousands of items) where a good share of entries looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Title:   Anna.Author-The.Winter.House.SWEDISH.AUDIOBOOK.WEB-GRP
Author:  (empty)
Genres:  Speech
Description: Ripped by someone, enjoy.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the &lt;em&gt;folder name&lt;/em&gt; as title, no author, a genre that is really an ID3 content type, and a description that is release-group graffiti. Multiply by thousands and the library is unbrowsable: no author pages, no series, search that only works if you remember the folder name.&lt;/p&gt;

&lt;p&gt;This post is about the nightly job that fixed it, the rules that kept it from making things worse, and the four bugs it took to get right. Everything below is Python standard library plus public APIs; nothing exotic.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhmybrnsodx25okz1rgh8.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhmybrnsodx25okz1rgh8.png" alt="Pipeline: folder name or existing metadata -&gt; candidate queries -&gt; Libris / Storytel / Audible / Google Books -&gt; clear-match rule -&gt; plan file -&gt; apply -&gt; per-night rollback" width="800" height="267"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule one: fill only what is empty
&lt;/h2&gt;

&lt;p&gt;The single most important design decision: &lt;strong&gt;never overwrite a field that has a value&lt;/strong&gt;. Title and author are the exception for items that have no author at all (the release-name cases above, where the "title" is garbage by definition). Everything else (subtitle, ISBN, language, series and sequence, genres, narrators, publisher, year, description) is filled only when empty, or when it is &lt;em&gt;certainly junk&lt;/em&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;descriptions matching known release-group signatures or shorter than a few words,&lt;/li&gt;
&lt;li&gt;genres that are ID3 leftovers (&lt;code&gt;Speech&lt;/code&gt;, &lt;code&gt;Other&lt;/code&gt;, &lt;code&gt;F&lt;/code&gt;, &lt;code&gt;(12)&lt;/code&gt;),&lt;/li&gt;
&lt;li&gt;the same "description" appearing on books by three or more different authors (a copy-pasted blurb).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Junk is cleared even when there is no replacement. Weak values (a one-word genre like &lt;code&gt;Fiction&lt;/code&gt;) are replaced only when a matched record has something better.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the metadata comes from
&lt;/h2&gt;

&lt;p&gt;Three sources, in this order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The national library catalogue.&lt;/strong&gt; In Sweden that is &lt;a href="https://libris.kb.se/" rel="noopener noreferrer"&gt;Libris&lt;/a&gt;, whose &lt;code&gt;xsearch&lt;/code&gt; API is open, needs no key, and returns JSON or MARCXML. It knows every published edition with ISBN, publisher, year and, crucially for later, the &lt;em&gt;original title&lt;/em&gt; of a translation (MARC field 240).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A store API&lt;/strong&gt; with rich blurbs, series information, narrators and covers. Storytel's search endpoint turned out to be usable as-is for Swedish titles.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Audible&lt;/strong&gt; through Audiobookshelf's own provider endpoint (&lt;code&gt;/api/search/books?provider=audible&lt;/code&gt;) for English titles, and Google Books as a last resort: for descriptions and ISBNs, and for identifying a book nothing else found (more on its stricter rule below).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A query is just &lt;code&gt;author + title&lt;/code&gt;. For items with a release name, the name is parsed into &lt;em&gt;candidates&lt;/em&gt;: the &lt;code&gt;Firstname.Lastname.Words.Of.Title&lt;/code&gt; form has no separator, so every split into 1-4 author words is tried, and the catalogue decides which one is right.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule two: a match must be clear
&lt;/h2&gt;

&lt;p&gt;A record is accepted only when &lt;strong&gt;the author matches&lt;/strong&gt; (same normalised name, or at least the same surname) &lt;strong&gt;and the title is close&lt;/strong&gt; (normalised equality, prefix, or a similarity ratio above 0.85). Anything less is logged as a miss and left alone. Misses are retried after 30 days, because stores add titles all the time.&lt;/p&gt;

&lt;p&gt;That rule rejected a lot of plausible-looking matches, which is exactly the point: an audiobook library with a thousand confident wrong matches is worse than one with a thousand gaps.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;need&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.85&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;norm_title&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;norm_title&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt; \
        &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;difflib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;SequenceMatcher&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;ratio&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;need&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The prefix clause is there for subtitles: "The Winter House" should match "The Winter House: A Novel". It is also the loosest part of the rule. When Google Books became the last identification attempt, it happily matched one book in a children's series, say &lt;em&gt;Harbour Town: The Dragon Summer&lt;/em&gt;, to another, &lt;em&gt;Harbour Town: The End&lt;/em&gt;, because both titles start with the series name. The other sources rarely have that problem; Google's broad catalogue does. So a Google match must pass a stricter test: whole-title similarity of 0.85 or more, no prefix shortcut, and the same language as the release's own tag when Google states one. The looser the source, the stricter the rule.&lt;/p&gt;

&lt;p&gt;Google Books has two more quirks worth knowing about. The free quota is 1,000 requests per day, counted per &lt;em&gt;Pacific&lt;/em&gt; day, so a per-run limit is not enough: my daytime tests plus the night job went over it, and the rest of the night got nothing but "quota exceeded". The counter now lives in a small file shared by every run. And roughly half of all requests get a &lt;code&gt;503 Service temporarily unavailable&lt;/code&gt; at random, whatever the pacing, the headers or the query; the same request a few seconds later usually works. So each lookup gets two retries, and a book that only ever got 503s is not counted as "tried". It comes back the next day instead of in 30. It now runs as its own short daily job right after the quota resets, and identifies about one in ten of the books nothing else could. (The first day also showed why the language check matters: Google happily returns the Norwegian original of a Swedish translation, and my table of Google's language codes only knew Swedish and English, so "Norwegian" looked like "no language stated". A handful of books got Norwegian or Danish titles before I noticed; the plan file put them back.)&lt;/p&gt;

&lt;h2&gt;
  
  
  The night job and its parachute
&lt;/h2&gt;

&lt;p&gt;The job runs from late evening to early morning, before the server's own scheduled scan. Four things make it safe to let loose on a library this size:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A fresh native backup first.&lt;/strong&gt; It triggers Audiobookshelf's own backup through the API, verifies that a new backup of plausible size exists, and refuses to start otherwise.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A plan file per night.&lt;/strong&gt; Every change is appended, &lt;em&gt;with the previous values&lt;/em&gt;, to that night's JSON file before it is applied. &lt;code&gt;--rollback &amp;lt;file&amp;gt;&lt;/code&gt; puts everything back, item by item. I have used it. More than once.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A deadline fixed at start.&lt;/strong&gt; The first version computed the deadline &lt;em&gt;after&lt;/em&gt; the multi-minute backup; one night that rolled the deadline into the next day and the job ran into the morning scan.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A weekly spot-check.&lt;/strong&gt; Rollback only helps if someone notices. Once a week a separate script reads every change from the last seven nights and flags the patterns that have bitten me before: an author the folder name does not mention at all, a publisher in the author field, two people glued into one "name", a series name used as the author, a language that contradicts the release's own tag. Each flag is re-checked against the item &lt;em&gt;as it is now&lt;/em&gt;, so problems that were already repaired don't count. The result is one push notification with counts and a few examples. The first run flagged a dozen or so out of well over ten thousand changes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Throughput was a few thousand items a night, which meant the backlog took about a week. Since then it only touches new items and retries.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bug one: the language tag that read "En" as English
&lt;/h2&gt;

&lt;p&gt;Release names carry a language tag somewhere after the title: &lt;code&gt;...-SE-2020-GRP&lt;/code&gt;, &lt;code&gt;...SWEDISH.AUDIOBOOK...&lt;/code&gt;, &lt;code&gt;...-EN-...&lt;/code&gt;. The first regex took the &lt;em&gt;first&lt;/em&gt; match. Swedish has a very common word spelled &lt;strong&gt;"En"&lt;/strong&gt; ("a", "one"). So &lt;code&gt;Firstname_Lastname_-_En_Gang_I_Byn-...-SE-2020&lt;/code&gt; was tagged English, got an English ISBN, lost its Swedish one, and was looked up on Audible for good measure.&lt;/p&gt;

&lt;p&gt;The fix is one word: take the &lt;strong&gt;last&lt;/strong&gt; tag, because tags follow the title.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;tags&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;(?i)(?&amp;lt;=[._-])(swedish|english|danish|norwegian|se|en)(?=[._-])&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;lang&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;se&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;Swedish&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;en&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;English&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;}.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tags&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;tags&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;capitalize&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tags&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Bug two: the original title finds the original edition
&lt;/h2&gt;

&lt;p&gt;Some Swedish releases are named after the book's &lt;em&gt;original&lt;/em&gt; title: &lt;code&gt;Author.Name.The.Winter.House.SWEDISH...&lt;/code&gt; for a translation sold as &lt;em&gt;Vinterhuset&lt;/em&gt;. Searching the stores by that title finds the &lt;strong&gt;English&lt;/strong&gt; edition: English ISBN, English blurb, English narrator, language "English". The library said Swedish, the metadata said otherwise, and a good four per cent of one night's matches were like that.&lt;/p&gt;

&lt;p&gt;Two changes:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The release's own language tag wins. A matched edition in another language contributes only title, author and series; its ISBN, publisher, description and narrators are dropped. A record with no language but an English-range ISBN (978-0 / 978-1) counts as foreign too.&lt;/li&gt;
&lt;li&gt;Then &lt;strong&gt;find the Swedish edition&lt;/strong&gt;: a Libris MARCXML query restricted to Swedish records (&lt;code&gt;spr:swe&lt;/code&gt;) where field 240, the original title, matches the title we have. That returns &lt;em&gt;Vinterhuset&lt;/em&gt; with its Swedish ISBN and publisher; the store search with the Swedish title then finds the Swedish blurb and narrator.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Danish and Finnish ISBNs on Swedish audiobooks, by the way, are normal: several big audiobook publishers are registered there. The first version of my check flagged hundreds of those as errors. Check your assumptions against the data before you "fix" it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bug three: the author that was never right
&lt;/h2&gt;

&lt;p&gt;Fill-only-when-empty has a blind spot: metadata that is &lt;em&gt;wrong&lt;/em&gt; rather than missing. A scan found a surprising number of items whose stored author shared &lt;strong&gt;no name word&lt;/strong&gt; with the author in the folder name. Some were the right book with a translator or narrator credited as author. Others were a different book entirely: a Swedish crime novel tagged as a German dictionary, a children's series carrying a biography's description.&lt;/p&gt;

&lt;p&gt;So a second pass re-identifies those from the folder name alone, with two outcomes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Same book&lt;/strong&gt; (old title close to the found one): the found author goes first, empty fields are filled, nothing is cleared. What happens to the old author name turned out to be bug four.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Different book&lt;/strong&gt;: the new identity replaces the old one, and fields the new record cannot fill are cleared, because they belonged to the other book.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Plus a guard: if the volume numbers disagree (&lt;code&gt;Series 1&lt;/code&gt; matched to &lt;code&gt;Series 3&lt;/code&gt;), skip. The dry run caught that one.&lt;/p&gt;

&lt;p&gt;The first batch of a few hundred was reviewed by hand before applying; it then joined the nightly job with its own state, so an item fixed by the first pass is not "done" for the second.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bug four: "lossless" kept the wrong name
&lt;/h2&gt;

&lt;p&gt;My first version of the same-book case kept the old author next to the found one. The reasoning sounded careful: the old name might be a co-author, so don't throw anything away. Lossless.&lt;/p&gt;

&lt;p&gt;The first full night of it said otherwise. Of the few hundred books re-identified that night, roughly four in ten now listed two authors, and in most of them the second name was simply the leftover of the original wrong match: a Swedish crime novel credited to its real author &lt;em&gt;and&lt;/em&gt; an American romance writer. A few dozen had a publisher as co-author. And Audiobookshelf keeps an item's existing author first when you add one, so the wrong name was the one you saw.&lt;/p&gt;

&lt;p&gt;"Keep it, it might be right" is not lossless. It is keeping an error with a note attached. The rule now is that an old name stays only when something confirms it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the matched record lists that person as a creator (library records include illustrators and translators),&lt;/li&gt;
&lt;li&gt;the folder name contains the name, or&lt;/li&gt;
&lt;li&gt;a source finds this title under that name.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Publisher-like names never stay, and a found "author" that is really the series name (yes, that happened: a children's series whose title character was taken for the writer) is rejected. A one-off repair script re-planned every affected item from its pre-night values with the fixed rule and rewrote only the author field, with its own rollback file. Picture books came through with writer and illustrator intact, because the library record names both.&lt;/p&gt;

&lt;p&gt;The same week, the opposite case: folder names that list two authors, where only one was found. A name like &lt;code&gt;Anne_Author_Bea_Writer_-_Title&lt;/code&gt; cannot be split reliably, but the stores list every author of a book. So after a match, any other author from the record whose full name also appears in the folder name is added. A narrator or translator the folder doesn't mention is never added.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it looks like now
&lt;/h2&gt;

&lt;p&gt;Author pages exist. Series are grouped and numbered. Swedish books have Swedish titles and blurbs. Searching by a title you half remember works. The job still runs every night, mostly to catch new arrivals, and reports a one-line summary with the rollback command for that night.&lt;/p&gt;

&lt;p&gt;If you take one thing from this: &lt;strong&gt;plan, then apply, and keep the previous values next to every change.&lt;/strong&gt; The rest is regex and patience.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Tools: Audiobookshelf, Python 3 (standard library only), the Libris xsearch API, Storytel's search endpoint, Audible via Audiobookshelf, Google Books. Scripts are plain cron jobs on a small Linux box. They are tailored to this library, so I have not published them; if a step is unclear or you want more detail on one, ask in the comments.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>selfhosted</category>
      <category>homelab</category>
      <category>automation</category>
    </item>
  </channel>
</rss>
