DEV Community: chovy

nmaptui: an admin console for nmap

chovy — Fri, 11 Sep 2026 10:15:10 +0000

nmaptui is an admin console for nmap that runs in the terminal. It is open source (MIT), it is on npm as @profullstack/nmaptui, and it installs with one line:

curl -fsSL https://raw.githubusercontent.com/profullstack/nmaptui/main/install.sh | sh

nmap itself is fine. What it lacks is a place to live between scans. You run it, read the output, lose the output, run it again a month later, and wonder what changed. nmaptui keeps every scan, shows it in eight screens, and diffs any two.

What is in it

A scan builder with eighteen profiles (Quick, Intense, Ping sweep, Vulnerability scripts, Web services, Windows and SMB, Databases, Top 100 UDP, and so on) and a form for every nmap option that matters, each with a one-line explanation. The exact command nmap will get sits underneath the form, with a count of how many addresses the targets cover and a warning when the scan needs root.

A live view. nmaptui runs nmap with XML on stdout and reads it as it streams, so tasks appear as they start, nmap's own percent and time remaining show once it reports them, and each host lands in the table the moment nmap finishes it. Abort keeps what came back.

Hosts, Services and Findings. The hosts screen is the usual per-host detail: ports, versions, OS matches, uptime, traceroute, script output. The services screen pivots the scan the other way, so "which boxes still run OpenSSH 7" is one row. Findings is a triage list from rules over the scan: clear-text protocols, anonymous FTP, SMBv1 and unsigned SMB, databases and RDP reachable from the network, expired certificates, weak TLS, end-of-life operating systems, and anything an NSE script marked VULNERABLE.

History and diff. Scans are kept as nmap's own XML under ~/.local/share/nmaptui. Mark two and you get hosts that appeared or vanished, ports that opened or closed, and services whose version moved. Any file made with nmap -oX can be imported.

Export to text, JSON, CSV, Markdown, HTML or the raw XML. nmaptui print and nmaptui diff work without a terminal, so a nightly diff in a cron mail is a few lines of shell.

How it is built

It is TypeScript on HQTUI, with no other runtime dependency. The XML reader is 290 lines and has a streaming mode that hands over each complete host element as the bytes arrive. Screens are pure functions of one state object, and the tests render real frames headlessly and assert on the text, so a layout regression fails a test instead of waiting for someone to notice. 53 tests run under both node --test and bun test.

The screenshots in the README are a real connect scan of profullstack.com, captured through a pty. It sits behind Pingora with two ports open and 998 filtered, which is about the right answer.

Repo: https://github.com/profullstack/nmaptui

nichedb 0.8: symbols, price history, fundamentals and the crypto universe; advis0r and b1dz read it

chovy — Fri, 11 Sep 2026 06:50:03 +0000

Two more sites stop fetching for themselves today. advis0r.com built every research report on four hundred days of Alpaca bars, the company's SEC facts and Alpaca's asset list; b1dz.com asked CoinGecko and four exchanges every five minutes which pairs exist and what they did. nichedb.dev 0.8 keeps all of it once, and both sites read it there.

What moved, and what did not

Not everything a trading site does is a feed. b1dz's WebSocket tickers, its two-second arbitrage ticks, its private broker and exchange accounts, its DEX quotes and its DealDash session stay where they are; so do advis0r's signal extraction, corroboration, scoring, speech-to-text and the LLM analysis. What moved is the data both sites, and any site like them, need the same copy of.

nichedb 0.8

The symbol directory. alpaca-assets keeps every tradable US equity and crypto asset Alpaca lists, about 14,400 rows, refreshed daily. Feed: us-symbols.

Price history. equity-history keeps one item a symbol carrying its last four hundred daily bars, appended after each session settles from the multi-symbol bars endpoint, about ninety requests an evening for 12,900 symbols. An adapter can read its own previous items back now, so a window is kept, not rebuilt. Feed: price-history.

Fundamentals. sec-fundamentals walks every SEC filer's XBRL facts once a week at five requests a second: revenue, income, EPS, assets, liabilities, cash, operating cash flow, shares, from 10-K, 10-Q, 20-F and 40-F, the last twelve points of each. Feed: fundamentals.

The crypto universe. A new crypto collection: coingecko-assets (the top 500 by market cap every quarter hour) and crypto-pairs (every spot pair on Kraken, Coinbase, Binance.US and Gemini with its 24-hour ticker, every five minutes, one row a pair a venue, 2,190 of them). The doc says where each venue is odd: Kraken's daily open is midnight UTC, Gemini publishes no volume on any all-pairs endpoint, Coinbase has no bid or ask. Feeds: usd-pairs, top-100.

Two things found on the way. A source seeded off because its key was missing stayed off after the key arrived, since the seed never touched the flag; the Alpaca corporate-actions source had sat at zero items for that reason, and now has 6,400. And the seed judged a source's needs against a slice of the config that did not hold the contact email the SEC wants in a user agent. Both fixed.

The sites

advis0r.com runs with NICHEDB_MARKETS=1: a report build reads the symbol's history item and the company's fundamentals item, two requests, and falls back to the live path when either is missing or stale; the symbol sync walks the directory by cursor; a ticker's news refresh reads nichedb's market news wire before the RSS feeds. Fresh reports record their bars source, and today's say nichedb.

b1dz.com runs with NICHEDB_CRYPTO=1 and NICHEDB_MARKETS=1: pair discovery reads the dollar-quoted pairs and the assets from nichedb, eleven requests every five minutes instead of six calls to five hosts, and applies the same rules, two venues or more with fifty thousand dollars a day on each and ten million of market cap; backtests read the history item and fall back to Yahoo when the window is longer than it. Its first discovery pass from nichedb picked 150 pairs from 1,582 rows.

That makes five sites on nichedb: tipoffwatch, genrewatch, bittorrented, advis0r and b1dz, plus nixamp. The contract for each collection is one document in the repo.

nichedb: https://nichedb.dev.

nichedb 0.7 closes the gaps, tipoffwatch and genrewatch switch to it, and nixamp 0.12 shows the score

chovy — Thu, 10 Sep 2026 19:34:53 +0000

Yesterday nichedb.dev 0.6.0 took over the fetching three of our sites did for themselves. Today 0.7 closes the gaps that kept those sites on their own pollers, the sites switch, and nixamp learns the score.

What was missing

A site reading nichedb instead of ESPN loses nothing on schedules and scores, but tipoffwatch.com also showed every play of a game and a box score when it ended, and it filled in the broadcaster for games ESPN lists without one, which is most of the world outside the US. genrewatch.com showed where a film could be rented, not just streamed, and it gave a poster and a synopsis to the 400,000 titles that only exist in the IMDb ratings dump. None of that was in nichedb. Now it is.

nichedb 0.7

Play-by-play and recaps. espn-plays reads the ESPN summary for every fixture in play or ended in the last six hours, eight summaries a run every two minutes, which is exactly the line tipoffwatch already held. One item per fixture: the last 400 plays while the game is on, the whole log with the box score, leaders, officials and closing line once it is final. Feed: play-by-play.

TV listings. sportsdb-tv keeps TheSportsDB's day listings, one item per event, channel and market, fourteen days ahead. ESPN's broadcast field is US-only; this is where "7 Queensland" for an AFL game and "Abema TV" for a sumo basho come from. Feed: tv-listings.

Rent and buy. A TMDB title now carries providers: { stream, rent, buy } beside the flat-rate watch list, and every release row says which shops carry it, so "included where I subscribe" and "available to rent" stay two questions.

Posters for the IMDb titles. A new enricher, tmdb-artwork, takes a title that came in from the IMDb dumps with nothing but a name, a year and a rating, and asks TMDB's find endpoint for the poster, the synopsis and the TMDB id. The backlog is 430,000 titles at about 29,000 a day, so it takes a couple of weeks; new titles get theirs within a tick, and a mirror sees the poster land because the row's updated_at moves with it.

A game by the name a playlist gives it. GET /api/v1/match?q=NFL: Chiefs vs Bills answers with the fixture. The parser reads two sides and a league out of "Lakers @ Celtics", "Arsenal v Chelsea", "Rangers at Celtic 19:45" and the messier things IPTV playlists actually say, "US (ESPN+ 017) | MLB: Rays vs. Braves (ESP) (2026-09-10 11:30:10)", and the match scores each fixture in the window by team-name agreement in either order: abbreviations, short names, "Man Utd", "Alcorn St". One side only scores below the floor, so "Chiefs" alone is never a Kaizer Chiefs game.

The contract for anyone mirroring a collection is one document, docs/consolidation.md in the repo, and the CLI is on npm as @profullstack/nichedb.

The sites switched

tipoffwatch.com runs with SPORTS_PROVIDERS=nichedb-sports: leagues, teams, fixtures, live scores, plays, recaps and broadcasters all come from nichedb, and the site's own ESPN, Live Tennis and TheSportsDB polling is off. Its first pass took 8,045 fixtures from 41 requests and filled 43 broadcasters from nichedb's listings with 9 more; a steady minute is one request. genrewatch.com runs with CATALOG_PROVIDERS=nichedb,musicbrainz,spacedevs: films, TV and anime come from nichedb at 250 pages a pass; music and launches stay where they were. bittorrented.com has the same switches behind NICHEDB_TITLES and NICHEDB_MIRROR, and its nightly IMDb dump import can be retired the night the mirror runs.

Two things bit on the way and are worth knowing if you mirror nichedb yourself. Bun's postgres driver hands jsonb back as text in production where PGlite parses it in tests, so a cursor spread into an object was one key per character and every tick was a full backfill. And nichedb keeps a fixture once per source that saw it, the schedule pass and the live pass both, so a page can carry the same fixture twice; collapse a page to one row per external id, latest updated_at first, before a batch upsert or Postgres will refuse the statement.

nixamp: the score under the player

nixamp 0.12 finishes the enrichment story: films, shows and channels got their posters and logos in 0.11; sports was the piece left. A channel or playlist entry named for two teams is a game, and the player asks nichedb which one and draws the score line under the video: both teams with their logos, the score, the quarter and clock while it is on, the kickoff time before, FINAL after, and the broadcaster. It re-asks once a minute while the game is live, and a miss on a game name is believed only a minute too, since the fixture may be listed by the time the channel is on.

Install or update nixamp with one line:

curl -fsSL https://nixamp.com/install.sh | sh

nichedb: https://nichedb.dev. nixamp: https://github.com/profullstack/nixamp and r/nixamp.

nichedb 0.6.0 keeps sports, screen and channels once; nixamp 0.11 is the first thing built on it

chovy — Thu, 10 Sep 2026 14:17:33 +0000

Two releases today that belong together: nichedb.dev 0.6.0 gains the data three of our sites used to fetch for themselves, and nixamp 0.11.0 is the first thing built on it.

nichedb: sports, screen and channels

tipoffwatch.com polled ESPN for every league and fixture. genrewatch.com polled TMDB, TVmaze, AniList and the IMDb dumps for every film and show. bittorrented.com imported the same IMDb dumps again and kept its own TMDB cache. Three copies of the same fetching, three places for it to break.

nichedb.dev now keeps all of it once, as three collections anybody can read without a key:

sports: every league, team and fixture ESPN publishes across 17 sports and 354 leagues, with schedule, live score, line, venue and broadcast, plus tennis by tour from the Live Tennis API. Live scores refresh every minute for leagues with a game on.
screen: films, TV and anime as titles and releases, to cinemas, to rent, to a streaming service, to air, with posters, genres and the IMDb rating behind 400,000 titles.
channels: the whole iptv-org directory, 31,000 television channels with logo, country, category and network, and the public stream where one exists.

And one new question the API answers: what is this name? GET /api/v1/match?q=Top.Gun.Maverick.2022.1080p.WEB-DL.mkv reads the name the way a torrent site learned to, year and episode and quality tags and release group stripped, and answers with the best title and a score. The same parser lives in one module, so a player, a torrent site and a TV guide all match the same way. The items API gains tag filters, date windows and a since parameter for anyone keeping a copy. The CLI gains nichedb match; the MCP server gains match_items.

The sites will switch to reading nichedb next, and drop their own pollers.

nixamp: the player knows what it is playing

A file called "Oppenheimer (2023) 1080p HDTS.mkv" is a film. A playlist entry called "US: ESPN2 HD" is a channel. nixamp knew neither, because ffprobe reads tags and a download's tags are its file name.

nixamp 0.11.0 asks nichedb. The server you are connected to asks once per name and remembers the answer, misses too, so a library of five thousand files is not five thousand requests. Under the title the page now shows the poster, the year, the rating, the genres and a line of synopsis for a film, and the logo, country, category and network for a channel. Below half similarity nothing is shown: a wrong poster is worse than none.

Install or update nixamp with one line:

curl -fsSL https://nixamp.com/install.sh | sh

nichedb: https://nichedb.dev. nixamp: https://github.com/profullstack/nixamp and r/nixamp.

nixamp 0.10.0: paste any link and it plays, keep it with one click, and every live has its own room

chovy — Thu, 10 Sep 2026 12:40:05 +0000

nixamp 0.10.0 is out. Three things: paste any link and it plays, keep it with one click, and every live stream has its own phone room.

Paste any link

Under the player there is a box. Paste a page into it, a podcast episode, a SoundCloud track, an Internet Archive film, a TikTok live, and the server you are connected to works out where the media actually is and plays it to you as a channel of its own. It is played the way a catalog entry is played: started for whoever asked, stopped a minute after the last viewer leaves, and shared by two people who paste the same link. A bare link to an MP3 or an HLS playlist needs no working-out at all.

The working-out is yt-dlp's, which knows a thousand sites and is updated weekly. The installer fetches it as a standalone binary beside nixamp when it is not already there, so nixamp update is the whole setup. Some sites refuse a datacenter without a signed-in browser, YouTube and Vimeo among them; the player says so in its note, and a Netscape cookies file at ~/.local/state/nixamp/cookies.txt is passed through when you put one there.

Download now

Beside the picture there is a Download button for a link that is a whole file somewhere. The server fetches it through yt-dlp and hands the bytes straight on as a download, named for the title, so your own machine ends up with the file and the server keeps nothing. Sound alone for a track, the picture too for a film. A live stream has no whole yet, so it gets no button.

Every live has its own room

nixamp streams have a call-in number. Until today one code per server put the people calling about the basketball in with the people talking about the film. The directory now gives each channel a code of its own, keeps it while the channel stays on, and the phone line greets a caller by the channel: "Welcome to the live room for FIBA World Cup. There are 3 people in here." The code is shown on the Live rows, on the line under the title, and in the directory beside each channel with how many are on the phone for it.

Also in this run

Since the morning's 0.9 post: the top of the page says LOADING with a turning ring while a stream starts instead of STOPPED, a meta line under the picture says what is playing and who is watching, GO LIVE sits beside Play for whoever administers the server and puts anything on the air with the link copied, a joiner is told the number and the code on the line under the title, and the code shown is always the one the phone line knows.

Install or update with one line:

curl -fsSL https://nixamp.com/install.sh | sh

Source and releases: https://github.com/profullstack/nixamp. Talk about it at https://www.reddit.com/r/nixamp.

nixamp 0.9.10: the phone line says hello, LOADING while it loads, and Go Live beside Play

chovy — Thu, 10 Sep 2026 12:01:44 +0000

nixamp 0.9.10 went out today with three things people asked for after using 0.9 for a morning, plus a phone line that finally says hello.

The phone line answers properly

nixamp streams have a call-in number. You dial 408-357-2326, key the six digit code shown beside the stream, and you land in a room with everyone else watching. Yesterday that room was silent. Telnyx's own call log showed why: the welcome was spoken on the leg and then the leg was put into the conference, and the conference cut the speech off in the same millisecond. Callers heard nothing, waited, and hung up.

The welcome is now spoken inside the room, addressed to the person who just arrived: "Welcome to the live room for server1, playing And Justice For All. There are 3 people in here." The first caller is told they are first. A partial code is answered in four seconds with how many digits were heard, a voice that fails is swapped for a plain one, and three empty prompts end the call with a goodbye instead of a fourth prompt.

LOADING, not STOPPED

Playing something from an IPTV catalog can take half a minute. The server probes the source, starts a decoder, and only then has bytes to send. For all of that the top of the page said STOPPED, which reads as "that click did nothing". It now says LOADING with a ring that turns, the row you clicked spins, and the same state covers a catalog's groups and entries while they are fetched.

A line about what is playing

Under the picture there is a row of chips: live or on demand, the picture's size, how many people are watching a channel and how long it has been on, which catalog and group it came from with the provider's logo, and the phone code when the server is listed. It is text only. Every word of it was written by a provider or a stranger, and none of it is rendered as markup.

Go Live, beside Play

If you administer the server you see GO LIVE next to Play, and the same icon on every row that plays. Play plays it for you. Go Live puts it on the air for everyone: a catalog film or channel becomes a channel that stays up with nobody watching, is written down so a restart brings it back, and is announced to the directory immediately. A file from your library is played on the server. If the server was not listed it gets listed, and the share link is copied so you can send it.

The channels a listed server carries are now rows in the directory, each with its own Play, so what somebody went live with is the thing a visitor can press.

Smaller things

The jingle no longer plays over a stream a link asked for, or over the click that starts one. It only plays into silence.

Install or update with one line:

curl -fsSL https://nixamp.com/install.sh | sh

Source and releases: https://github.com/profullstack/nixamp

nixamp 0.9: a server that names itself, and a CNN feed that stopped talking

chovy — Thu, 10 Sep 2026 10:20:48 +0000

nixamp 0.9: a server that names itself, and a CNN feed that stopped talking

A day of nixamp releases, 0.7.36 through 0.9.3. It started with a bug report ("I see CNN but get no audio or video, and the graph is moving") and ended with servers that get their own DNS name and certificate without anyone touching a registrar.

The frozen channel

CNN had been on the air for twelve hours and showed nobody anything. The graph moved because it was the server's own player mirrored over SSE, not the channel. On the box, the ffmpeg was alive, its byte count frozen, and its stderr socket had 131 KB queued that nothing had ever read. Node only reads a piped stream that is flowing. An IPTV transport stream logs a line for every corrupt packet, the 64 KB pipe filled, and ffmpeg blocked on its next write with every thread waiting on the one that was stuck. It never exited, so nothing redialled.

Every ffmpeg a channel spawns now has its stderr drained. Remote pulls get a read timeout and a watchdog, and a redial resets the stream header instead of appending a second one mid-picture.

Then the channel "looped." A late joiner was handed only future bytes, so it started on the live edge with nothing buffered and stalled on every hiccup. A newcomer now gets the last six seconds first, from a fragment boundary. Measured in Chrome: six stalls in thirty seconds before, zero after.

Catalogs

An m3u with three thousand IPTV entries used to become three thousand flat rows in the playlist. A server now keeps named catalogs, reads group-title and logos, and sorts entries into live and on demand. Anyone with the link browses by group and plays. A live entry becomes a channel started on the first click and stopped a minute after the last viewer leaves, capped at four at once.

Names and certificates

server2 came up as http://152.53.47.37:4321. The only reason server1 had a name was that someone typed an A record into Porkbun and copied a wildcard certificate onto the box by hand.

Now a signed-in server asks nixamp.com for a name and gets label.handle.nixamp.com with A and AAAA records, written with keys only nixamp.com holds. It asks for the handle's certificate and gets one Let's Encrypt wildcard per handle, issued by DNS-01 and renewed on nixamp.com, cached on the box and swapped into the running listener once a day. The API is account-gated and metered with @profullstack/throttle. nixamp dns manages names by hand, and it is 2026, so v6 is not an afterthought.

The one that mattered most

A daemon restarted from a home directory served the home directory, under a public listing. That default is gone. The library is a saved setting, asked for at sign-in or first start, and the whole filesystem, the home directory, anything above it, and anything under a hidden folder in home are refused however they are asked for.

Smaller things

The installer restarts a running daemon with its saved flags, and pulled channels survive a restart. The directory shows what is on each server and offers Viewer and Admin, with Admin greyed unless the server is yours. Favourites are a heart. The tab says what is playing.

Install: curl -fsSL https://nixamp.com/install.sh | sh, then nixamp login, nixamp library ~/Music, nixamp daemon start.

Drafted with Claude Code from the session that did the work.

myna lists your product now, over MCP

chovy — Wed, 09 Sep 2026 19:40:20 +0000

myna posts to social networks. As of 0.17.0 it also lists the product itself, in a software directory, over that directory's MCP server.

myna directory saasrow https://example.com --dry-run
myna directory saasrow https://example.com

It reads the page, works out the name, writes the description, picks the category, and submits.

A listing is not a post

I started by trying to make a directory just another network adapter. That was wrong. A post is text with optional media. A listing is a name, a website, a description, a category and some controlled vocabulary, and a human being at the directory reads it before anyone else does. Pushing that through post() would have given all 30 network adapters fields they have no use for.

So directories are a separate subsystem with a separate command and separate credentials.

The credentials part turned out to matter more than I expected. myna post --to all resolves to every account whose network is not marked as an explicit target, and a network it does not recognise reads as "not explicit". A directory key stored among the accounts would therefore have quietly become a posting target. Somebody's stray thought about lunch, submitted as a product. Directory keys live in their own part of the vault, and there is a test that fails if that ever changes.

Reading the server instead of guessing

The interesting half is that myna is now an MCP client, not only a server.

SaaSRow publishes an MCP endpoint. So rather than writing a REST client against its listing API, myna calls its tools. The tool schemas describe the fields, which means SaaSRow can add one without a myna release.

That works so well it seemed silly to stop at directories myna ships:

myna directory add acme https://acme.example/api/mcp
myna directory tools acme
myna directory acme https://example.com

There is no adapter behind that. myna reads the tool table, finds whichever tool creates a listing whatever it happens to be called, and sends each field under the name that tool's own schema uses. A directory whose field is product_url rather than website needs no code. A field the server marks required and myna could not work out fails before the call rather than after it.

myna directory tools <id> prints what a server actually offers, which is the first useful thing to look at when a submission gets refused.

Signing in is the exception. There is no MCP method for it, and SaaSRow's is an emailed one-time code, which is a conversation with a person rather than a tool call. So that adapter stays written, and a custom directory takes a key you already have.

Two things I got wrong on the way

The product name came out as widget.test in testing. fetchPage falls back to the hostname when a page declares no og:site_name, and I was preferring site name over title. A site name equal to the host carries no information. Fixed, with a test.

And an empty result printed nothing at all, which reads as a command that never ran rather than a directory with nothing to say. Empty output is not a pass.

Where it is

myna directory in the shell, /directory in the TUI, a Directories screen in the desktop app, and four tools over MCP so an agent can do it. cli-tools 0.31.0 adds /myna:directory as a slash command.

myna 0.17.0 is out: https://github.com/profullstack/mynaposter/releases/tag/v0.17.0

This post was drafted with AI assistance and edited by me.

nichedb 0.5.0: the sea, the surf and the aircraft worth watching

chovy — Wed, 09 Sep 2026 19:29:24 +0000

nichedb 0.5.0: the sea, the surf and the aircraft worth watching

Three more sources, all keyless, all verified against the live upstream before a line of the reader was written.

The whole ocean in one file

The National Data Buoy Center publishes the newest observation from every buoy it is listening to in a single hundred-kilobyte file. 853 reports when I read it, 184 of them carrying a wave height, and each one holding wind, gust, pressure, air temperature and water temperature alongside.

A surf report is three of those numbers and everything else is presentation. How big the swell is, how far apart the waves are, and where it is coming from. The middle one is the number people leave out, and it is the one that decides the day: two metres at 18 seconds and two metres at 5 seconds are not the same sea at all. So nichedb publishes a wave row as "8.9 ft at 8 s, mixed swell out of the W" as well as storing the numbers, and it stores the height in feet as well as metres, because every surf forecast in the United States is in feet and converting at read time is how a nine-foot day quietly becomes a three-foot day.

The forecaster's own words next to the buoy's numbers

A buoy off Oahu reading 2.4 metres at 16 seconds is a fact. "Advisory level surf along south facing shores through today" is what that fact means to somebody standing on the beach, and no amount of arithmetic over the buoy produces it.

So the surf zone forecast the National Weather Service issues from every coastal office is in there too, in full. The high surf advisory and the rip current risk are lifted out of the forecaster's own wording into tags, so a coast under an advisory can be found without reading all 851 products, and the text itself is kept as the authority.

Aircraft, but only the ones that mean something

There is a version of this that stores every aeroplane in the sky every few minutes. It would be tens of thousands of rows a day, each one a position that was true for four seconds, and nobody would ever read a single one of them.

A position is not news. An aircraft squawking 7700 is. So nichedb reads the event surfaces of the ADS-B network instead: the three emergency transponder codes, and the military list. When I wrote this the emergency codes returned zero aircraft between them and the military query returned 422, which is exactly the shape you want. The rare thing is rare, so its appearance means something.

And an emergency is an episode rather than a ping. An aircraft squawking 7700 shows up in poll after poll and then simply stops showing up, and nothing anywhere records that it stopped. So the open episodes are held between polls, the row is keyed on when the aircraft was first seen, and when it leaves the list it is written once more with a duration on it. "N123AB squawked 7700 for 22 minutes" is the row worth keeping. The forty positions in between are not.

Four things the live data decided

NDBC writes a missing reading as MM, in every column, in a fixed-width table. Anything that coerces that to a number turns "this buoy has no anemometer" into a flat calm and "no wave sensor" into a dead flat sea, and 669 of the 853 stations report no wave height at all.

The station register really does publish an empty name. Station 15009 in the Atlantic array is one of several, and an empty string left alone beats the fallback and titles the row with nothing at all. That showed up in the very first smoke run as a list of observations called ": marine observation".

Some stations report a dominant period of 0 or 1 second. That is a sensor saying nothing, and calling it windswell states something about the water that nobody measured.

And the ADS-B network rate limits its military list far harder than its squawk lists, answering 429 while the endpoint next to it answers 200. A rate limit is not a failure, so it returns nothing and asks to be called back later, which also keeps the cursor intact. A reader that threw there would leave every open episode untouched and then report all of them as ended on the run after.

Reading it

Twenty-seven collections, seventy-nine adapters.

https://nichedb.dev/c/water
https://nichedb.dev/c/aviation

nichedb 0.4.0: four feeds that only mean something together

chovy — Wed, 09 Sep 2026 19:16:26 +0000

nichedb 0.4.0: four feeds that only mean something together

nichedb 0.4.0 adds three collections and thirteen sources. Every one is keyless, every one was queried live before it was written, and the reason they shipped together is that none of them is worth much alone.

Aviation: why the flight is late, and what happened the last 31,000 times it went wrong

The FAA publishes, every couple of minutes, the traffic management initiatives in force right now. Ground stops, ground delay programs, airspace flow programs, airport closures. It is the document the airlines read and it is about two kilobytes.

It is also a snapshot rather than a log. It tells you a ground stop exists. It never tells you one ended, and nothing anywhere records how long it ran. So nichedb holds the open programs between polls and writes each one a second time when it disappears, with the duration attached. "BOS ground stop, 3h 40m, thunderstorms" is a fact about a day of flying that exists only if something was watching the whole time.

Two feeds sit next to it. Every SIGMET and AIRMET in force over the country, and the decoded observation at each reporting airport: flight category, wind, visibility, cloud layers. When the FAA says ORD is stopped for thunderstorms, the observation at ORD in that minute and the convective SIGMET drawn over the field are right there.

Then the part that makes it a record rather than a status board: all 31,124 accident and incident investigations the NTSB has published. The NTSB has no working query API, so this reads the bulk Access archive it rebuilds monthly. Inside, the events table carries a column holding the raw METAR at the moment of each accident, in the identical format the hourly feed publishes today. The same field, from the same service, thirty years apart.

An NTSB investigation is published long before it is finished. A preliminary report within days, a probable cause that can take two years, all under one event id. So the archive is deliberately re-read and each accident updated in place. In the newest slice, 6 of 300 have been ruled on. Eight thousand rows deeper it is 307 of 400. Those rows arriving later are the reason to re-read at all.

Water: too much and too little, measured rather than forecast

Every NOAA river gauge at or above its action stage, observed and forecast kept as separate rows, because "is it flooding" and "will it" are different questions and a forecast that turns out wrong belongs next to the observation that contradicted it.

Coastal water levels are the harder half. A tide gauge returns a number of feet and nothing else, and whether 3.1 feet is a Tuesday or a flooded car park depends entirely on the station. The thresholds live in a different service, one call per gauge, so nichedb reads them once and keeps them. A reading arrives as "minor coastal flooding, 0.13 ft over stage" instead of as a number with no scale.

Underneath both, the US Drought Monitor, redrawn weekly. A gauge says what is happening this hour. The drought map says what the last six months did to the ground that river runs through, which decides whether the next storm runs off or soaks in.

Consumer finance: what people say their bank did, and who the bank actually is

The Consumer Financial Protection Bureau publishes every complaint Americans file about a bank, a lender, a credit bureau or a debt collector. 17.6 million of them, around seven thousand a day, live within a day or two of being filed.

Each names a company as a bare uppercase string with no identifier of any kind attached. So the collection carries the FDIC register beside it: every insured institution with its charter, regulator and assets, and every merger, failure, conversion and branch opening or closing on it. A complaint about a bank absorbed eighteen months ago belongs to whoever absorbed it, and the structure feed is the only public record of that.

One thing worth knowing before reading any count from this dataset. Of the 10,000 complaints filed on 7 and 8 September, 9,843 were about credit reporting and one was about a mortgage. It is overwhelmingly a record of disputes with the three credit bureaus, and any comparison across products that does not say so is misleading.

What the live data actually decided

Half the work in a release like this is finding out that an upstream is lying to you politely.

The CFPB search accepts an offset, echoes it back, and ignores it. Three requests at three different offsets return byte-identical pages. The first version of the reader paged that way and stored the same 500 complaints ten times over while reporting 5,000 new rows. It walks days instead.

A METAR bounding box silently caps at 400 stations. A box over the continental United States returns exactly 400. Its eastern half returns 247 and its western half 240. Nothing in the response says it was truncated. The reader now splits a box whose answer comes back at the cap: 1,101 stations where the single box had claimed 400.

NOAA's river gauge service accepts state, office and region filters, ignores all three, and answers with all 12,000 gauges, which is the kind of failure that looks exactly like success.

FDIC transaction numbers are not row identifiers. One merger writes a row for every institution and office it touches. Keyed on the transaction number, a third of the register overwrites the rest, and the loss looks like a quiet quarter.

And the NTSB writes its dates as 08/22/26. Handed that, a date parser returns 1926. Every accident in the file would publish a century early and sort ahead of everything else in the database.

Reading it

Twenty-seven collections, seventy-six adapters.

https://nichedb.dev/c/aviation
https://nichedb.dev/c/water
https://nichedb.dev/c/consumer-finance

Podcasts, split by who serves the feed

chovy — Wed, 09 Sep 2026 14:36:06 +0000

Every podcast directory is a search box over one undifferentiated pile. Apple has about 2.5 million shows and one ranking. Spotify has the same shows and a different ranking. Neither will tell you the thing that actually sorts the medium in two, because both are in the business of the half that pays.

That split is: who serves the feed. A show either sits on a service that hosts thousands of other shows, or it sits on the domain of the show itself. One side has a company and a hosting bill behind it. The other side is a person publishing from their own site.

NicheDB now has a podcasts collection, and that split is the whole thing. Two sources, podcasts-commercial and podcasts-self-hosted, disjoint by construction.

Measuring it instead of guessing

Nothing in a feed declares which side it is on, so we counted. From the Podcast Index bulk dump of 2026-08-23: 4,718,246 feeds, of which 421,928 were live (HTTP 200 on the last crawl, an episode inside 90 days).

307 registrable domains carry 25 or more of those live feeds. Between them they hold 395,690, which is 93.8% of the live catalogue. anchor.fm alone is 97,805. The remainder, about 13,400 domains, is the self-hosted half.

A hand-written list of "the podcast hosts" would have been a list of the ones we had heard of. The long tail here is German church networks, Japanese radio stations and Czech public broadcasters. Counting finds those. Memory does not.

The trap that would have quietly emptied it

The Podcast Index host column sometimes holds a bare public suffix, co.uk or com.br or org.au, rather than the registrable domain. Every one of those clears a 25 feed threshold.

Left in the platform list, a suffix match files every .co.uk podcast in Britain as commercially hosted, and the independent half goes to nearly zero. That is 453 feeds whose real host we do not know. They are dropped, which sends them to the self-hosted side. When the whole point is finding independent shows, the error you want is the one that keeps them.

The bug we shipped, and what it taught us

The collection went live reading down from the newest entries of the upstream directory until it met the last run's marker. That is right for keeping up and useless for starting. It sat at 2,000 items: 1,988 self-hosted and 12 commercial.

Nothing errored. No run failed. It just would never have grown.

The reason is worth writing down. We sampled the directory's listing at ten evenly spaced depths. The first page is 2 commercial out of 200. Every other depth runs 178 to 194 out of 200. Overall, 84%.

The head of that list is not a sample of the catalogue. It is inverted against it. Recent arrivals are whatever was submitted lately, and lately that has been almost entirely independent shows, while the bulk behind them came from old bulk imports and looks nothing like it. So the commercial source read 2,000 feeds, found 12, and roughly 230,000 sat three pages further down.

The fix is a walk: page forward through the whole listing once, then switch to keeping up. Drift helps here rather than hurting. The listing grows at the head, so a row at offset N moves to N+k while you walk, which means you re-read rows you have seen instead of stepping over rows you have not. Re-reading costs nothing. Skipping would have been silent.

First run after the fix: the commercial source went from 12 items to 11,490. Second run: 30,224.

Where it is

Both halves are at nichedb.dev/c/podcasts, with feeds for each and language cuts of the independent side. Sources poll every 15 minutes, which matches the upstream directory's own floor for a feed that just published.

The commercial half is already addressable in every podcast app there is. The other half is not addressable anywhere, which is the only reason to build this.

NicheDB 0.2.0: the domain layer, counted daily

chovy — Wed, 09 Sep 2026 10:45:05 +0000

Twenty seven commits had merged into NicheDB since the first release and none of them had reached anybody. The CLI publishes from a GitHub release and from nothing else, so a merge to main is a midpoint rather than a finish. 0.2.0 cuts them loose.

The headline is a new collection: domains.

The layer everything else runs on

NicheDB tracks packages, filings, launches, weather, markets, crime, public money, housing, jobs and news. It had no view of the layer all of that lives on, which is the domain name system's own market. It does now, built from ntlddata.com's free JSON API. No key, CC-BY, and aggregate only, because ICANN's zone file terms permit analysis but not redistribution.

Eight feeds, each with a page, RSS, JSON Feed, an API endpoint and MCP tools:

How many domains exist across the new gTLDs and which way that moved, day by day, with the pre-2012 generics counted separately
The whole register of 1,114 TLDs with its registry operator, inferred backend, root signing and delegation date
The biggest daily gainers and the TLDs losing the most names
Sunrise and claims windows with the dates registries filed with ICANN, strings contracted but not yet delegated, and registries being transitioned to an emergency operator
TLDs changing operator, registrars moving platform, accreditations granted and terminated

What the collection refuses to conflate

Two clocks. Domain counts come from zone files and are a day old. Anything attributed to a registrar comes from ICANN's monthly reports and runs about a quarter in arrears, because zone files carry no registrar information at all. Every figure of the second kind is carried with the month it describes and is never merged with the daily count.

Two populations. New gTLDs and the pre-2012 generics are never added together. A source names one and labels its rows with it.

Three traps, found by probing rather than by reading

The overview endpoint never answers without an explicit scope, and never answers for scope=all. Both hang until the client gives up, while scope=new returns in under a second. So the totals source refuses scope=all rather than scheduling a guaranteed timeout on every run.

The page size parameter is floored at ten. Asking for three rows returns ten while the response's own metadata says three, so a "top five movers" source that trusted the parameter would publish ten.

Detected timestamps carry no zone, and a bare Date parse reads them as local time, which files a midnight row under the previous day for every reader west of UTC.

One more came from the live data rather than the docs. The first day of the daily series carries zero adds, zero drops and zero net, because it had no day before it to difference against. Across a thousand registries that cannot be a real day, so it publishes as movement not yet known instead of "+0 on the day", which would read as a flat day rather than a missing one.

Everything else in 0.2.0

Weather as it is issued rather than forecast. Automotive with a VIN history report scored against a measured complaint baseline. Markets with the world's exchanges as a register and the corporate actions that move them. Crime by city, state and force area. Public money across three governments. Housing and jobs as governments measure them. AI incidents. News, where every story files under the desk that published it.

Plus the fixes worth naming: dedup now folds a source against its own second feed, not only against the other sources, so a publisher exposing one article through two of its feeds stops arriving twice. The dedupe keys reach Postgres as a real array rather than a comma joined string, which had been aborting whole ingestion runs.

Reading it

The collection is at https://nichedb.dev/c/domains and every feed on it is public. The CLI is npm i -g @profullstack/nichedb@latest.

Data on the domains collection is from nTLDData (ntlddata.com), CC-BY-4.0.