nichedb 0.5.0: the sea, the surf and the aircraft worth watching
Three more sources, all keyless, all verified against the live upstream before a line of the reader was written.
The whole ocean in one file
The National Data Buoy Center publishes the newest observation from every buoy it is listening to in a single hundred-kilobyte file. 853 reports when I read it, 184 of them carrying a wave height, and each one holding wind, gust, pressure, air temperature and water temperature alongside.
A surf report is three of those numbers and everything else is presentation. How big the swell is, how far apart the waves are, and where it is coming from. The middle one is the number people leave out, and it is the one that decides the day: two metres at 18 seconds and two metres at 5 seconds are not the same sea at all. So nichedb publishes a wave row as "8.9 ft at 8 s, mixed swell out of the W" as well as storing the numbers, and it stores the height in feet as well as metres, because every surf forecast in the United States is in feet and converting at read time is how a nine-foot day quietly becomes a three-foot day.
The forecaster's own words next to the buoy's numbers
A buoy off Oahu reading 2.4 metres at 16 seconds is a fact. "Advisory level surf along south facing shores through today" is what that fact means to somebody standing on the beach, and no amount of arithmetic over the buoy produces it.
So the surf zone forecast the National Weather Service issues from every coastal office is in there too, in full. The high surf advisory and the rip current risk are lifted out of the forecaster's own wording into tags, so a coast under an advisory can be found without reading all 851 products, and the text itself is kept as the authority.
Aircraft, but only the ones that mean something
There is a version of this that stores every aeroplane in the sky every few minutes. It would be tens of thousands of rows a day, each one a position that was true for four seconds, and nobody would ever read a single one of them.
A position is not news. An aircraft squawking 7700 is. So nichedb reads the event surfaces of the ADS-B network instead: the three emergency transponder codes, and the military list. When I wrote this the emergency codes returned zero aircraft between them and the military query returned 422, which is exactly the shape you want. The rare thing is rare, so its appearance means something.
And an emergency is an episode rather than a ping. An aircraft squawking 7700 shows up in poll after poll and then simply stops showing up, and nothing anywhere records that it stopped. So the open episodes are held between polls, the row is keyed on when the aircraft was first seen, and when it leaves the list it is written once more with a duration on it. "N123AB squawked 7700 for 22 minutes" is the row worth keeping. The forty positions in between are not.
Four things the live data decided
NDBC writes a missing reading as MM, in every column, in a fixed-width table. Anything that coerces that to a number turns "this buoy has no anemometer" into a flat calm and "no wave sensor" into a dead flat sea, and 669 of the 853 stations report no wave height at all.
The station register really does publish an empty name. Station 15009 in the Atlantic array is one of several, and an empty string left alone beats the fallback and titles the row with nothing at all. That showed up in the very first smoke run as a list of observations called ": marine observation".
Some stations report a dominant period of 0 or 1 second. That is a sensor saying nothing, and calling it windswell states something about the water that nobody measured.
And the ADS-B network rate limits its military list far harder than its squawk lists, answering 429 while the endpoint next to it answers 200. A rate limit is not a failure, so it returns nothing and asks to be called back later, which also keeps the cursor intact. A reader that threw there would leave every open episode untouched and then report all of them as ended on the run after.
Reading it
Twenty-seven collections, seventy-nine adapters.
Top comments (0)