JSON Feed is comfortable. It is also bigger than it needs to be.
While building a feed engine that emits six formats from one canonical model, I wanted a native format that stayed human-readable but stopped paying for the same strings over and over. The result, NWF, comes out roughly 62% smaller than the equivalent JSON Feed, and it still round-trips losslessly back to the same model.
Three tricks get it there. None of them are gzip. You can still open the file and read it.
1. Intern repeated values. Authors, tags, and sources are listed once and then referenced by index. A feed of 50 posts by one author stores that author string one time, not fifty.
2. Relativize links. Every URL is stored relative to a shared base prefix instead of repeating https://your-domain.com/... on every single entry.
3. Store dates as deltas. Instead of a full ISO timestamp per item, each date is a small delta from the feed's own update time.
The wire format itself is line-oriented on purpose: LF-separated lines, TAB-separated cells, a one-letter kind per line. Media type text/x-neurowire, extension .nwf. Compact enough for machines, still legible when you are debugging it by eye.
Why bother when gzip exists? Because interning and relativizing shrink the model before the transport layer ever sees it, and the file stays diffable and greppable in a way a gzipped blob never will. You get the size win and keep the ability to eyeball a feed in your terminal.
The full write-up covers the rest of the engine: the "taps" that turn feed-less HTML pages into real feeds, "meshes" that merge multiple sources into one, and why one canonical model beats writing the same brittle scraper twice. It's here: https://mederic.me/blog/neurowire-docs-launch
If you have ever designed a wire format, what did you strip out first: the repeated strings, or the timestamps?
Top comments (0)