Two months ago a repository turned up in my corner of the internet: 3,358 stars, 744 forks, an MIT license, and a release advertised as 518,400 training samples — 5.5 GB, split across three zip parts. My corner is Chinese astrology software, a domain with almost no machine-readable data, so a corpus that size was either the most useful thing published in years or it was nothing at all. I wanted to know which.
Then I looked at the sample count for one second longer. 518,400 = 60 × 12 × 30 × 12 × 2. Sixty years, twelve months, thirty days, twelve two-hour periods, two genders. That isn't a tally of things somebody observed. That's the size of a nested loop.
Which is a wonderful reason to be curious and a terrible reason to conclude anything. So I went and read the archive — 48 MB of it, over HTTP range requests, never touching the other 5.83 GB. The trip turned out to be more interesting than the destination, and the method is the part that travels well beyond my strange little niche.
So: here's how you read a multi-gigabyte release for the price of a few photos, and then the four things it told me.
First, the trick: reading a 5.5 GB release without downloading it
Three facts make this work, and they're pleasant ones to know:
- A split zip made with split is plain concatenation. The parts aren't independent archives. Part 1 opens with the first local file header, and the byte offsets in the central directory are absolute across the whole set.
- The central directory sits at the end and lists every entry: name, uncompressed size, compressed size, local-header offset. Fetch the last couple of megabytes and you're holding the table of contents.
- GitHub release assets honour Range requests. Once you know an entry's offset and size, you can ask for exactly that entry and nothing else.
There was a bonus in this particular archive. It stores its .jsonl.gz shards with method 0 — stored, not deflated — because they had already been gzipped. Every entry is independently addressable, and the small text files at the tail sit there uncompressed, so a plain curl -r hands you readable source code.
# 1. the table of contents: last 3 MB of the last part
curl -sL -r 1890639808-1893639807 \
".../ziwei-samples-v3-part3.zip.003" -o p3.tail
# then find PK\x05\x06 (EOCD), walk the PK\x01\x02 entries,
# and honour the ZIP64 extra field — cdoff came back as 0xFFFFFFFF
# 2. one 8 MB data shard, at the offset the directory gave me
curl -sL -r 48646942-56753200 \
".../ziwei-samples-v3-part1.zip.001" -o feb.bin
917 entries came back. 781 of them are the data everybody downloaded it for. The other 136 are where the story turned out to be.
Total fetched: 47,897,905 bytes — 0.81% of the release. Full script in the appendix.
Finding 1: the release contains the file the repo says it held back
The public repo ships lib/ziwei/db-analysis.ts as a 2,124-byte stub, and its header comment is refreshingly direct about why (my translation):
The analysis content library is not part of the open-source scope. The full online version contains detailed readings for 14 major stars × 13 palace contexts — core content, not open-sourced along with the chart engine.
And at the bottom: export const STAR_DB: Record<string, unknown> = {};
The same path inside the release is 377,157 bytes, with STAR_DB populated starting at line 161: 524 string literals, 145,580 characters of hand-written Chinese prose. The library that was held back from git went out in the release anyway — just parked 5.87 GB deep inside a three-part split zip, where nobody was looking.
That's the first and most portable lesson, and it has nothing to do with astrology: redaction leaks through large binary artifacts. A file you carefully stubbed in git will cheerfully ride along in the release tarball, the Docker layer, the model checkpoint, the training-data dump. The bigger the artifact, the less likely anyone notices — which is exactly backwards from the way the risk actually runs.
Finding the source also settled what the dataset is. Across the whole lib/ziwei/ directory there are 216,172 characters of TypeScript, 183,182 of them inside string literals. The generator is five nested loops over that grid, longitude hardcoded to 120, and — outside the audit script's own sampler — no call to Math.random anywhere. The output is a pure function of a five-integer key.
Which makes the release a memo table. Uncompressed it comes to roughly 32.9 GB (I measured one shard at 45,649,978 bytes, and there are 720 of them), holding something like 1.09 × 10¹⁰ characters of generated prose, all of it assembled from 183 K characters of source — source that ships in the same zip, next to the npm run full command that regenerates the whole thing.
Measured reuse, on the 3,600 samples I pulled (0.69% of the corpus): 46,800 emitted topic blocks, 16,359 distinct — and each of those is a permutation of the same literals. Fine-tuning on this doesn't teach a model a domain. It pays a GPU bill to compress a template engine into weights, badly, when you could simply call the engine.
| Release size | 5.5 GB (3 parts) |
| Uncompressed | ~32.9 GB |
| Generated prose | ~1.09 × 10¹⁰ chars |
| Hand-written source it came from | 183,182 chars |
| Math.random in the generator | none |
Finding 2: 2,520 of the samples are dates that never happened
days: range(1, 30) — thirty days for every month, and only ever thirty.
That has two consequences. Every 31st is missing: seven months × 60 years × 12 hours × 2 genders = 10,080 real birthdays with no row at all. And February gets days 29 and 30 regardless of the year, which leaves 2,520 rows keyed to dates that never existed.
I couldn't resist pulling February 1962, a non-leap year, to see what the generator does with an impossible input. It doesn't fail. It rolls straight past the end of the month and keeps counting:
| Gregorian input | lunar day the engine assigned |
|---|---|
| 1962-02-27 | month 1, day 23 |
| 1962-02-28 | month 1, day 24 |
| 1962-02-29 (not a date) | month 1, day 25 |
| 1962-02-30 (not a date) | month 1, day 26 |
So the sample labelled 1962-02-30 is a complete, confident, 63 KB chart — for a person born on March 2nd. Nothing anywhere in the pipeline noticed.
Two more shape quirks live in the same grid. The year axis is range(1924, 1983), with a source comment explaining that "60 years covers a full sexagenary cycle" — the window was chosen to make the number come out round, not to cover anybody's users. Nobody born after 1983 is in here, which in 2026 makes it a corpus exclusively about people aged 43 and up. And longitude is 120 on every single row, which stings a little: in a system where the birth hour is the highest-entropy input, the one variable most worth varying got pinned to a constant.
Finding 3: "518,400 / 518,400 validated" turns out to be a substring check
The release ships its own validation and audit logs, and they look great. Zero failures, zero warnings, lines like these:
health contains 「liver/kidney/spleen-stomach」: 518400/518400 (100.00%)
health contains 「子午流注」 and 「經絡」: 518400/518400 (100.00%)
female health contains 「gynaecology/menses/pregnancy」: 259200/259200 (100.00%)
male/female wealth topic pairs differ: 259200/259200 (100.00%)
failures: 0 warnings: 0
Every one of those is String.includes. They establish that a template fired. They cannot tell present apart from correct — and here that distinction is load-bearing, because five lines earlier in the very same log:
daXians[0] contains siHua: 0/518400 (0.00%)
any daXian contains siHua: 0/518400 (0.00%)
daXians[0] contains stemIndex: 0/518400 (0.00%)
samples containing palace.selfSihua: 0/518400 (0.00%)
The spec handed to the generating agent — README-CODEX.md, also in the archive — calls one of those fields "the single easiest thing to get wrong" and makes it hard requirement #6, with a post-generation checklist that says to verify it. I went and checked the shipped records directly: each daXians entry has exactly four keys — startAge, endAge, palaceBranch, palaceName. The field the spec built itself around simply isn't there.
The packaging manifest, meanwhile, lists this in English under the heading "Verified results":
Verified results:
- Total samples: 518,400
- Validation failures: 0
- daXians[].siHua: absent
- daXians[].stemIndex: absent
- palace.selfSihua: absent
An earlier audit report, still sitting in the box, shows those same fields populated with real values across 20 sampled rows. So somewhere between that run and the shipped one, the three most important computed fields went missing, and the validator scored their absence as a pass. The scorecard file then awards 10/10 to the dimension "hard-constraint implementation and audit," on the stated basis that verification commands exist for each constraint.
One more from this log, because it's the detail I keep thinking about. The audit counts how often each of twelve canned warning phrases appears, and three of them land at exactly 518,400 — including the sentence "if the fortune palace is afflicted and the spouse palace shows no separation in life, then it must be separation by death." That line is meant to be conditional on a configuration most charts don't have. It's in 100% of the corpus. A model trained on this learns to tell everybody they'll be widowed.
Finding 4: a file in the archive audits the citations, and it's brutally honest
Tucked into the release is corpus/annotations.json — a review of every quotation in the analysis library attributed to a named teacher, a physician-lecturer who died in 2012. Here's its own summary, in its own categories:
| status | count | the file's definition |
|---|---|---|
| verified | 18 | appears in the lecture transcripts, source checked |
| traditional | 42 | generic classical maxim, attributed to him but not his |
| suspect | 15 | source doubtful, possibly another school or later invention |
| fabricated | 6 | "highly likely invented by a previous author or by me on the spot" |
| methodology | 14 | concept correct, not a quotation |
| total | 95 |
19% of the quoted attributions are traced. The notes are first-person and completely unsparing — "this line I made up, should be changed to 'the Northern school holds…'"; "'precise month-derivation method' is a name I invented, not his term"; "⚠️ major error: his illness readings key on the palace branch, not the star element — this whole mapping deviates from the system."
There's also a top_priority_fixes array. One P1 item is the fortune-palace sentence I quoted above, flagged as probably not his words and in need of a rewrite. It is the sentence that appears in 518,400 of 518,400 samples. The fix list shipped inside the artifact it was supposed to gate. (The generated prose also attributes it, and dozens of other lines, using a misspelling of the teacher's name.)
And here's the part that makes this worth writing up rather than piling on. That annotations file is more provenance diligence than almost any repo in this space has, and the maintainer wrote it themselves, about their own text, before shipping. Sitting next to it is a rights policy that sorts collected material into four tiers, forbids storing full text without a license, and forbids rights_status=unknown material from entering trainable corpora. The source registry reads: 211 sources, 1 cleared for full text, 167 needing a license, 15 prohibited outright. Somebody thought hard about all of this.
So the failure isn't missing diligence. It's that the diligence was never wired to a gate. A file listing six fabricated citations doesn't block a release. A policy saying "unknown rights don't enter training corpora" doesn't stop a corpus derived from that material from going out under MIT. A validator that greps for a string can't fail a build for being wrong. Every artifact of trust in that box was produced by the same pipeline it was meant to check — the agent generated the data, ran the audit, and reported success, and each of those three steps was scored on whether it completed.
That's the 2026 failure mode, and I don't think it's rare. Not slop. Slop with a passing test suite.
The three checks I run now
- Factor the sample count. If N decomposes into small round factors, you're looking at a grid — coverage of a key space rather than observations of anything. Ask what the unit of observation is. If the answer is "a possible input," there's no signal to learn, and no outcome, adjudication, or human in the loop to check a claim against.
- Look for the generator before you look at the data. If it ships — and it very often does, in the tail of the archive — the generator is the dataset, and it's five orders of magnitude smaller. Then check it for randomness. No randomness means the release is a memo table of a pure function, and you should just call the function.
- Read the validator, not the validation. grep for what it actually asserts. Substring presence, field presence, and file counts are shape checks; they pass with equal enthusiasm on correct output and inverted output. A green log tells you nothing until you've seen the predicate. Mine assert relations between computed values, for what that's worth, and I still don't trust them as far as I'd like.
A small footnote on why any of this needed doing. The download counts on that release: 10,814 for part 1, 7,506 for part 2, 7,989 for part 3, and 1,250 for the checksum file. About 30% of the people who started never finished, and roughly one in ten verified what they got. A 5.5 GB three-part download is its own kind of moat — nobody reads what they can't open, and 3,358 stars measured the appeal of the claim rather than the contents.
The audit cost 48 MB.
Why I was poking at it at all: I build software in the neighbouring system — BaZi rather than Zi Wei Dou Shu (auspiceoracle.com) — so this corpus landed on my desk as something I might use. I've deliberately left the repo, the tag, and the file paths out of the prose, because the point here is the method rather than the maintainer. I'm also not drawing any licensing conclusions: I'm not a lawyer, and every count I've quoted is the repo's own. Everything else reproduces in about two minutes with the script below, and I'd much rather be corrected than believed.
Appendix: reproduce it
import struct, json, urllib.request
REL = "https://github.com/{owner}/{repo}/releases/download/{tag}/"
PART = ["...part1.zip.001", "...part2.zip.002", "...part3.zip.003"]
SIZE = [1992294400, 1992294400, 1893639808] # from the releases API
def grab(part, start, end, out): # HTTP range fetch
req = urllib.request.Request(REL + PART[part],
headers={"Range": f"bytes={start}-{end}"})
open(out, "wb").write(urllib.request.urlopen(req).read())
# 1. central directory: tail of the last part.
# find PK\x05\x06 for the EOCD, then walk PK\x01\x02 entries.
# cdoff == 0xFFFFFFFF means ZIP64 -> read the real values from
# extra field header id 0x0001 (usize, csize, local-header offset).
grab(2, SIZE[2] - 3_000_000, SIZE[2] - 1, "p3.tail")
# 2. any entry, given its absolute offset from the directory:
# absolute -> (part, local offset) via the cumulative sizes,
# then re-read the local header to skip name+extra, and the
# remaining csize bytes are the stored member, byte for byte.
# Entries with method == 0 (stored) need no decompression —
# the .jsonl.gz shards drop straight into gzip.open().
Three things to take away, none of them about astrology. Redaction leaks through big artifacts. Grids aren't data. And a green audit log is a claim like any other — it's worth asking what it measured.
All numbers measured 2026-08-05 against the release as published.
Top comments (0)