DEV Community: Shan Liu

Your access log already knows whether ChatGPT is citing you

Shan Liu — Sat, 12 Sep 2026 22:00:19 +0000

There's a lot of guessing about whether AI systems are citing your site. Analytics won't tell you — most of these referrals arrive with no referrer at all.

Your access log will, and it's more specific than people realize: two different classes of user-agent mean two completely different things, and only one of them is evidence that you were actually cited.

I've been logging this on a small site since launch. Here's the mechanism, the distinction that makes it useful, and the trap that will make you over-count.

The distinction that matters

Group the AI user-agents you see into two buckets:

Bucket 1 — crawlers. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Claude-SearchBot, Applebot-Extended, CCBot. These are building or refreshing an index. A hit here means you're in a crawl queue. That's necessary, it's not an outcome, and it says nothing about whether anyone ever saw your content in an answer.

Bucket 2 — user-triggered fetches. ChatGPT-User, Perplexity-User, Claude-User. Different job entirely: a person was in a conversation, and the assistant fetched your page to answer them — either browsing live or following a citation the user clicked.

That second bucket is the closest thing to direct evidence that your page participated in an answer. Naming convention across vendors is conveniently consistent: -Bot and -SearchBot suffixes are infrastructure, -User suffix means a human is on the other end.

Most write-ups on this topic lump all of these into one "AI traffic" number, which throws away the only distinction you actually care about.

Capturing it without a log pipeline

My stack has no nginx in front of the app, so there's no access log on disk to grep. If you're in the same position, you don't need to add one — the edge proxy can log the hit itself.

The whole mechanism is a regex and a console.log in the request path:

const AI_BOT_RE =
  /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|anthropic-ai|PerplexityBot|Perplexity-User|Google-Extended|Applebot-Extended/i

const match = (request.headers.get('user-agent') ?? '').match(AI_BOT_RE)
if (match) {
  console.log(`[ai-bot] ${new Date().toISOString()} ${match[0]} ${request.method} ${pathname}`)
}

Then whatever collects your stdout is your dataset:

# which agents, how often
awk '{print $3}' archive.log | sort | uniq -c | sort -rn

# which pages they took
awk '{print $3, $5}' archive.log | sort | uniq -c | sort -rn

# the evidence bucket — usually few enough to read line by line
grep -E 'ChatGPT-User|Perplexity-User|Claude-User' archive.log

Two things worth doing before you trust the numbers:

Archive out of the log buffer. Process managers rotate, restart, and flush. A -User hit may appear exactly once, ever — if it lands in a buffer that gets cleared, it's gone and you'll never know it happened. A small script on a periodic cron that appends new lines into an append-only file, deduped, is enough. Make it idempotent so re-running is free.

Check what your matcher excludes. Mine initially skipped llms.txt, robots.txt, and sitemap.xml, which are precisely the files an AI crawler hits first. Those counts read as zero for two days and the zero was an artifact. If your proxy has a path matcher, audit it against the paths bots actually want.

The trap: fake AI user-agents

Here's the part I hadn't seen written up anywhere, and it will corrupt your data if you don't handle it.

Some of your "AI crawler" traffic is a vulnerability scanner wearing a costume. One 83-second burst on my site: 60 requests rotating through 6 different AI user-agent strings, hitting paths like /aws-credentials and /id_rsa, plus query strings carrying shell-injection probes.

Spoofing a user-agent is trivial, and AI crawler strings are a good disguise precisely because everyone has decided to allow them.

If you count those as AI interest, you'll conclude the robots love you. Tag them instead:

const SCANNER_RE =
  /\/(?:etc\/passwd|proc\/self|cgi-bin\/|id_(?:rsa|dsa|ed25519)|private-key|credentials|secrets|api-keys|wp-(?:admin|login)|phpmyadmin|xmlrpc)|[?&](?:cmd|exec|command)=[^&]*(?:%3B|%60|%7C|;|`|\|)/i

const tag = SCANNER_RE.test(`${pathname}${search}`) ? 'ai-bot-scan' : 'ai-bot'

Two deliberate choices there. Blacklist suspicious paths rather than whitelisting your real ones — if a blacklist misses something you get a few junk lines, but if a whitelist misses a page you just shipped, it misclassifies a genuine citation as a scan. One failure mode is noise; the other is silently destroying your only copy of a signal that occurs once.

And keep the scan lines, just tagged differently. They're still useful for security review; they just don't belong in the citation count.

Five weeks later, the trap was most of my data

I wrote the section above from a single 83-second burst. Then I left the log running and came back on 2026-09-12 — 3,392 lines, five and a half weeks. The naive count:

User-agent	Raw hits
ChatGPT-User	1,462
ClaudeBot	872
OAI-SearchBot	443
PerplexityBot	385
GPTBot	177
Claude-User	19
Claude-SearchBot	10
Perplexity-User	6

1,487 *-User hits. Read straight off the log, that says a human-driven assistant fetched my pages fifteen hundred times. It says nothing of the kind.

Roughly 1,330 of those — about 89% — are one machine on a timer. The tell is in the path distribution: / 665 times and /en 664 times — near-perfectly paired, and together dwarfing every real page on the site by two orders of magnitude. Pulling the timestamps for the root hits alone: 661 of them since 2026-08-11, median gap 49 minutes, 84% of gaps falling in a 25–95 minute band, with the / and /en requests fired 0.1–0.7 seconds apart. People clicking citations in conversations do not arrive on a metronome, in pairs, on two URLs.

The rest of the costume is more familiar. Scanner probes wearing *-User strings went after cloud credentials across all three major providers, plus the usual file-read attempts — the AWS link-local metadata address for IAM info, the Azure OAuth token endpoint, the GCP service-account token path, ..%2F..%2F.env, and file:///proc/self/environ. Claude-User asked for /Dockerfile and /metrics. Perplexity-User asked for /log-viewer.

Strip the metronome, the scanner paths, robots.txt and the login probes, and what remains is on the order of 80 *-User hits that actually landed on content — spread across the true-solar-time explainer, the day-master and how-to-calculate pages, the method page in both languages, the case studies, and the conformance test.

Eighty is a real number and a much better one. It is also about 5% of the raw count. That is the size of the error you inherit by trusting the user-agent string: not a rounding difference, a factor of nineteen — and it points the wrong way, flattering you. Had I shipped a dashboard in August that plotted *-User hits without filtering, it would have shown a beautiful curve, and every point on it would have been a lie.

The lesson isn't that logs don't work. It's that the filtering is the measurement. The regex that finds AI user-agents is the easy half; the half that decides which of those hits are real is where the signal actually gets made.

What it actually looked like

Small numbers, and I'm going to keep them small rather than dress them up. Three readings from one site:

Reading	GPTBot	ClaudeBot	PerplexityBot	`*-User`
Pre-launch baseline (2026-08-03)	1	11	0	0
Launch day, first proxy log	0	10	0	0
Two days later (2026-08-05)	9	10	2	2

Three things showed up in that data that I would not have gotten any other way.

Crawlers behave differently from each other. On launch day one crawler hit all ten new pages exactly once each, no repeats — walking a list. Another arrived two days later and only touched the homepage and one section, which reads like discovery rather than indexing.

The first -User hits are legible individually. With two of them, you read the lines:

Time (UTC)	Agent	Path
2026-08-04 15:47	ChatGPT-User	`/zh/method`
2026-08-05 04:29	ChatGPT-User	`/en/method`

Both landed on the methodology page — the one documenting how the product computes what it computes. Neither hit a content page. With n=2 I'm not going to build a theory on it, but it's a specific, checkable observation about which page an answer engine went to get facts from, and it pointed me at something I'd have deprioritized.

Zero is information, if you took the baseline first. The pre-launch row exists because I recorded it before shipping. Without it I couldn't distinguish "no one is citing us" from "we've always looked like this."

The honest limits

Spoofable, and not conservatively so. Everything above is a self-reported header. The scanner section and the metronome are the proof. It's tempting to assume spoofing means you're under-counting — the five-week data says the opposite: the raw number was ~19x the filtered one, inflated by traffic pretending to be an assistant. An unfiltered count is not a floor. It isn't a ceiling either. It's just not a measurement.
A -User hit is not a citation you can quote. It means a page was fetched to serve a conversation. You don't get the question, the answer, or whether you were quoted or contradicted.
Small n stays small — and big n is usually small wearing a costume. Two hits is two hits. Fifteen hundred hits was eighty hits and a monitoring bot. Neither number gets to become a trend just because it's convenient.
Your CDN may absorb it. If a cached response never reaches your app, your proxy never logs it. Worth checking whether your edge is answering on your behalf.

Why bother

Because the alternative is inferring your way to a conclusion you like. The AEO advice market is full of tactics with no measurement attached, and most of them can't be evaluated by the person selling them either.

This costs one regex, one console.log, and a cron job. It won't tell you what the model said about you. It will tell you, with dates, whether anything on the other side ever came to your server to find out — and which page it wanted.

If you set it up, take the baseline reading before your next launch. That's the row you can't reconstruct later.

The page those first two fetches went for is the method page — the computation, the constants, and where the numbers come from. The product is auspiceoracle.com.

I Spent Three Weeks Doing SEO Wrong. One Missing Function Was the Whole Problem.

Shan Liu — Sun, 30 Aug 2026 23:29:15 +0000

Everyone tells you SEO is a long game. Publish, wait, compound. That advice is true and it almost ruined my launch, because it gave me a comfortable explanation for a problem that wasn't slow — it was broken.

Here's what actually happened, and the growth lesson I'd hand to anyone launching something new.

The symptom

I launched. I wrote real pages — long ones, researched, not filler. I submitted a sitemap. Google accepted it. Status: Success.

Then nothing. Weeks of nothing.

Search Console showed my most important page — the actual product page — as "URL is unknown to Google." Not "low quality." Not "crawled, not indexed." Unknown. As if it didn't exist.

The comfortable story was right there: new site, no authority, be patient. I almost took it. That's the trap. "SEO takes time" is the most expensive excuse in growth, because it's indistinguishable from "your site is silently broken."

The actual cause

I stopped guessing and inspected every URL individually.

Three of my content pages came back "Duplicate, Google chose different canonical." Google had decided my pages were copies of something else. My product page wasn't even that — it was invisible.

The cause was one missing function.

My framework generates page metadata per route. The product page didn't define any. So it silently inherited the parent layout's — including the canonical tag. Every visitor and every crawler that hit that page was handed a tag that said, in effect: this page is a duplicate of the homepage, don't index it.

I had spent weeks writing content for a page that was telling Google not to index it. No error. No warning. Valid HTML. Perfect Lighthouse score. A sitemap reporting Success.

That's the part worth internalizing. The bug didn't look like a bug.

What I fixed, and how I proved it

I found three more problems in the same audit:

Redirect chains. www → apex, http → https, trailing slash, and a legacy ?lang= parameter each triggered its own redirect. Chained together, a crawler hit four hops before reaching content. Crawlers give you a budget. I was spending it on hops. I collapsed all four into exactly one 301.

Language sniffing. I routed visitors by browser language. Elegant for humans, poison for crawlers — Googlebot crawls from one place, so it only ever saw one version of my site. I removed the sniffing entirely and routed on path alone.

Cross-language duplication. My English pages echoed fragments of the other language. Enough for Google to read them as near-duplicates of each other.

Then — and this is the part most people skip — I verified every single fix against production. Not locally. Not in staging. Against the live site, from outside.

Every hostname variant, every scheme, every trailing-slash and parameter combination: exactly one hop. Every sitemap URL: zero redirects, HTTP 200. I checked those same invariants again today, weeks later, and they still hold. A fix you haven't verified from the outside isn't a fix. It's a hypothesis.

The result

The product page went from "unknown to Google" to indexed, with Google selecting the correct canonical — itself. The duplicate verdicts cleared. Impressions started flowing within days.

Then the part I didn't plan.

I'd been treating classic SEO as the goal. But my first confirmed AI citation — an assistant pulling my page as a source — didn't land on a product page or a landing page. It landed on a concept page. A page that just explains an idea, patiently, with no sales motion in it.

That reframed my whole content strategy. Assistants don't cite your pitch. They cite your explanation.

The five things I'd tell you

1. Rule out broken before you accept slow. If a page isn't indexed after a few weeks, inspect that exact URL before writing another word. "Unknown to Google" is a bug report, not a waiting period.

2. Read what your pages actually ship. Not what your framework should generate — what it does. curl your own page and look at the canonical tag. Mine was lying about my most important page for weeks.

3. Count your redirect hops. From every variant: www, non-www, http, https, with and without a trailing slash. Each one should reach the destination in a single hop. This is a fifteen-minute audit that most sites fail.

4. Verify from the outside, and re-verify later. Production is the only environment that counts, and a fix that held in July can quietly regress in August.

5. Start before you feel ready. The pages I published in week one outrank the pages I published last week. Same effort. Different age. Indexing age is the one growth input you cannot buy, borrow, or accelerate later — you can only start earlier. Publishing on day one is a decision that pays out in month six.

The real lesson

The growth hack isn't a trick. It's this: when something isn't working, the most likely explanation is not the most comfortable one.

"SEO takes time" let me feel patient for three weeks while my site quietly told Google to ignore it. The moment I replaced that story with a URL inspection, the problem took an afternoon.

Patience is a strategy for things that are working. Diagnosis is what you owe everything else.

I benchmarked 8 LLMs for a niche production app. The flagship cost 5.8x more - and lost.

Shan Liu — Sat, 29 Aug 2026 01:48:02 +0000

My app generates personalized readings for BaZi — Chinese "Four Pillars" birth charts. Every reading is an LLM call, every call costs money, and the domain is full of trap terminology that models love to botch. So before launch I benchmarked every candidate model on my actual workload, and then built the routing layer around what the benchmark found.

The results generalize to any "LLM in a niche domain" app, so here they are — including the part where the most expensive model lost to one costing 5.8× less.

What "good" means in a niche domain

Generic benchmarks were useless to me. My acceptance criteria were:

Domain accuracy: 甲 is Yang Wood. A model that renders it "Yin Wood" in English output is not 5% wrong, it's categorically wrong — the way a compiler that flips one bit is wrong.
No invented jargon: the system has a closed vocabulary (the Ten Gods, fixed star names). A model that confidently introduces terms my engine never computed is a liability.
Cost per reading, because a free tier exists and every free reading is marketing spend.

Running my own corpus through the candidates produced findings no leaderboard would have surfaced:

The flagship preview won't let you turn thinking off. Hybrid reasoning models "think" by default, and the preview snapshot rejects the flag outright (400 InvalidParameter: The value of the enable_thinking parameter is restricted to True), so you pay for the inner monologue whether you want it or not. On one streamed request that was 286 reasoning events before the first character of the actual answer: 2.1s to the first reasoning token, 10.5s to the first character a user can read. On list price the flagship already costs 5.8× the mid-tier model per call; the reasoning tokens bill on top of that, for prose I could not tell apart. Excluded.
Three mid-tier models flunked domain accuracy — English chart output with elements flipped (Yang Wood → "Yin Wood" class of errors). Excluded regardless of price.
The "character roleplay" fine-tunes hallucinated worst of all — invented relationships between the Ten Gods that don't exist in the system. The models optimized for persona were the least safe choice for a persona product. Excluded.
Two well-known open-weight models had quietly been delisted from the provider's international endpoint between planning and testing. A model choice is a dependency with an EOL you don't control.

What survived: a cheap-and-accurate small model for the free tier, and a mid-tier model for paid — with the surprise that the mid-tier's previous generation was equally accurate at lower cost, which is exactly what you want in a fallback.

Price is data: keep it next to the routing

The eval's outputs — which models are allowed, in what order, at what price — live in one file. A Route is a provider (endpoint + key) plus a model plus that model's list price:

const PRICE: Record<string, [number, number]> = {
  'small-fast':   [0.1, 0.4],   // USD per 1M tokens, in/out
  'mid-plus':     [0.4, 1.6],
  'mid-plus-prev':[0.5, 3.0],
  'flagship':     [2.5, 7.5],
}

const DEFAULT_CHAINS: Record<Tier, string[]> = {
  free: ['small-fast', 'legacy-plus'],
  paid: ['mid-plus', 'mid-plus-prev', 'flagship'],
}

Each tier gets an ordered fallback chain: the head is the workhorse, the tail is who serves the request when the workhorse can't. If a backup API key is configured, the chain ends with the primary model on the backup account — because when your account balance dies, every model on it dies together, and only a different key helps.

"The model failed" is three different problems

The subtle part of fallback chains isn't trying the next model — it's knowing when the next model helps at all. Every failure gets classified into one of three moves:

function classify(e: unknown): 'retry' | 'next' | 'fatal' {
  if (e instanceof LLMHttpError) {
    const { status, body } = e
    if (status === 401 || status === 403) return 'fatal'   // new model won't fix your key
    if (status >= 500) return 'retry'                       // transient, same route
    if (status === 429)
      return /RateQuota|rate limit/i.test(body) ? 'retry' : 'next'
    if (status === 400 || status === 404)
      return /model|not.?found|InvalidParameter/i.test(body) ? 'next' : 'fatal'
    return 'next'
  }
  return 'retry'  // network-layer: ECONNRESET, DNS, timeout
}

The one that bites people: 429 is two different errors wearing one status code. Rate-limit throttling is transient — back off and retry the same model. Quota/allocation exhaustion is not — retrying the same model just burns time; skip to the next route. You can only tell them apart by sniffing the response body, and the distinction is provider-specific. Learn your provider's error taxonomy; it's load-bearing.

When the whole chain is exhausted, the app returns placeholder text with an ok: false flag — and the flag exists because of a real trap: never persist a fallback stub. A paid user whose reading gets cached as "(placeholder)" sees that placeholder on every revisit, forever, and the system never retries because a cached reading exists. ok gates the database write; failures stay ephemeral and self-heal on the next request.

Cost telemetry that survives fallback

Every business action (one reading = up to 7 parallel calls) emits a usage event, and each call's cost is computed against the model that actually served it, not the one you intended:

const intended = primaryModel(tier)
const fellBack = served.some((m) => m !== intended)
capture('llm_usage', {
  kind, tier, model: servedModels, primary_model: intended,
  fell_back: fellBack, input_tokens, output_tokens, cost_usd,
})

fell_back: true is the alert condition — it means your workhorse is degraded and your margins quietly changed. With this wiring, real numbers per call (~3.7k in / 0.4k out): $0.0021 on the paid-tier model, $0.0005 on the free-tier one — so a two-call free reading lands near $0.001. Those aren't estimates; they're what the meter read.

Takeaways

Benchmark on your own corpus. Leaderboards can't see that your domain has a closed vocabulary, and "most capable" models can be your worst performers on it.
enable_thinking (or your provider's equivalent) is the biggest single cost lever on hybrid reasoning models — and verify each snapshot actually honors it. A preview build that rejects the flag bills you for reasoning tokens on every call, on top of an already higher list price.
Classify failures before you retry. Same-model retry, next-model failover, and give-up-now are different errors sharing status codes.
End the chain with a different account, not a different model. Balance exhaustion kills models in bulk.
Never persist fallback output. Gate the cache write on "this is real content."
Emit cost per actual served model with a fell_back flag. Silent fallback is silent margin change.

The app all this serves is auspiceoracle.com — a bilingual BaZi calculator where a deterministic engine computes the chart and the LLM is only allowed to phrase it. That constraint is its own article (next in the series).

Your birth time is lying to you: a time-zone rabbit hole in a Chinese astrology calculator

Shan Liu — Sat, 22 Aug 2026 18:09:26 +0000

I built a calculator for BaZi — Chinese "Four Pillars" birth charts. Whatever you think of the interpretive tradition (and I'll get to that), the input math turned out to be a genuinely deep time-zone problem, and that's what this post is about. If you've ever thought "time zones, how hard can it be" — this is a tour of exactly how hard, with working TypeScript.

The problem

BaZi divides the day into twelve two-hour "branches", so your birth hour is one of the chart's four pillars. Get the hour wrong and you get a different chart — not slightly different, categorically different.

Every calculator I could find feeds the system the wall-clock time from your birth certificate. But the tradition predates time zones by about two thousand years; it obviously means solar time — where the sun actually was over your birthplace. Clock time and solar time differ by more than most people think, and the difference decomposes into exactly three parts:

1. Daylight saving time — and it's historical. You need the DST rules in force on the birth date, not today's. China ran a now-forgotten DST experiment from 1986–91; Harbin kept its own zone before 1949. If you were born in Beijing in July 1988, your certificate is an hour ahead of standard time and no modern-day lookup will tell you that.

2. Longitude. Solar time shifts 4 minutes per degree from your zone's standard meridian. China spans five geographic zones but uses one clock — born in Ürümqi, your clock runs about two hours ahead of the sun. It's not just a China quirk: Vancouver sits at 123°W in a zone whose meridian is 120°W, so that's another 12 minutes, everywhere, always.

3. The equation of time. The sun itself runs up to ±16 minutes fast or slow over the year, thanks to orbital eccentricity and axial tilt. NOAA publishes an approximation that's accurate to under a minute:

/** Equation of time (minutes), NOAA approximation */
export function equationOfTimeMinutes(dayOfYear: number): number {
  const b = (2 * Math.PI * (dayOfYear - 81)) / 364
  return 9.87 * Math.sin(2 * b) - 7.53 * Math.cos(b) - 1.5 * Math.sin(b)
}

Stack all three and a July birth in Vancouver needs ~78 minutes of correction. That's easily a different hour branch — a different chart.

Getting historical offsets without shipping a tz database

Here's the part that surprised me: you don't need to bundle tz data. Node's Intl is backed by ICU, which ships the full IANA tzdb — including the historical oddities. The trick is that Intl.DateTimeFormat will happily format a UTC instant in any zone, and from the formatted parts you can recover the offset:

The recipe, in words: format the UTC instant into the target zone with Intl.DateTimeFormat.formatToParts(), then re-read those wall-clock fields as if they were UTC. The gap between that and the real instant is the zone's offset at that moment — historical rules included, because ICU carries them.

What it gets you:

tzOffset('Asia/Shanghai', 1988-07-01)  →  +540 min  (+9h — the forgotten DST)
tzOffset('Asia/Shanghai', 2001-11-03)  →  +480 min  (+8h — normal)

That +9 is the whole point: a 1988 Shanghai birth certificate is an hour ahead of standard time, and ICU knows it without you shipping a byte of tz data.

Three things bit me getting there, and they're the difference between a snippet and something you run a few hundred thousand times a day: hourCycle: 'h23' is load-bearing (some runtimes hand you hour 24 for midnight, and Date.UTC cheerfully rolls that into the next day), a fresh DateTimeFormat per call is the most expensive thing in the whole pipeline, and zones ICU doesn't recognize need a fallback rather than a throw.

Going the other way — wall time to UTC — has the classic chicken-and-egg problem (you need the offset to compute the instant, but the offset depends on the instant). Two fixed-point iterations settle it everywhere except inside the one-hour DST gap, where no exact answer exists anyway.

There's a subtler one hiding in "was DST active?". JavaScript has no isdst API, so I sample the zone's offset on Jan 1, Jul 1, and the birth instant, and take the minimum as the standard offset — DST always moves clocks forward, so the minimum is standard time in both hemispheres. Sampling the birth instant too matters because of Morocco, which observes negative DST during Ramadan; without it, the heuristic reports a +60-minute DST that never happened.

The two edge cases I didn't see coming

The date line. The Chatham Islands sit at 176.5°W and use UTC+12:45. Do the naive thing — longitude × 4 minutes from Greenwich — and the computed local mean solar time lands a full day off. In a birth chart that silently corrupts the day pillar, which is the pillar the whole reading hangs on.

The fix is to normalize into the ±180° window centered on the zone's standard meridian, not the one centered on Greenwich — the Greenwich version is what you get for free, and it's what silently breaks:

Chatham Islands: longitude -176.5°, zone UTC+12:45 (meridian 183.75°)

  naive, normalized against Greenwich → -176.5°  → mean solar time off by ~24h
  normalized against the meridian     → +183.5°  → correct

Same input, and the difference is a whole day in the day pillar.

Rounding that has to add up. The UI shows the three components as an addition table: DST + longitude + equation of time = total. Round each part independently and the table stops summing — off-by-one minutes that make the whole thing look broken. So the rounded parts are forced to sum exactly to the rounded total, with the residual assigned to whichever part had the largest rounding error. A tiny thing, but "the math visibly doesn't add up" is not a good look for a calculator.

Unknown birth hour: compute all twelve

Most calculators, when you don't know your birth hour, silently default to noon or midnight — producing a confident chart of a person who doesn't exist. But there are only twelve possible hour branches, and the chart function is pure. So: compute all twelve charts (~1ms), intersect the results, show blanks where they disagree. On a 184-sample test set, 52% of charts still have a unique strength verdict with no hour information at all — which means half the time we can give a real answer instead of a fabricated one.

And when the corrected time lands within 8 minutes of a two-hour boundary, we flag it and suggest comparing both charts, instead of pretending to a certainty the input data can't support.

"But isn't this astrology?"

The interpretive layer is a cultural system — take it or leave it. The computational layer is not: calendar conversion, historical time-zone resolution, solar position, and the sexagenary cycle all have objectively right and wrong answers, and most tools get them wrong. That's the part worth engineering carefully, and honestly it's the same rigor any birth-time-sensitive system (astronomy tooling, historical databases) deserves.

The stack: Next.js, lunar-typescript for the sexagenary calendar, Intl/ICU for time zones. No external API calls for the chart itself. We also publish our nayin translation table as open data (CC BY 4.0): github.com/Shann5/bazi-nayin.

The calculator is free, no signup, English and Chinese: auspiceoracle.com/en. The full write-up of the solar-time correction, with a city-by-city table, lives at auspiceoracle.com/en/content/true-solar-time.

Happy to go deeper on any of the time handling in the comments.

Your LLM Skill can't do astronomy: why packaged divination Skills compute the wrong answer

Shan Liu — Mon, 17 Aug 2026 07:01:19 +0000

There's been a wave of Chinese-divination Skills on GitHub this year. The biggest BaZi (Four Pillars astrology) one has ~2.5k stars — interactive chart casting against nine classical texts, plus MBTI mapping, compatibility, and elemental remedies. Another has ~800 stars and leads with engineered hallucination prevention: fixed casting steps, a structured knowledge base, external scripts. Below those: a ~450-star everything-bundle covering a dozen systems, a ~330-star one that converts chart output into AI-readable structured prompts and ships an API and an MCP server, and a ~270-star offline runtime that computes locally and exposes results to the model.

I've spent a year writing a BaZi engine. I've read through these projects, and to be fair: there are people doing serious work here. The one that treats anti-hallucination as goal #1 has the right instinct. The one that pushes computation into a local runtime has the right instinct too.

But almost all of them are inaccurate in a place nobody looks — not in the interpretation, in the arithmetic that happens before it.

That's backwards from how people assume this works. The casting step looks like the easy mechanical part and interpretation looks like the hard part. It's the reverse. Interpretation has no ground truth, so anything reads as plausible. Casting has exactly one right answer — and most implementations get it wrong.

Everything below generalizes past this domain. The pattern is: an LLM asked to do a lookup or a calculation will produce something that looks like the answer, and if your domain has no error signal, you will never find out.

Failure 1: the year boundary is a timestamp, not a date

Here's a run of my engine. Shanghai, February 3–5 1990, only the clock time changes:

1990-02-03 23:00 → 己巳 丁丑 己亥 乙亥
1990-02-04 10:00 → 己巳 丁丑 庚子 辛巳
1990-02-04 12:00 → 庚午 戊寅 庚子 壬午
1990-02-04 14:00 → 庚午 戊寅 庚子 癸未
1990-02-05 01:00 → 庚午 戊寅 辛丑 戊子

Look at the three rows for Feb 4. At 10:00 the first two pillars are 己巳丁丑. At 12:00 they're 庚午戊寅. The year and month pillars both changed in the middle of a single day.

Because the BaZi year doesn't turn at midnight on Jan 1, or at Lunar New Year. It turns at Lichun — the solar term that begins when the sun reaches 315° of ecliptic longitude. That's an instant, precise to the minute, different every year. Two people born the same calendar day, one in the morning and one after lunch, get different year and month pillars. Not slightly different — 4 of the 8 characters differ.

So: how does a Skill know what time Lichun occurred in 1990?

There are exactly two ways. Either it ships a table of every solar term for every year to minute precision (1900–2100 × 24 terms ≈ 48,000 timestamps), or it computes the astronomy at runtime. Most of these Skills do neither — they encode solar term dates in a Markdown knowledge base. Day granularity. Which means for everyone born on the boundary day, the year pillar is either entirely right or entirely wrong depending on which half the model guessed.

This is not an interpretive style difference. A quarter of the chart is simply incorrect.

Failure 2: the clock on the wall is not the sun

Same wall-clock time, same national timezone, two cities. June 15 1992, 07:10 Beijing time:

Shanghai  121.5°E : solar correction   +6 min → 壬申 丙午 壬戌 甲辰
Ürümqi     87.6°E : solar correction −130 min → 壬申 丙午 壬戌 壬寅

Different hour pillar. 甲辰 vs 壬寅 — both characters change.

China spans roughly four geographic time zones (73°E to 135°E) and runs on one official clock. BaZi uses the sun where you were born. At 07:10 on the Ürümqi clock, local solar time is barely past 05:00 — the sun is only just up. That's the 寅 hour, not the 辰 hour. Two full branches apart.

Computing this correction needs three quantities: the birth longitude, the UTC offset in effect at that place on that date, and the equation of time — the ±15-minute daily discrepancy between true and mean solar time caused by Earth's elliptical orbit and axial tilt. That last one is a trigonometric series, evaluated per day-of-year.

Can a model do this in-context? Realistically, no. Asking it to evaluate the equation of time is asking for a plausible number. Asking it to recall longitude fails differently: the user says "I was born in Ürümqi," the model supplies an approximate longitude, and one degree of error is four minutes — which near an hour boundary is enough to flip the pillar.

Failure 3: the hour that never existed

This is the one I find most instructive.

1988-07-01 08:00 Shanghai: total correction −58 min
  decomposed: DST −60, longitude +6, equation of time −4

8 AM on the clock in Shanghai in July 1988 was really 7:02, because China observed daylight saving time from 1986 to 1991. Almost nobody remembers this. But people born in those six years are in their thirties now — squarely the demographic that goes looking for a reading. Their birth certificates record a clock that had been moved forward an hour. An hour is enough to cross an entire hour-branch, and at the right time of night, the day pillar too.

It gets worse at the transition itself. On the spring-forward date, local 02:00 jumps straight to 03:00 — that hour never existed locally. If a birth record says 1988-04-17 02:30, that timestamp is not a real instant.

My own engine got this wrong first. The original implementation reverse-solved the UTC instant by fixed-point iteration, which doesn't converge for a nonexistent input — but it returned a result anyway. Consequence: 02:59 and 03:00 mapped to the same real instant while producing different hour pillars, and their reported corrections differed by a full 60 minutes. It's now pinned by a test that walks the gap across all six Chinese DST years, plus the minutes on either side of it.

I bring this up not to show off the fix but to make the central point: none of these errors throw. The system quietly hands you eight tidy characters. Nothing about the output looks degraded. Feed that chart to any competent LLM and it will fluently explain what those characters mean about your life — and the better the prose, the more thoroughly wrong the result.

Why everyone thinks it works anyway

Because this domain has no verification loop.

You enter a birthday, you get a paragraph, and it feels like it fits. That feeling has three sources: the Barnum effect (sufficiently general descriptions fit everyone), the model's genuine fluency (it was trained to generate text you find apt), and your own cooperation (people unconsciously supply confirming evidence for a reading they've already heard).

Together those are more than enough to make a system with a completely incorrect chart read as accurate.

This is the uncomfortable part of working in this space. In most domains a miscalculation produces feedback — the build breaks, the ledger doesn't balance, the bridge falls down. Here there's nothing. The user won't know, the model won't know, and the Skill author won't know either. Unless someone writes a test.

So there's really one question I use to judge whether a divination project is serious: does it have failing test cases for the chart casting? Not tests that assert it runs — tests that assert this input must be rejected, or must return A and never B. The anti-hallucination project mentioned earlier is pointed the right way with its external scripts and fixed steps; so is the local-runtime one. But between "pointed the right way" and "computes correctly" sit three separate mountains: the solar term ephemeris, the equation of time, and the historical timezone database.

The actual bug is the division of labor

To be clear, Skills as a format aren't the problem. The task split is.

Chart casting is pure computation. Given an instant and a place, the eight characters are uniquely determined; there is zero interpretive latitude. That belongs in code: a pure function, unit tested, identical every run.

Interpretation is language. The same chart can be read gently or bluntly, weighted toward career or temperament. Models are genuinely good at that — better than I am.

The failure is handing the first job to the model too. An LLM's reliability at table lookup and its reliability at prose are not remotely the same number. Ask it to recall 48,000 solar term timestamps and it will hand you something that looks a great deal like one.

My rule, which I'd apply to any LLM product that must not invent things:

The engine decides what is said. The model decides only how to say it.

Eight characters, elemental strengths, favorable elements, luck cycles — all computed in TypeScript, unit tested, with the constants published on a public method page. The model receives a block of facts and one directive: cite only what's given. It's a translator with a persona, not an oracle.

There's a second benefit that's easy to miss: once the engine owns the facts, it can also own the uncertainty. When a user doesn't know their birth hour, I don't pick a default and hope. There are exactly 12 possibilities, and computeChart is a pure function, so I compute all 12 and intersect them — only conclusions that hold in every candidate reach the prompt; the rest are left blank, explicitly. When a corrected time lands within a few minutes of an hour boundary, I cast the chart on the other side too and show precisely what changes and what doesn't.

A stated hole beats a silent one. Leave the slot empty and the model backfills it, convincingly. Mark it and instruct it, and the model routes around it.

What to actually check

If you're using one of these Skills, three questions you can verify yourself:

Born on a solar-term boundary day — does it know the exact time that term began that year?
Born far from your timezone's meridian — does it correct for birth longitude? Born 1986–1991 in China (or any DST region) — does it subtract the offset?
Don't know your birth hour — does it leave it blank, or quietly assign one?

If it can't answer one of the three, the chart has a meaningful chance of being wrong. It will read exactly as convincing either way.

If you're writing one, the advice is a single line: pull the casting out into code, and give it tests that can fail. That last clause matters. I once deleted a validator I'd shipped months earlier because a code review revealed both of its loops asked whether a set contained elements drawn from that same set — always true, never able to fire. It had caught zero violations and structurally could not catch any. Meanwhile it appeared in every architecture discussion as "we validate that," so everyone stopped thinking about it. A guardrail that cannot fire is worse than no guardrail, and an assertion that has never gone red is a hypothesis, not a gate.

Get the arithmetic right first. Then hand the prose to the model — it really is better at that part than you are.

Every chart above is a live run of the engine behind Auspice Oracle. The handling of solar term instants, true solar time, and historical DST is documented on the method page.

Should I replace my marketing intern with AI...

Shan Liu — Wed, 12 Aug 2026 07:55:06 +0000

The translation model that cost 15 more was also the most confidently wrong

Shan Liu — Wed, 12 Aug 2026 05:14:52 +0000

My app serves the same content in Chinese and English: personalized readings for Chinese birth charts, generated by an LLM. The readings are cached; when a user flips language, regenerating from scratch costs ~3 seconds and produces a different reading — which is exactly what you don't want, because now your two languages disagree about the user's life. The fix is obvious: translate the cached reading instead of regenerating. Language switch went from 2.8s to ~60ms once cached, and both languages finally say the same thing.

But "just translate it" walked straight into the domain-terminology buzzsaw, and the eval I ran to pick a model produced my favorite result of the year: the flagship translation model, at 15× the price, was the most dangerously wrong of the lot.

The baseline eval: fluent nonsense scales with price

The domain (BaZi astrology) has a closed set of technical terms with established English renderings. I ran four sizes of a dedicated translation model family over my own corpus, no glossary, and scored the terms:

Term	Correct EN	flash (cheap)	plus (flagship, 15×)
七杀	Seven Killings	Seven Killers ✗	Seven Killers ✗
印星	Resource star	Inheritance Star ✗	"Hidden Stems of the Mind" ✗✗
比劫	Companion star	Peer & Robbery Stars ✗	"Rat–Ox combinations" ✗✗✗
食神	Eating God	Food God ✗	Food God ✗

Every model missed terms — expected; the terms are jargon. The pattern that wasn't expected: the flagship's misses weren't near-misses, they were inventions. "Hidden Stems of the Mind" is not a bad translation of 印星 (a chart-analysis category); it's a hallucinated concept delivered with total fluency. The cheap model's "Inheritance Star" is wrong but recognizably adjacent. Paying more bought more fluent wrong answers — a bigger model has more capacity to confabulate plausibly, and in translation, plausible-but-wrong is strictly worse than clunky-but-wrong because nobody catches it in review.

The glossary is the product

The provider's API takes a terms list — source term, forced target rendering — injected at translation time (the docs call it term intervention; most serious MT APIs have an equivalent). With ~15 domain terms pinned:

All four models became term-perfect. Every one. The entire quality race collapsed into prose style, latency, and price:

Model	Terms (with glossary)	Prose	Latency	Cost per domain snippet
flash	perfect	most natural	~740ms	$0.00015
lite	perfect	slightly flatter	~610ms	$0.00011
plus	perfect	marginally more idiomatic	~1100ms	$0.0022

So the model decision — the thing I'd expected to agonize over — was over. The cheap-fast one wins on quality-per-dollar; the flagship's "marginal polish" is not worth 15×. All the real engineering turned out to be in what goes into the glossary. Three findings, each from a production incident or a near-miss:

1. Don't glossary everything you could glossary

My first instinct: pin all the domain vocabulary, including the sexagenary "pillar" characters (庚 → Yang Metal, 午 → Horse Fire). The output was technically correct and read like a parts catalog: "Yang Metal seated atop Horse Fire." The unglossed baseline had written "Geng seated atop Wu Fire" — pinyin, the way English-language practitioners actually write. The glossary is for terms where deviation is an error (the analysis vocabulary). Terms where the natural rendering varies by register (name-like characters) read better left to the model, with the fixed mapping reserved for UI labels. Scope the forced list to what must never drift; let prose be prose.

2. A glossary is directional

Reusing the zh→en term table for the en→zh direction doesn't fail loudly — it fails by leaving English terms embedded in Chinese output. Real production sentence: 「你的Day Master是Rén」. The table has to flip per direction, so the lookup is translationTerms(targetLang), not a constant.

3. Reverse-mapping pinyin is a minefield of English collisions

Going back into Chinese, the pinyin names in English prose (Rén, Shēn) must become characters again (壬, 申). I generate that map from the engine's own data tables — but only the tone-marked forms are included. The bare romanizations collide with ordinary English: You is a branch name (酉), Yin is one (寅), Wu is two different ones (戊/午). An unfiltered map happily rewrites every English "you" into 酉. Diacritics, which I'd initially treated as typographic fussiness, turned out to be the only thing making the mapping injective.

Practical constraints worth knowing

Dedicated MT endpoints are shaped differently from chat models, and the shape matters for cost:

Single-turn, no system prompt. You send source text; the glossary rides in a config field that isn't billed as prompt tokens. A chat model doing the same job re-bills you the instruction preamble on every call.
Hard input cap (8,192 tokens here) — long documents need a chunker.
Measured production numbers: a full reading section zh→en in ~2.1s at $0.0003; short strings at $0.00008. Output tokens dominate MT cost, so the output-price column of the pricing table is the one to sort by.
The chain still ends in a fallback: flash → lite → regenerate in the target language if the MT service is down. Degraded consistency beats an error page.

Takeaways

In a domain with fixed terminology, eval the glossary mechanism, not just the models. It flattened a 15× price range into a tie.
Model size doesn't fix domain terms — it upgrades the failure mode from clunky-wrong to fluent-wrong. Fluent-wrong is worse.
Scope the glossary to must-never-drift terms; over-pinning makes prose robotic.
Glossaries and transliteration maps are directional; test the round trip, not just the forward pass.
Translating cached content instead of regenerating buys you speed and cross-language consistency — the second one is the sleeper benefit.

The bilingual app in question: auspiceoracle.com. The eval harness (corpus, runner, full transcripts) lives in the repo and reruns with one command — evals you can't rerun are anecdotes.

I audited a 5.5 GB AI training dataset by downloading 0.8% of it

Shan Liu — Thu, 06 Aug 2026 20:06:58 +0000

Two months ago a repository turned up in my corner of the internet: 3,358 stars, 744 forks, an MIT license, and a release advertised as 518,400 training samples — 5.5 GB, split across three zip parts. My corner is Chinese astrology software, a domain with almost no machine-readable data, so a corpus that size was either the most useful thing published in years or it was nothing at all. I wanted to know which.

Then I looked at the sample count for one second longer. 518,400 = 60 × 12 × 30 × 12 × 2. Sixty years, twelve months, thirty days, twelve two-hour periods, two genders. That isn't a tally of things somebody observed. That's the size of a nested loop.

Which is a wonderful reason to be curious and a terrible reason to conclude anything. So I went and read the archive — 48 MB of it, over HTTP range requests, never touching the other 5.83 GB. The trip turned out to be more interesting than the destination, and the method is the part that travels well beyond my strange little niche.

So: here's how you read a multi-gigabyte release for the price of a few photos, and then the four things it told me.

First, the trick: reading a 5.5 GB release without downloading it

Three facts make this work, and they're pleasant ones to know:

A split zip made with split is plain concatenation. The parts aren't independent archives. Part 1 opens with the first local file header, and the byte offsets in the central directory are absolute across the whole set.
The central directory sits at the end and lists every entry: name, uncompressed size, compressed size, local-header offset. Fetch the last couple of megabytes and you're holding the table of contents.
GitHub release assets honour Range requests. Once you know an entry's offset and size, you can ask for exactly that entry and nothing else.

There was a bonus in this particular archive. It stores its .jsonl.gz shards with method 0 — stored, not deflated — because they had already been gzipped. Every entry is independently addressable, and the small text files at the tail sit there uncompressed, so a plain curl -r hands you readable source code.

# 1. the table of contents: last 3 MB of the last part
curl -sL -r 1890639808-1893639807 \
  ".../ziwei-samples-v3-part3.zip.003" -o p3.tail
# then find PK\x05\x06 (EOCD), walk the PK\x01\x02 entries,
# and honour the ZIP64 extra field — cdoff came back as 0xFFFFFFFF

# 2. one 8 MB data shard, at the offset the directory gave me
curl -sL -r 48646942-56753200 \
  ".../ziwei-samples-v3-part1.zip.001" -o feb.bin

917 entries came back. 781 of them are the data everybody downloaded it for. The other 136 are where the story turned out to be.

Total fetched: 47,897,905 bytes — 0.81% of the release. Full script in the appendix.

Finding 1: the release contains the file the repo says it held back

The public repo ships lib/ziwei/db-analysis.ts as a 2,124-byte stub, and its header comment is refreshingly direct about why (my translation):

The analysis content library is not part of the open-source scope. The full online version contains detailed readings for 14 major stars × 13 palace contexts — core content, not open-sourced along with the chart engine.

And at the bottom: export const STAR_DB: Record<string, unknown> = {};

The same path inside the release is 377,157 bytes, with STAR_DB populated starting at line 161: 524 string literals, 145,580 characters of hand-written Chinese prose. The library that was held back from git went out in the release anyway — just parked 5.87 GB deep inside a three-part split zip, where nobody was looking.

That's the first and most portable lesson, and it has nothing to do with astrology: redaction leaks through large binary artifacts. A file you carefully stubbed in git will cheerfully ride along in the release tarball, the Docker layer, the model checkpoint, the training-data dump. The bigger the artifact, the less likely anyone notices — which is exactly backwards from the way the risk actually runs.

Finding the source also settled what the dataset is. Across the whole lib/ziwei/ directory there are 216,172 characters of TypeScript, 183,182 of them inside string literals. The generator is five nested loops over that grid, longitude hardcoded to 120, and — outside the audit script's own sampler — no call to Math.random anywhere. The output is a pure function of a five-integer key.

Which makes the release a memo table. Uncompressed it comes to roughly 32.9 GB (I measured one shard at 45,649,978 bytes, and there are 720 of them), holding something like 1.09 × 10¹⁰ characters of generated prose, all of it assembled from 183 K characters of source — source that ships in the same zip, next to the npm run full command that regenerates the whole thing.

Measured reuse, on the 3,600 samples I pulled (0.69% of the corpus): 46,800 emitted topic blocks, 16,359 distinct — and each of those is a permutation of the same literals. Fine-tuning on this doesn't teach a model a domain. It pays a GPU bill to compress a template engine into weights, badly, when you could simply call the engine.


Release size	5.5 GB (3 parts)
Uncompressed	~32.9 GB
Generated prose	~1.09 × 10¹⁰ chars
Hand-written source it came from	183,182 chars
Math.random in the generator	none

Finding 2: 2,520 of the samples are dates that never happened

days: range(1, 30) — thirty days for every month, and only ever thirty.

That has two consequences. Every 31st is missing: seven months × 60 years × 12 hours × 2 genders = 10,080 real birthdays with no row at all. And February gets days 29 and 30 regardless of the year, which leaves 2,520 rows keyed to dates that never existed.

I couldn't resist pulling February 1962, a non-leap year, to see what the generator does with an impossible input. It doesn't fail. It rolls straight past the end of the month and keeps counting:

Gregorian input	lunar day the engine assigned
1962-02-27	month 1, day 23
1962-02-28	month 1, day 24
1962-02-29 (not a date)	month 1, day 25
1962-02-30 (not a date)	month 1, day 26

So the sample labelled 1962-02-30 is a complete, confident, 63 KB chart — for a person born on March 2nd. Nothing anywhere in the pipeline noticed.

Two more shape quirks live in the same grid. The year axis is range(1924, 1983), with a source comment explaining that "60 years covers a full sexagenary cycle" — the window was chosen to make the number come out round, not to cover anybody's users. Nobody born after 1983 is in here, which in 2026 makes it a corpus exclusively about people aged 43 and up. And longitude is 120 on every single row, which stings a little: in a system where the birth hour is the highest-entropy input, the one variable most worth varying got pinned to a constant.

Finding 3: "518,400 / 518,400 validated" turns out to be a substring check

The release ships its own validation and audit logs, and they look great. Zero failures, zero warnings, lines like these:

health contains 「liver/kidney/spleen-stomach」:  518400/518400 (100.00%)
health contains 「子午流注」 and 「經絡」:        518400/518400 (100.00%)
female health contains 「gynaecology/menses/pregnancy」: 259200/259200 (100.00%)
male/female wealth topic pairs differ:           259200/259200 (100.00%)
failures: 0     warnings: 0

Every one of those is String.includes. They establish that a template fired. They cannot tell present apart from correct — and here that distinction is load-bearing, because five lines earlier in the very same log:

daXians[0] contains siHua:        0/518400 (0.00%)
any daXian contains siHua:        0/518400 (0.00%)
daXians[0] contains stemIndex:    0/518400 (0.00%)
samples containing palace.selfSihua: 0/518400 (0.00%)

The spec handed to the generating agent — README-CODEX.md, also in the archive — calls one of those fields "the single easiest thing to get wrong" and makes it hard requirement #6, with a post-generation checklist that says to verify it. I went and checked the shipped records directly: each daXians entry has exactly four keys — startAge, endAge, palaceBranch, palaceName. The field the spec built itself around simply isn't there.

The packaging manifest, meanwhile, lists this in English under the heading "Verified results":

Verified results:
- Total samples: 518,400
- Validation failures: 0
- daXians[].siHua: absent
- daXians[].stemIndex: absent
- palace.selfSihua: absent

An earlier audit report, still sitting in the box, shows those same fields populated with real values across 20 sampled rows. So somewhere between that run and the shipped one, the three most important computed fields went missing, and the validator scored their absence as a pass. The scorecard file then awards 10/10 to the dimension "hard-constraint implementation and audit," on the stated basis that verification commands exist for each constraint.

One more from this log, because it's the detail I keep thinking about. The audit counts how often each of twelve canned warning phrases appears, and three of them land at exactly 518,400 — including the sentence "if the fortune palace is afflicted and the spouse palace shows no separation in life, then it must be separation by death." That line is meant to be conditional on a configuration most charts don't have. It's in 100% of the corpus. A model trained on this learns to tell everybody they'll be widowed.

Finding 4: a file in the archive audits the citations, and it's brutally honest

Tucked into the release is corpus/annotations.json — a review of every quotation in the analysis library attributed to a named teacher, a physician-lecturer who died in 2012. Here's its own summary, in its own categories:

status	count	the file's definition
verified	18	appears in the lecture transcripts, source checked
traditional	42	generic classical maxim, attributed to him but not his
suspect	15	source doubtful, possibly another school or later invention
fabricated	6	"highly likely invented by a previous author or by me on the spot"
methodology	14	concept correct, not a quotation
total	95

19% of the quoted attributions are traced. The notes are first-person and completely unsparing — "this line I made up, should be changed to 'the Northern school holds…'"; "'precise month-derivation method' is a name I invented, not his term"; "⚠️ major error: his illness readings key on the palace branch, not the star element — this whole mapping deviates from the system."

There's also a top_priority_fixes array. One P1 item is the fortune-palace sentence I quoted above, flagged as probably not his words and in need of a rewrite. It is the sentence that appears in 518,400 of 518,400 samples. The fix list shipped inside the artifact it was supposed to gate. (The generated prose also attributes it, and dozens of other lines, using a misspelling of the teacher's name.)

And here's the part that makes this worth writing up rather than piling on. That annotations file is more provenance diligence than almost any repo in this space has, and the maintainer wrote it themselves, about their own text, before shipping. Sitting next to it is a rights policy that sorts collected material into four tiers, forbids storing full text without a license, and forbids rights_status=unknown material from entering trainable corpora. The source registry reads: 211 sources, 1 cleared for full text, 167 needing a license, 15 prohibited outright. Somebody thought hard about all of this.

So the failure isn't missing diligence. It's that the diligence was never wired to a gate. A file listing six fabricated citations doesn't block a release. A policy saying "unknown rights don't enter training corpora" doesn't stop a corpus derived from that material from going out under MIT. A validator that greps for a string can't fail a build for being wrong. Every artifact of trust in that box was produced by the same pipeline it was meant to check — the agent generated the data, ran the audit, and reported success, and each of those three steps was scored on whether it completed.

That's the 2026 failure mode, and I don't think it's rare. Not slop. Slop with a passing test suite.

The three checks I run now

Factor the sample count. If N decomposes into small round factors, you're looking at a grid — coverage of a key space rather than observations of anything. Ask what the unit of observation is. If the answer is "a possible input," there's no signal to learn, and no outcome, adjudication, or human in the loop to check a claim against.
Look for the generator before you look at the data. If it ships — and it very often does, in the tail of the archive — the generator is the dataset, and it's five orders of magnitude smaller. Then check it for randomness. No randomness means the release is a memo table of a pure function, and you should just call the function.
Read the validator, not the validation. grep for what it actually asserts. Substring presence, field presence, and file counts are shape checks; they pass with equal enthusiasm on correct output and inverted output. A green log tells you nothing until you've seen the predicate. Mine assert relations between computed values, for what that's worth, and I still don't trust them as far as I'd like.

A small footnote on why any of this needed doing. The download counts on that release: 10,814 for part 1, 7,506 for part 2, 7,989 for part 3, and 1,250 for the checksum file. About 30% of the people who started never finished, and roughly one in ten verified what they got. A 5.5 GB three-part download is its own kind of moat — nobody reads what they can't open, and 3,358 stars measured the appeal of the claim rather than the contents.

The audit cost 48 MB.

Why I was poking at it at all: I build software in the neighbouring system — BaZi rather than Zi Wei Dou Shu (auspiceoracle.com) — so this corpus landed on my desk as something I might use. I've deliberately left the repo, the tag, and the file paths out of the prose, because the point here is the method rather than the maintainer. I'm also not drawing any licensing conclusions: I'm not a lawyer, and every count I've quoted is the repo's own. Everything else reproduces in about two minutes with the script below, and I'd much rather be corrected than believed.

Appendix: reproduce it

import struct, json, urllib.request

REL  = "https://github.com/{owner}/{repo}/releases/download/{tag}/"
PART = ["...part1.zip.001", "...part2.zip.002", "...part3.zip.003"]
SIZE = [1992294400, 1992294400, 1893639808]          # from the releases API

def grab(part, start, end, out):                      # HTTP range fetch
    req = urllib.request.Request(REL + PART[part],
headers={"Range": f"bytes={start}-{end}"})
open(out, "wb").write(urllib.request.urlopen(req).read())

# 1. central directory: tail of the last part.
#    find PK\x05\x06 for the EOCD, then walk PK\x01\x02 entries.
#    cdoff == 0xFFFFFFFF means ZIP64 -> read the real values from
#    extra field header id 0x0001 (usize, csize, local-header offset).
grab(2, SIZE[2] - 3_000_000, SIZE[2] - 1, "p3.tail")

# 2. any entry, given its absolute offset from the directory:
#    absolute -> (part, local offset) via the cumulative sizes,
#    then re-read the local header to skip name+extra, and the
#    remaining csize bytes are the stored member, byte for byte.
#    Entries with method == 0 (stored) need no decompression —
#    the .jsonl.gz shards drop straight into gzip.open().

Three things to take away, none of them about astrology. Redaction leaks through big artifacts. Grids aren't data. And a green audit log is a claim like any other — it's worth asking what it measured.

All numbers measured 2026-08-05 against the release as published.

The LLM in my app is not allowed to decide anything

Shan Liu — Tue, 04 Aug 2026 21:46:55 +0000

I build software in the single worst domain for LLM truthfulness: fortune-telling. A BaZi (Chinese Four-Pillars astrology) reading app, where the model's job is to sound like a wise master — and where user reviews of competing AI products converge on one complaint: "pure nonsense." An LLM asked to "read a birth chart" will hallucinate chart elements that aren't there, invent rules that don't exist in the tradition, and deliver it all in a voice of total confidence. In a domain with zero external ground truth to check against, users can't tell — until two readings of the same chart contradict each other.

Whatever you think of the domain (I wrote about its genuinely hard timezone math earlier), the engineering answer is portable to any LLM product that must not make things up. It's one rule:

The deterministic engine decides what is said. The LLM decides only how to say it.

The chart, the element strengths, the favorable-element analysis, every derived fact — computed by a rules engine in TypeScript, unit-tested, published constants and all. The model receives those facts as a compact block and a directive: cite only what's given. It's a translator with a persona, not an oracle.

That's the easy 80%. The interesting engineering is in three places where the rule almost broke.

1. The hard case: a hole in the input

Many users don't know their birth hour — and the hour is one of the four pillars. The naive options are both bad: refuse the user, or let the model improvise around the gap. Guess which one a model does if you just omit the hour: it fills the hole. Silently. With a specific, plausible, invented pillar.

The fix is to make the engine handle the uncertainty, deterministically. Unknown hour → there are exactly 12 possible charts. Compute all twelve, then take the intersection: only facts that hold in every candidate chart survive into the prompt. Element strength agrees across all 12? State it. It splits 7/5? Then the prompt says, verbatim: "strength undetermined (7 of 12 candidates lean strong) — you may not build on this."

And crucially, the hole itself is made explicit rather than omitted:

Chart: 己未 丙寅 庚午 ▢   (hour pillar unknown — 12 candidate
charts computed, only facts true in all of them are listed)

That ▢ earned its place. A stated hole beats a silent one: leave the slot empty and the model backfills it; mark it and instruct ("do not mention the hour pillar; do not discuss the life areas it governs; inventing one is lying") and the model routes around it. The prompt even tells the model how to end gracefully — one sentence noting what more could be seen if the user learns their birth time. Uncertainty became a product feature instead of a hallucination site.

2. The second gate: validate the output like you don't trust the first gate

Prompts are policy, not enforcement. So generated text passes through a validator before it's stored:

Invented-pillar detection. For unknown-hour charts, scan the output for any of the 12 candidate hour pillars. The trick is matching the two-character stem-branch pair, never single characters — 子 alone appears inside the ordinary word 孩子 ("child"), 金 inside 资金 ("funds"). Pair matching has essentially no false positives; single-char matching would flag every other sentence.
Forbidden-pattern scan. Regex list of fatalistic/fear-mongering constructions ("will surely divorce", "short-lived", "incurable") — the domain's dark patterns, encoded. This isn't just taste: every platform policy that governs this vertical (search quality guidelines, ad policies, payment processors) draws its allow/ban line at concrete doom claims vs. interpretive reflection. The regex list is the compliance boundary as code.
Closed-vocabulary check. A list of star/deity terms the engine never computes; if one appears in the output, the model imported folklore from its training data. Flag for review.

3. The confession: my guardrail was dead code and I didn't notice

The validator originally had a third, stronger check: a whitelist assertion that every stem, branch, and "ten god" term in the output came from the chart JSON. Code-reviewing it months later, I found both of its loops were asking whether a set contained items taken from that same set — a condition that is always true, wired to a check that could therefore never fire. Two supporting arrays were never read at all. It had caught zero violations, ever, and couldn't.

I deleted it and wrote a comment explaining why, including what a real version would need to solve (the same single-character collision problem as above). Two lessons I now apply everywhere:

A guardrail that cannot fire is worse than no guardrail — it shows up in every architecture diagram and code review as "we validate that," and everyone stops thinking about it.
Test your validators the way you test code: with inputs that must fail. A validation function with no failing test case is a hypothesis, not a gate.

Bonus fight: prompt-language gravity

The persona and all instructions are written in Chinese; the app also serves English readings. A one-line "respond in English" does not survive contact with a 2,000-character Chinese prompt: the model would ship hybrid sentences into production — actual example: "Your盘的里，其实事业和财这两条线比性格更有讲头" — lifted straight from a Chinese example sentence inside the persona. The fix that held: an explicit paragraph stating that every quoted sentence above is a tone demonstration only, must not be copied or translated, and that the output may contain no Chinese characters except glossed pinyin. When your prompt is bilingual, the language directive has to out-shout the entire rest of the prompt.

Why bother, beyond truthfulness

Cost. The model writes 300 words of styled prose per section instead of "reasoning" about the chart. No chain-of-thought needed — thinking mode is off, calls are ~1.5s and fractions of a cent.
Consistency. Two users with the same chart get stylistic variation on the same facts, not two different fates. Re-reads don't contradict.
The line is auditable. When a user asks "why does it say that?", there's an engine fact to point to — the same one published on the site's method page.

The pattern is old, honestly — it's a compiler emitting facts and a pretty-printer rendering them. The only new part is that the pretty-printer went to art school and will invent facts if you let it. Don't let it: compute the truth, mark the holes, validate the output, and test that your validators can actually fail.

The app: auspiceoracle.com — the engine's scoring constants are public on the method page, which is the same "show your work" rule applied to marketing.

I spent a month doing AEO for a tiny niche site. Most of the advice was wrong.

Shan Liu — Tue, 04 Aug 2026 21:35:23 +0000

"Answer Engine Optimization" is the new gold rush: get your site cited by ChatGPT, Perplexity, and Google's AI Overviews. There is an entire cottage industry selling advice on how — add schema markup, publish more pages, buy a tool.

I run a small bilingual Chinese-astrology calculator (auspiceoracle.com). It's about as niche and low-authority as a site gets, which makes it a decent lab rat: zero brand signal, zero backlinks, nothing to confound the measurement. Before writing a single content page I did two things most AEO advice skips — I read the actual studies, and I set up measurement before launch. Here's what survived contact with the data.

Finding 1: schema markup is not an AEO lever

This one hurt, because adding JSON-LD is the single most-repeated piece of AEO advice.

The best evidence available is an Ahrefs difference-in-differences study: 1,885 pages that added JSON-LD, each matched to 3 control URLs on other domains at similar pre-period citation levels, 30-day windows, four statistical approaches. Result:

Platform	Citation change after adding JSON-LD
Google AI Overviews	−4.6% (small but significant decline)
Google AI Mode	+2.4% — indistinguishable from zero
ChatGPT	+2.2% — indistinguishable from zero

The much-quoted counter-stat ("AI-cited pages are 3× more likely to have JSON-LD") is confounding, and Ahrefs says so themselves: schema lives on better-maintained sites. Four independent mechanism studies agree on why — when LLMs fetch a live page they extract visible HTML and ignore the structured-data layer. One test planted facts that existed only in FAQ schema; no platform used them. Another fed models deliberately invalid schema and they happily extracted from it — the script block is being read as plain text.

What I kept: an extractable, plain-language definition in the first two visible sentences of every content page. That's the thing the machines actually read. Schema stays on the pages as cheap rich-result table stakes, but I budget zero AEO effort against it.

Finding 2: page volume is a weak signal, and you can't shortcut brand

The other standard advice is programmatic content: generate hundreds of pages, win on surface area. Ahrefs' correlation study across 75,000 brands ranks the signals that track AI visibility:

Signal	Spearman ρ
YouTube mentions	~0.74
Branded web mentions	0.66–0.71
Branded search volume	0.35–0.47
Domain Rating	0.27–0.33
Number of site pages	~0.19
Backlinks	~0.18–0.23

Read the fine print, though, before acting on any row: the sample is filtered to DR>40 brands, the correlations are zero-order (nobody partialled out brand size), and there's no independent replication. The honest inference isn't "make YouTube videos" — it's that AI visibility tracks composite brand prominence, which a new site does not have and cannot fake with page count. For a small site, both the vendor pitch ("more pages!") and the counter-pitch ("pages don't matter!") are extrapolations from a population you're not in. The studies literally sampled pages that already had 100+ AI citations. Yours have zero. Nobody has published data about you.

Finding 3: don't out-define Wikipedia; map the entity gaps

My original plan assumed the English terminology in my niche was unclaimed. It wasn't — Wikipedia holds the head term with an actively-growing article, and LLMs demonstrably over-index on encyclopedic sources. Any page whose job is to out-define Wikipedia is dead on arrival.

But the MediaWiki API tells you something more useful than "Wikipedia exists": which sub-concepts have no article and no redirect. In my niche, a half-dozen core glossary terms return missing — definitionally seated at the head, structurally scattered below. That gap map, not keyword volume, became the content plan.

The part nobody sells: measurement

AI crawlers have no submission channel. You can't ping GPTBot. Zero crawls means "not discovered yet," not "misconfigured." The only proactive lever is indirect: Bing's index feeds OpenAI's and Copilot's retrieval, so IndexNow (one key file + one POST per publish) is the single highest-leverage submission you can make. Everything else is external links doing discovery work.

My production box runs Next.js behind a tunnel with no nginx, so there were no access logs to mine. The fix was one line in the middleware — match the AI user-agents, console.log a line, and the process manager's logs become the dataset:

# UA × hit count, from pm2 logs
grep -h '[ai-bot]' ~/.pm2/logs/app-out*.log | awk '{print $3}' | sort | uniq -c | sort -rn

Two distinctions matter when you read those logs:

GPTBot / ClaudeBot / PerplexityBot = your page entered a crawl queue. Necessary, not sufficient.
ChatGPT-User / Perplexity-User / Claude-User = a human saw your site cited in an answer and the assistant fetched the page for them. This is the metric. Everything else is leading-indicator noise.

And one thing you cannot retrofit: a baseline. Before the pages went live I ran a fixed panel of six prompts through ChatGPT, Perplexity, and Google (18 cells), recorded who got cited, and archived screenshots. All 18 cells: zero citations of us, as expected. The discipline is the same panel every month, questions never edited — change the questions and you've changed the ruler. Without the pre-launch zero row, any future citation could be "maybe we already had that."

Early returns, for honesty's sake: on launch day one crawler (ClaudeBot) fetched all ten new pages exactly once each, like it was walking a checklist. The others: zero. Citations: zero. This is a 90-day experiment, not a success story — which is exactly why the baseline row matters.

The contrarian call: let the training crawlers in

Standard advice for content sites is to allow retrieval bots but block training crawlers (CCBot, GPTBot-as-trainer, Google-Extended). I did the opposite — explicit allow for everything.

The reasoning is cold-start economics. Nobody's model "knows" my site's terminology or that it exists. Being ingested into training data is how that changes, and the lag is a full model generation — a cost you pay now for visibility later. Blocking training crawlers protects content whose value is exclusivity; a new site has none. I wrote down the reversal condition (if content gets scraped-and-republished at scale, or citations stabilize, revisit), which keeps it a decision instead of a default.

What I'd tell you to do

Write the definition in the first two visible sentences. Skip the schema debate entirely.
Map entity gaps with the MediaWiki API before writing anything.
Set up IndexNow; accept that everything else is discovery-by-links.
Log AI user-agents at the edge, and learn the -Bot vs -User distinction.
Record a citation baseline before launch. Same prompts, monthly, forever.
Treat every AEO study as data about someone else's population until your own logs say otherwise.

The meta-lesson: AEO in 2026 is where SEO was in 2003 — long on vendors, short on mechanisms. The mechanisms are checkable. Check them.

The site in question, if you want to see the "extractable first two sentences" pattern live: how the engine works, and the true solar time deep-dive that became the first post in this series.