DEV Community: Shan Liu

The LLM in my app is not allowed to decide anything

Shan Liu — Tue, 04 Aug 2026 21:46:55 +0000

I build software in the single worst domain for LLM truthfulness: fortune-telling. A BaZi (Chinese Four-Pillars astrology) reading app, where the model's job is to sound like a wise master — and where user reviews of competing AI products converge on one complaint: "pure nonsense." An LLM asked to "read a birth chart" will hallucinate chart elements that aren't there, invent rules that don't exist in the tradition, and deliver it all in a voice of total confidence. In a domain with zero external ground truth to check against, users can't tell — until two readings of the same chart contradict each other.

Whatever you think of the domain (I wrote about its genuinely hard timezone math earlier), the engineering answer is portable to any LLM product that must not make things up. It's one rule:

The deterministic engine decides what is said. The LLM decides only how to say it.

The chart, the element strengths, the favorable-element analysis, every derived fact — computed by a rules engine in TypeScript, unit-tested, published constants and all. The model receives those facts as a compact block and a directive: cite only what's given. It's a translator with a persona, not an oracle.

That's the easy 80%. The interesting engineering is in three places where the rule almost broke.

1. The hard case: a hole in the input

Many users don't know their birth hour — and the hour is one of the four pillars. The naive options are both bad: refuse the user, or let the model improvise around the gap. Guess which one a model does if you just omit the hour: it fills the hole. Silently. With a specific, plausible, invented pillar.

The fix is to make the engine handle the uncertainty, deterministically. Unknown hour → there are exactly 12 possible charts. Compute all twelve, then take the intersection: only facts that hold in every candidate chart survive into the prompt. Element strength agrees across all 12? State it. It splits 7/5? Then the prompt says, verbatim: "strength undetermined (7 of 12 candidates lean strong) — you may not build on this."

And crucially, the hole itself is made explicit rather than omitted:

Chart: 己未 丙寅 庚午 ▢   (hour pillar unknown — 12 candidate
charts computed, only facts true in all of them are listed)

That ▢ earned its place. A stated hole beats a silent one: leave the slot empty and the model backfills it; mark it and instruct ("do not mention the hour pillar; do not discuss the life areas it governs; inventing one is lying") and the model routes around it. The prompt even tells the model how to end gracefully — one sentence noting what more could be seen if the user learns their birth time. Uncertainty became a product feature instead of a hallucination site.

2. The second gate: validate the output like you don't trust the first gate

Prompts are policy, not enforcement. So generated text passes through a validator before it's stored:

Invented-pillar detection. For unknown-hour charts, scan the output for any of the 12 candidate hour pillars. The trick is matching the two-character stem-branch pair, never single characters — 子 alone appears inside the ordinary word 孩子 ("child"), 金 inside 资金 ("funds"). Pair matching has essentially no false positives; single-char matching would flag every other sentence.
Forbidden-pattern scan. Regex list of fatalistic/fear-mongering constructions ("will surely divorce", "short-lived", "incurable") — the domain's dark patterns, encoded. This isn't just taste: every platform policy that governs this vertical (search quality guidelines, ad policies, payment processors) draws its allow/ban line at concrete doom claims vs. interpretive reflection. The regex list is the compliance boundary as code.
Closed-vocabulary check. A list of star/deity terms the engine never computes; if one appears in the output, the model imported folklore from its training data. Flag for review.

3. The confession: my guardrail was dead code and I didn't notice

The validator originally had a third, stronger check: a whitelist assertion that every stem, branch, and "ten god" term in the output came from the chart JSON. Code-reviewing it months later, I found both of its loops were asking whether a set contained items taken from that same set — a condition that is always true, wired to a check that could therefore never fire. Two supporting arrays were never read at all. It had caught zero violations, ever, and couldn't.

I deleted it and wrote a comment explaining why, including what a real version would need to solve (the same single-character collision problem as above). Two lessons I now apply everywhere:

A guardrail that cannot fire is worse than no guardrail — it shows up in every architecture diagram and code review as "we validate that," and everyone stops thinking about it.
Test your validators the way you test code: with inputs that must fail. A validation function with no failing test case is a hypothesis, not a gate.

Bonus fight: prompt-language gravity

The persona and all instructions are written in Chinese; the app also serves English readings. A one-line "respond in English" does not survive contact with a 2,000-character Chinese prompt: the model would ship hybrid sentences into production — actual example: "Your盘的里，其实事业和财这两条线比性格更有讲头" — lifted straight from a Chinese example sentence inside the persona. The fix that held: an explicit paragraph stating that every quoted sentence above is a tone demonstration only, must not be copied or translated, and that the output may contain no Chinese characters except glossed pinyin. When your prompt is bilingual, the language directive has to out-shout the entire rest of the prompt.

Why bother, beyond truthfulness

Cost. The model writes 300 words of styled prose per section instead of "reasoning" about the chart. No chain-of-thought needed — thinking mode is off, calls are ~1.5s and fractions of a cent.
Consistency. Two users with the same chart get stylistic variation on the same facts, not two different fates. Re-reads don't contradict.
The line is auditable. When a user asks "why does it say that?", there's an engine fact to point to — the same one published on the site's method page.

The pattern is old, honestly — it's a compiler emitting facts and a pretty-printer rendering them. The only new part is that the pretty-printer went to art school and will invent facts if you let it. Don't let it: compute the truth, mark the holes, validate the output, and test that your validators can actually fail.

The app: auspiceoracle.com — the engine's scoring constants are public on the method page, which is the same "show your work" rule applied to marketing.

I spent a month doing AEO for a tiny niche site. Most of the advice was wrong.

Shan Liu — Tue, 04 Aug 2026 21:35:23 +0000

"Answer Engine Optimization" is the new gold rush: get your site cited by ChatGPT, Perplexity, and Google's AI Overviews. There is an entire cottage industry selling advice on how — add schema markup, publish more pages, buy a tool.

I run a small bilingual Chinese-astrology calculator (auspiceoracle.com). It's about as niche and low-authority as a site gets, which makes it a decent lab rat: zero brand signal, zero backlinks, nothing to confound the measurement. Before writing a single content page I did two things most AEO advice skips — I read the actual studies, and I set up measurement before launch. Here's what survived contact with the data.

Finding 1: schema markup is not an AEO lever

This one hurt, because adding JSON-LD is the single most-repeated piece of AEO advice.

The best evidence available is an Ahrefs difference-in-differences study: 1,885 pages that added JSON-LD, each matched to 3 control URLs on other domains at similar pre-period citation levels, 30-day windows, four statistical approaches. Result:

Platform	Citation change after adding JSON-LD
Google AI Overviews	−4.6% (small but significant decline)
Google AI Mode	+2.4% — indistinguishable from zero
ChatGPT	+2.2% — indistinguishable from zero

The much-quoted counter-stat ("AI-cited pages are 3× more likely to have JSON-LD") is confounding, and Ahrefs says so themselves: schema lives on better-maintained sites. Four independent mechanism studies agree on why — when LLMs fetch a live page they extract visible HTML and ignore the structured-data layer. One test planted facts that existed only in FAQ schema; no platform used them. Another fed models deliberately invalid schema and they happily extracted from it — the script block is being read as plain text.

What I kept: an extractable, plain-language definition in the first two visible sentences of every content page. That's the thing the machines actually read. Schema stays on the pages as cheap rich-result table stakes, but I budget zero AEO effort against it.

Finding 2: page volume is a weak signal, and you can't shortcut brand

The other standard advice is programmatic content: generate hundreds of pages, win on surface area. Ahrefs' correlation study across 75,000 brands ranks the signals that track AI visibility:

Signal	Spearman ρ
YouTube mentions	~0.74
Branded web mentions	0.66–0.71
Branded search volume	0.35–0.47
Domain Rating	0.27–0.33
Number of site pages	~0.19
Backlinks	~0.18–0.23

Read the fine print, though, before acting on any row: the sample is filtered to DR>40 brands, the correlations are zero-order (nobody partialled out brand size), and there's no independent replication. The honest inference isn't "make YouTube videos" — it's that AI visibility tracks composite brand prominence, which a new site does not have and cannot fake with page count. For a small site, both the vendor pitch ("more pages!") and the counter-pitch ("pages don't matter!") are extrapolations from a population you're not in. The studies literally sampled pages that already had 100+ AI citations. Yours have zero. Nobody has published data about you.

Finding 3: don't out-define Wikipedia; map the entity gaps

My original plan assumed the English terminology in my niche was unclaimed. It wasn't — Wikipedia holds the head term with an actively-growing article, and LLMs demonstrably over-index on encyclopedic sources. Any page whose job is to out-define Wikipedia is dead on arrival.

But the MediaWiki API tells you something more useful than "Wikipedia exists": which sub-concepts have no article and no redirect. In my niche, a half-dozen core glossary terms return missing — definitionally seated at the head, structurally scattered below. That gap map, not keyword volume, became the content plan.

The part nobody sells: measurement

AI crawlers have no submission channel. You can't ping GPTBot. Zero crawls means "not discovered yet," not "misconfigured." The only proactive lever is indirect: Bing's index feeds OpenAI's and Copilot's retrieval, so IndexNow (one key file + one POST per publish) is the single highest-leverage submission you can make. Everything else is external links doing discovery work.

My production box runs Next.js behind a tunnel with no nginx, so there were no access logs to mine. The fix was one line in the middleware — match the AI user-agents, console.log a line, and the process manager's logs become the dataset:

# UA × hit count, from pm2 logs
grep -h '[ai-bot]' ~/.pm2/logs/app-out*.log | awk '{print $3}' | sort | uniq -c | sort -rn

Two distinctions matter when you read those logs:

GPTBot / ClaudeBot / PerplexityBot = your page entered a crawl queue. Necessary, not sufficient.
ChatGPT-User / Perplexity-User / Claude-User = a human saw your site cited in an answer and the assistant fetched the page for them. This is the metric. Everything else is leading-indicator noise.

And one thing you cannot retrofit: a baseline. Before the pages went live I ran a fixed panel of six prompts through ChatGPT, Perplexity, and Google (18 cells), recorded who got cited, and archived screenshots. All 18 cells: zero citations of us, as expected. The discipline is the same panel every month, questions never edited — change the questions and you've changed the ruler. Without the pre-launch zero row, any future citation could be "maybe we already had that."

Early returns, for honesty's sake: on launch day one crawler (ClaudeBot) fetched all ten new pages exactly once each, like it was walking a checklist. The others: zero. Citations: zero. This is a 90-day experiment, not a success story — which is exactly why the baseline row matters.

The contrarian call: let the training crawlers in

Standard advice for content sites is to allow retrieval bots but block training crawlers (CCBot, GPTBot-as-trainer, Google-Extended). I did the opposite — explicit allow for everything.

The reasoning is cold-start economics. Nobody's model "knows" my site's terminology or that it exists. Being ingested into training data is how that changes, and the lag is a full model generation — a cost you pay now for visibility later. Blocking training crawlers protects content whose value is exclusivity; a new site has none. I wrote down the reversal condition (if content gets scraped-and-republished at scale, or citations stabilize, revisit), which keeps it a decision instead of a default.

What I'd tell you to do

Write the definition in the first two visible sentences. Skip the schema debate entirely.
Map entity gaps with the MediaWiki API before writing anything.
Set up IndexNow; accept that everything else is discovery-by-links.
Log AI user-agents at the edge, and learn the -Bot vs -User distinction.
Record a citation baseline before launch. Same prompts, monthly, forever.
Treat every AEO study as data about someone else's population until your own logs say otherwise.

The meta-lesson: AEO in 2026 is where SEO was in 2003 — long on vendors, short on mechanisms. The mechanisms are checkable. Check them.

The site in question, if you want to see the "extractable first two sentences" pattern live: how the engine works, and the true solar time deep-dive that became the first post in this series.