DEV Community: Zhuoli Liang

Which clause wins? I benchmarked whether LLMs follow the rules that actually govern

Zhuoli Liang — Wed, 07 Oct 2026 16:04:27 +0000

This is a submission for the Kaggle Benchmarking Challenge.

What I benchmarked, and why

I enter a lot of contests and bounties. The rules for a single contest are never on one page. There's a landing page, an FAQ, a contest-rules page, and the platform's general Official Rules, and they drift apart. While building Fine Print (an agent that reads contest rules for you), I kept hitting the same question: when the friendly FAQ says one thing and the Official Rules say another, does the model notice which one controls?

So I built Fine Print Bench: fictional rule packets, each with a plain question ("What is the controlling close instant in UTC?", "Is a team of five allowed?", "Which required artifacts are missing?"). The model has to return a typed answer, the minimal clause IDs that justify it, and a confidence. 160 items in two splits:

easy (80 items): short packets, one precedence hop, eight categories:

Category	What it isolates
Explicit authority	Rules vs. FAQ/landing/blog when a precedence sentence exists
Scope & amendments	A newer FAQ is not an Amendment; an amendment that only names Track B
Unresolved authority	Equal-authority pages with no tie-break, plus controls where the pages agree
Exact deadlines	DST ending the same day, the date line, a submission 30 seconds late
Business-day windows	Holidays, "counting the decision date as day one", a trigger date nobody gave you
Eligibility	Missing residency, a subsidiary employer, age of majority â‰ 18
Submission artifacts	Conditional requirements, a draft post, a private repo
Notification & payout	"first attempted notification" vs. "after you receive our email"

hard (80 items): real-length rules (~800 words of boilerplate around the deciding clause), two-hop precedence (Amendment â†’ Contest Page â†’ Rules, withdrawn amendments, circular deference), IANA zone names around DST gaps/overlaps and the date line, business-day chains in a named zone, multi-turn corrections that flip the answer, compound packets, and aggregation over 6â€“40 entrant records.

About a fifth of the packets have no unique answer, and the correct output is insufficient_information. "Always abstain" can't win, because determinate and unresolved accuracy are reported separately.

The twist: every packet appears twice. The second copy is semantically identical. Only the source order, the fact order or the record order changes, or an irrelevant "About the organizer" page is added. A model that reads rules carefully should give the same answer both times.

Scoring (no LLM judge)

Strict accuracy: the status and value are correct (timestamps compared as instants; local-time questions must also use the requested offset), and the cited clauses equal a minimal evidence set.
Answer-only accuracy: the value alone, so citation strictness and reasoning errors can be told apart.
Presentation flip rate: the share of packet pairs where the answer (status + canonical value) changed after a harmless reorder. Set order, whitespace, offset notation and the reason label don't count as flips.
Confident-wrong rate: the share of answers with self-reported confidence â‰¥ 0.9 that were wrong.
95% CIs come from a bootstrap over packet families (40 per split). Both copies of a packet are resampled together, so 80 correlated items aren't treated as 80 independent ones.

Every calendar answer is recomputed by a second, independent oracle (Python zoneinfo); every aggregation answer by a second implementation that parses the clause text itself. There are corruption tests for the scorer itself. Every model gets the same plain-text prompt and the same tolerant parser (reasoning blocks stripped, JSON pulled from code fences); rate limits and outages are retried and reported as coverage, never scored as wrong answers.

Models

Kaggle's default benchmark model (google/gemini-3.7-flash) for the leaderboard task, then the same 160 items on google/gemini-2.5-flash, google/gemma-4-31b, qwen/qwen3-235b-a22b-instruct-2507, deepseek-ai/deepseek-r1-0528 and openai/gpt-oss-120b: two generations of the same provider (does a newer model fix authority errors?), an open-weights model from the same family, and three other open-weights families including two reasoning models.

Results

Final results (run 4, 2026-10-07 PT). The default model's row is the leaderboard run (Kaggle run 3, notebook Version 1); gemini-2.5-flash is from run 2 (same items except one clarified packet, see below). Qwen3-235B, DeepSeek-R1 and Gemma-4-31B are from the run 4 exploration (results/run4/explore). Qwen and DeepSeek answered all 160 items. Gemma has 150 of 160 results: 10 hard items are missing because Kaggle's model proxy rate-limited them, so its hard score is over 70 items, not 80. gpt-oss-120b has no results at all: Kaggle's model proxy kept returning HTTP 429 (rate limit) on every attempt, so I report nothing for it rather than counting the failures as wrong answers. Numbers are strict accuracy unless noted; flip rate counts packet pairs; confident-wrong = wrong among answers with confidence â‰¥ 0.9. 95% CIs (brackets) were computed for the two Gemini rows only.

Model	Easy strict	Hard strict	Hard answer-only	Hard flip rate	Hard confident-wrong
gemini-3.7-flash (default, leaderboard run)	0.975 [0.938, 1.0]	1.000	1.000	0.000	0.000
gemini-2.5-flash	0.975 [0.938, 1.0]	0.850 [0.775, 0.925]	0.850	0.275	0.150
gemma-4-31b (150/160; 70 hard)	0.988	0.929	0.929	0.057 (2/35)	0.071
qwen3-235b-a22b-instruct-2507	0.738	0.275	0.350	0.400 (16/40)	0.646
deepseek-r1-0528	0.900	0.650	0.688	0.200 (8/40)	0.312
gpt-oss-120b	no results: every call got HTTP 429 from Kaggle's model proxy

Notes on these numbers. The default model answered all 160 items correctly (answer-only 1.000, zero flips) in the leaderboard run; its two strict misses on easy were citations written as rules:S1 instead of S1. The scorer now accepts a source-qualified ID when the prefix really is that clause's source, which makes it 160/160 â€” but the leaderboard number stays 0.975 + 1.000 as run. In run 2 it scored 0.988 / 1.000. For gemini-2.5-flash (run 2): one easy packet (payout_02) now also accepts citing R2 alone, and one hard packet (h_mix_03) had a judge-supplier clause that could be read as a competing permission, so I clarified its wording; on the 78 unchanged hard items it scores 0.872. The pending runs use the clarified keys.

gemini-2.5-flash per hard category: aggregation_xl 0.6, tz_dst 0.7, business_day_chains 0.8, compound 0.7; long distractors, two-hop precedence, multi-turn and 6â€“12-record aggregation all 1.0. Its easy split is near ceiling (0.975).

Overall strict accuracy (all items): Gemma-4-31B 0.960 (n=150), DeepSeek-R1-0528 0.775 (n=160), Qwen3-235B 0.506 (n=160).

Per-category strict accuracy, both splits combined (run 4 exploration; results/run4/fine_print_per_category.csv):

Category	DeepSeek-R1	Gemma-4-31B	Qwen3-235B
explicit_authority	1.0	1.0	0.7
scope_amendment	1.0	1.0	0.7
unresolved_authority	1.0	1.0	1.0
two_hop_precedence (hard)	1.0	1.0	0.6
long_distractors (hard)	1.0	1.0	0.2
multi_turn (hard)	0.7	1.0	0.8
exact_deadlines	0.3	0.9	0.6
tz_dst (hard)	0.3	0.6	0.0
business_days	1.0	1.0	0.4
business_day_chains (hard)	0.9	1.0	0.2
eligibility	0.9	1.0	0.8
submission_artifacts	1.0	1.0	1.0
notification_payout	1.0	1.0	0.7
compound (hard)	0.6	0.9	0.4
aggregation (hard)	0.5	1.0	0.0
aggregation_xl (hard)	0.2	n/a (rate-limited)	0.0

Format errors: zero for every model that returned answers. The shared parser recovered JSON from every Gemma, Qwen and DeepSeek response, so every miss above is a real wrong answer or a wrong citation, not a parsing failure. The only non-answers were gpt-oss-120b's HTTP 429s and Gemma's 10 rate-limited items, and those are counted as missing coverage, not as errors.

What surprised me

1. The 31B model beat the bigger ones. Gemma-4-31B scored 0.988 easy / 0.929 hard strict, ahead of DeepSeek-R1 (a 671B reasoning model, 0.650 hard) and Qwen3-235B (0.275 hard). It never followed an FAQ over the Rules and never broke on long distractors or two-hop precedence. Its only real weak spot was DST edge cases (0.6), which also hurt every other model. The caveat: Gemma's hard score covers 70 of 80 items, because the 10 largest aggregation items were rate-limited. Even if it had missed all 10, it would still be ahead of DeepSeek on hard.

2. Qwen is confidently wrong about which clause governs. Qwen3-235B is the only model that actually lost authority questions: 0.6 on two-hop precedence, 0.2 on long-distractor packets, 0.7 on explicit authority. On those misses it usually cited the wrong clause (for example, a Track page over the Amendment that replaced it) with confidence 0.95â€“1.0. On the hard split, 65% of its answers that claimed confidence â‰¥ 0.9 were wrong. Its self-reported confidence carries almost no information.

3. Flip rate falls as accuracy rises. On the hard split: Qwen 0.275 strict / 0.40 flips, DeepSeek-R1 0.650 / 0.20, Gemma 0.929 / 0.057, gemini-3.7-flash 1.000 / 0.000. (gemini-2.5-flash, 0.850 / 0.275, is the outlier: it flips more than its accuracy predicts.) When a model gets a hard item wrong, a harmless reorder often gives a different wrong answer. In other words, instability tells you the model doesn't know. It is a cheap and useful signal.

4. Reasoning didn't fix the calendar. DeepSeek-R1 writes long chains of thought, but it scored 0.3 on exact deadlines and DST. On several DST-gap packets it answered insufficient_information with confidence â‰¥ 0.95, even though the packet said exactly what to do when the local time doesn't exist. It did fine on authority (1.0 on two-hop and long distractors), so its failures are arithmetic and abstention, not reading.

The older Gemini breaks on presentation, not authority. gemini-3.7-flash, Kaggle's default model, gave the same correct answer to every packet in both orders. gemini-2.5-flash got every precedence question right, but its hard-split losses move with harmless reorders: on 11 of 40 hard packet pairs its answer changed between the two copies.

Three concrete cases (rule: the most common error type in each failing hard category, gemini-2.5-flash, run 2):

Reordering a 40-record list changed the answer (aggregation_xl). Same clauses, same records, record order reversed: the base copy and the reversed copy disagreed on which IDs are disqualified in 4 of 5 packets; every one of those answers, right or wrong, had confidence â‰¥ 0.9.
A DST gap with an explicit rule became "insufficient information" (tz_dst). "Entries close at 2:30 AM on October 4, 2026, Australia/Sydney local time. If that local time does not exist because of a daylight-saving change, the deadline is the first valid local time after it." The base copy answered insufficient_information (missing_fact, confidence 1.0); the variant, with an irrelevant page added, answered the correct 2026-10-03T16:00:00Z.
Off-by-one in a Tokyo business-day chain (business_day_chains). Paperwork due three business days after first attempted notification, dates in Asia/Tokyo, with a listed holiday: the base copy got the correct 2026-11-05; the variant, with the two clauses in reverse order, said 2026-11-04 (confidence 1.0).

From run 4, using the same rule:

Qwen3-235B: the wrong governing clause, with high confidence (two_hop_precedence). In h_hop_04 and h_hop_05, an Amendment overrides a Track page, which overrides the Rules. Qwen cited the Rules plus the Track clause, skipped the Amendment, and answered at confidence 0.95â€“1.0. On h_hop_05 the two copies even disagreed (true vs. false).
DeepSeek-R1: abstaining on a DST rule it was given (tz_dst). For h_tz_01, h_tz_03 and h_tz_05, it answered insufficient_information at confidence 0.95â€“1.0, although the packet contained an explicit "if that local time does not exist" rule. Reasoning tokens didn't fix this. Both reasoning-heavy models I could run treated the tricky clock as missing data.
Gemma-4-31B: the same failure in a smaller dose (tz_dst). It abstained on the same DST-gap packet in both copies, and on a Nepal (+05:45) packet one copy abstained while the other committed to an answer.

Gemma and DeepSeek never followed an FAQ or landing page over the Rules when a precedence sentence was present. Qwen was the only model with results that did (explicit authority 0.7). gpt-oss-120b has no results (every call got HTTP 429), so I can't say whether it fixes the arithmetic.

What it changed about how I build agents

I went in expecting authority to be the hard part, meaning whether the model knows the Rules beat the FAQ. For the good models, it isn't. Gemma and both Gemini models basically never chose the wrong source. What broke was arithmetic (DST, business days, counting records) and consistency (the same question in a different order got a different answer).

So this is how I build Fine Print now, and how I'd build any agent that reads documents with legal weight:

The model picks the clause. Code does the math. The LLM's job is to say "R4 governs, and here is the instant it names". Deadlines, time zones, business-day windows and record filters go through deterministic code with zoneinfo and a holiday list. Every model I tested was weakest exactly where a Python function would be perfect.
Run it twice, shuffled. Every extraction runs twice with the sources reordered. If the two answers disagree, it goes to a human. The flip-rate result is why: disagreement is cheap to detect, and it shows up most on the items the model actually gets wrong.
Don't trust self-reported confidence. Almost every answer from every model claimed â‰¥ 0.9. For Qwen, 65% of those were wrong on hard items. Confidence isn't the signal. Agreement and verifiable citations are.
Pick the model by measurement, not size. A 31B open model beat a 235B and a 671B model on this task. I would never have guessed that, and I'd have paid for the bigger one.

None of this is new advice. But I wouldn't have known which parts to move into code until I measured where the models actually fail.

What I'd measure next

The same packets with deterministic date tools available: does the tool fix the arithmetic and the order-sensitivity?
More presentation variants per packet (k > 2), so flip rate has tighter intervals
Real (licensed) rule pages instead of fictional ones

Where to see it

Benchmark on Kaggle: https://www.kaggle.com/code/zhuoliliang/new-benchmark-task-e35ea
Code, items and tests: https://github.com/Zhuoli/fine-print-bench

Built with AI coding agents (Codex and Grok). I designed the question and checked the gold answers. Two different agents brainstormed the categories and implemented the scorer, and a second oracle checks every calendar answer.

Out the Door: tell Gemma what you want to do outside, get one time, close the tab

Zhuoli Liang — Mon, 05 Oct 2026 21:26:08 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Out the Door answers one question: when should I go outside today?

I work for myself, at a desk, in Seattle. "I'll go for a walk later" is a very easy promise to break. And when I do mean it, I open a weather app and start reading 72 rows of hourly numbers. Is 40% rain at 3 PM worse than 15 km/h wind at 5? Does a 45-minute walk still fit before sunset? Ten minutes later I'm still on my phone, and it's getting dark.

So the app is built to be the shortest part of going outside:

Say what you want in your own words: "take the pup around the lake once I log off at 5:30, I don't want to get rained on".
Gemma, running on your own computer, turns that into a plan: activity, duration, time limits, which day, what to avoid.
Plain code scores every hour of the free Open-Meteo forecast for that activity (rain chance and millimetres, wind and gusts, feels-like temperature, UV, daylight from sunrise and sunset) and picks the best window.
You get one time, what to bring, and a 3-sentence "door note" from Gemma. Then three buttons: add the reminder (an .ics event with a 15-minute alarm), save a pocket card (it opens with no signal), and I'm going, which turns the whole screen green and tells you to close the tab.

When you come back, it asks "Did you go?" and counts your minutes outside for the week, on your device only.

It's for anyone who means to go outside and then doesn't: people at desks, dog owners, the "I'll garden this weekend" crowd, and birders who want the morning when the wind is down.

Demo

Live app: https://out-the-door-jade.vercel.app

Here's a real run from today in Seattle with Gemma 4 E2B. I typed "take the pup around the lake once I log off at 5:30, I don't want to get rained on":

Gemma read it as a 45-minute walk, today, after 5:30 PM, avoiding rain. (45 minutes is the default walk length, since I didn't give one.) The forecast was dry and calm, so the best window was 6:00â€“6:45 PM, and because that ends 5 minutes after the 6:40 PM sunset, the note says to bring a headlamp. Gemma wrote: "Leave at 6 PM and be back by 6:45 PM. Bring a headlamp since you finish near sunset. Look for the light as you walk." The code checked every time and number in it against the forecast.

The public demo works without installing anything in "No model" mode (simple keyword rules), and there's an experimental in-browser Gemma 3 1B. The full version is Gemma 4 E2B on your own laptop through Ollama:

ollama pull gemma4:e2b
OLLAMA_ORIGINS="https://out-the-door-jade.vercel.app" ollama serve

Then pick "Gemma 4 via Ollama" in the app and click Load Gemma.

Code

Zhuoli / out-the-door

Find the hour to go outside, then close the tab. Gemma reads your plan on your own device; code picks the window from an open forecast. DEV Hacktoberfest 2026 Week 1: Touch Grass.

🌿 Out the Door

Find the hour to go outside, then close the tab. Say what you want to do in plain words ("45 min walk with the dog before dark, I hate wind"). Gemma, running on your own device, turns that into a plan; code scores the free Open-Meteo forecast and picks the best window; you get a calendar reminder and a pocket card that still opens with no signal.

Live: https://out-the-door-jade.vercel.app · Built for the DEV Hacktoberfest 2026 Open-Source AI Challenge, Week 1: Touch Grass. Repo started Oct 5, 2026.

Why

The screen should be the shortest part of going outside. Weather apps make you read 72 hourly rows and do the math yourself ("is 40% at 3 PM worse than 15 km/h wind at 5?"), and by then you've spent ten minutes on your phone and it's dark. Out the Door asks one question, gives…

View on GitHub

How I Built It

It's a static site: HTML, ES modules, a service worker, and no backend. One rule shaped the code: Gemma does the language and code does the arithmetic, and code checks everything Gemma writes.

Job	Who does it
Read "after I log off at 5:30, take the pup around the lake" into JSON: `{activity, durationMin, earliest, latest, days, avoid}`	Gemma
Check and repair that JSON: unknown activity becomes walk, bad times are dropped, durations are clamped, "stargazing but avoid the dark" is fixed. Every repair is shown to you	code
Score every hour and pick non-overlapping windows. A window scores 0.6 Ã— its average hour plus 0.4 Ã— its worst hour, so one soggy hour sinks it. Birding gets a dawn bonus, foliage a golden-hour bonus, and sooner beats later	code
If nothing good fits your limits, offer the best time outside them (a Phoenix "run after work" in a 36 Â°C afternoon moves to 6 AM)	code
What to bring	code, from the numbers
The door note: when to leave, what to bring, one thing to notice	Gemma, given only that window's facts
Check the note: every clock time must be the window's start, end or sunset, every number must be in the forecast, and no "rain" in a dry window. If it fails, you see a plain note first, with Gemma's version and the reason underneath	code

The open pieces:

Gemma 4 E2B (gemma4:e2b, Q4_K_M) through Ollama, on your own machine. It reads your request and writes the note.
Gemma 3 1B (onnx-community/gemma-3-1b-it-ONNX) through Transformers.js in a Web Worker (WebGPU, or WebAssembly on CPU). It's a one-time 1 GB download, then cached.
Open-Meteo for the forecast and town search: open data, free, no API key.

I measured it, and the numbers changed the design

I wrote two hand-labeled sets of requests. The first 20 I wrote while building, alongside a keyword-rules baseline. The other 12 I wrote after freezing the rules and the prompt, phrased the way people actually talk: "stretch my legs for twenty minutes", "I burn easily", "I melt above 20 degrees". A request only counts if all six fields are right.

Reader	Dev set (20)	Held-out set (12)	Avg per request (CPU)
Keyword rules, no model	15/20	0/12	<1 ms
Gemma 4 E2B via Ollama	13/20	7/12	5â€“6.5 s
Gemma 3 1B in Transformers.js	0/20	0/12	3.8â€“4.5 s

Three things I learned:

My keyword rules were overfit to my own examples. 15/20 on the set I wrote them against and 0/12 on new phrasing. "Twenty minutes" has no digit, "I burn easily" means avoid sun, and "tramp through the woods" is a hike. On the held-out set, Gemma 4 E2B got the activity, the duration and both time limits right on 12 of 12. Its misses were judgment calls: it read "once I log off at 5:30" as today, and it missed "avoid wind" in "I don't want to get blown around on the bike".
The first prompt let Gemma invent limits. On version 1 it scored 12/20 on the dev set and turned "leaf peeping, back before dark" into 17:00â€“21:00. Telling it "use null unless they named a time of day", and that "before dark" is something to avoid rather than a time limit, plus mapping 00:00â€“23:59 to "no limit" in code, got it to 13/20, and the invented limits went away.
1B was too small for this job. Gemma 3 1B copied the example's avoid list into almost every answer. I tried three prompt variants (one example, no example, everything in a single user turn) and all scored 0/20 and 0/12. So in the browser it only writes the note, and the keyword rules do the reading. I'd rather ship that honestly than pretend.

The notes, against six real forecasts I saved on October 5 (sunny Seattle, Bergen in steady rain, Phoenix in a heat spell):

Gemma 4 E2B: 6/6 passed the code check, and when I read them by hand they were all accurate but plain ("Look for the low golden light before sunset.").
Gemma 3 1B: 4/6 passed. The check caught "rain-soaked streets" in a dry window and a note that never said when to leave. It can't catch nature facts: 1B called a Pacific Wren "iridescent blue" and sent a Phoenix runner "through the redwoods". That's why code chooses the window and only Gemma's wording is ever at risk.

All raw outputs, misses included, are in results/. There are also 12 unit tests covering parsing and repair, scoring, daylight, overlap, the note checker and the .ics time-zone math.

I built this with an AI coding agent pairing with me. The design calls, the labeled test sets and the decision to demote the 1B model came from the measurements above.

Why Does Open Innovation Matter?

Where I'm going stays on my computer. "Take the pup around the lake once I log off at 5:30" is a routine, a schedule and a place someone lives near. With an open-weight model on my own laptop, the only thing that leaves my device is a coordinate sent to a free forecast API. There's no account, no analytics and no server of mine.
It costs nothing to run, so it can stay simple. There's no per-token bill and no key to hide, so it's a static page anyone can fork and host.
I could swap models and measure them. The most useful thing I did was run the same prompts through Gemma 4 E2B and Gemma 3 1B and compare them against a no-model baseline on requests I hadn't tuned for. That's how I found out my rules were overfit and the 1B model wasn't ready for this job. With a closed API I'd have one model, a version that can change under me, and no way to run it offline.
The pocket card works with no signal. The plan, the forecast it came from and the app itself are cached on the device, so the screen part ends before the trailhead.

I haven't taken it on a real trail yet. When I do, I'll add what I learn to the README.

Prize Categories

Best Use of Gemma: Gemma 4 E2B via Ollama reads the request and writes the door note, with Gemma 3 1B in the browser via Transformers.js as an experimental mode. Measured against a no-model baseline on held-out requests.

Lease Buddy: a private lease checker for a friend, with Gemma running on their own laptop

Zhuoli Liang — Sat, 03 Oct 2026 09:45:09 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

I built this for a friend who is about to sign a 12-month apartment lease: pages of dense legal English, plus an addendum. I can read it with them once, but I won't be there for the next lease, or the renewal letter. And a lease is exactly the kind of document you shouldn't paste into a random website: name, address, rent, signature.

So I built them Lease Buddy: paste a lease (or open a .txt / .pdf), and it gives you:

The short version: rent, due date, late fee, deposit, start and end dates, notice periods, early-termination cost. Every item cites its clause (§3, §13…).
Clauses that contradict each other. This is the part I care about most. In real leases the scary stuff usually isn't hidden; it's scattered. One clause says you need 60 days' notice to avoid auto-renewal, another says 30 days to vacate, and an addendum quietly moves the late-rent cutoff from the 5th to the 3rd.
Dates you can't miss, computed rather than guessed, with a .ics export and 7-day reminders. When two clauses disagree, it uses the earlier (safer) date and shows both.
Questions to send the landlord: each flag (auto-renewal, non-refundable fees, "reasonable notice" with no number, "tenant pays any repair under $250 regardless of cause", one-sided attorney fees) becomes a copy-paste question rather than legal advice.
Ask Gemma. "Can my landlord come in whenever they want?" or "Explain §13 in plain words." Gemma runs in the browser tab or on your own computer, never on a server.

Demo

Live: https://lease-buddy-gemma.vercel.app. Click "Try the sample lease" (a fictional lease with three contradictions planted on purpose). The rule checks are instant and need no model. Then hit Start Gemma to ask questions.

Here's Gemma 4 (E2B, via Ollama on my dev box) answering the question that matters most in this lease. The yellow line comes from the rule checker, not the model. The green line is the code verifying Gemma's answer:

Code

Zhuoli / lease-buddy

Read your lease like a friend who has read a lot of leases. Finds contradictory clauses + key dates; Gemma explains them on your own device.

🏠 Lease Buddy

Read your lease like a friend who has read a lot of leases. Paste a lease (or open a .txt/.pdf). Lease Buddy finds the money, the dates, and the clauses that contradict each other. Then Gemma, running on your own device, explains them in plain words.

Live: https://lease-buddy-gemma.vercel.app · Built for the DEV Hacktoberfest 2026 Weekend Challenge ("Build for a Friend").

Not legal advice. Lease Buddy points at things to ask about. A local tenant-rights group or a lawyer can tell you what the law says where you live.

Why

I built this for a friend who is about to sign a 12-month lease written in dense legal English. The scary parts of a lease are rarely hidden. They're scattered: the renewal clause says one notice period, the "notice to vacate" clause says another, and an addendum quietly changes when rent counts as late…

View on GitHub

Plain HTML + ES modules, no build step, no backend, no API keys. Apache-2.0. npm test runs 25 unit tests (clause splitting, fact extraction, contradiction detection, date math, retrieval, the answer verifier, .ics).

How I Built It

I built it this weekend with an AI coding agent doing a lot of the typing, while I made the calls on scope, tests and the parts below.

The split that made it work: code finds the facts, Gemma explains them.

Deterministic layer (src/lease.js): splits the lease into clauses (including addendum ids like A2), reads number words ("sixty (60) days"), and pulls out typed facts like end_of_term_notice = 60 days @ §13. If two clauses give different values for the same thing, that's a contradiction. Dates are plain date math in UTC.
Retrieval: a tiny BM25 over clauses, plus a few tenant-language synonyms ("moving out" → vacate / non-renewal). It's contradiction-aware: if a retrieved clause is part of a contradiction, the clause that contradicts it gets pulled in too.
Gemma only sees those clauses, plus a note from the checker listing the contradiction, and is told to cite [§x] after each fact.
Verifier: every number in Gemma's answer must appear in a clause it was shown (and in the clause cited in that sentence), every [§x] must be real, and if a contradiction is in play, both sides must be mentioned. Otherwise the answer gets a ⚠️ that says exactly what's off.

Two ways to run Gemma, both on hardware you control:

In the browser: gemma-3-1b-it (ONNX) via Transformers.js in a Web Worker. WebGPU (q4f16, or q4 when the GPU lacks fp16 shaders) with a CPU/WASM fallback (int8). About a 0.8–1 GB one-time download, then cached.
Ollama on your computer: ollama pull gemma4:e2b, allow the site's origin with OLLAMA_ORIGINS, and the page streams from localhost:11434.

I measured it instead of eyeballing it. npm run gemma:check runs the real prompts against the sample lease and saves every answer to results/ (all on CPU):

Model	Q&A answers passing the checker	Clause explanations passing	Same 3 contradiction questions without the checker's hint
Gemma 4 E2B (Ollama)	7/7	3/3	2/3
Gemma 3 1B (the in-browser model)	5/7	2/3	1/3

Three things I learned:

Small models answer from one clause. Without the hint, Gemma 4 answered "how much notice before moving out?" with "at least thirty (30) days' written notice [§14]". That's true, but it skipped §13's 60-day non-renewal deadline, which is the one that locks you into another year. The system prompt already said "if two clauses disagree, say so and cite both", and that wasn't enough. When the checker's note naming the specific contradiction was added, it cited both.
The verifier earns its keep on the 1B model. It caught Gemma 3 1B citing numbers to the wrong clause ("thirty days … [§13]"), inventing a clause id (§A9), and turning "sixty (60) days" into "six months".
It can't check meaning, and I'd rather say so. 1B once said "the lease doesn't allow you to bring a cat" when the clause says pets need written consent. That's why each answer shows the exact clauses Gemma saw, why the summary, contradictions and dates come from code, and why the app labels the 1B option "small, sometimes cites the wrong clause" and recommends Gemma 4 for answers.

Also: the first in-browser run crashed with "Program Sub requires f16 but the device does not support it". Lots of GPUs don't expose shader-f16, so the worker now checks the adapter, picks q4 instead of q4f16, runs a 2-token smoke test, and falls back to the CPU if the GPU path still fails.

Why Does Open Innovation Matter?

Because of what a lease is: legal name, address, rent, sometimes income and a signature. Uploading that to someone else's server to get it explained is exactly the wrong move. With an open-weight model:

The document never leaves the device. The site is static files on Vercel. Gemma runs in the tab (Transformers.js) or on your own machine (Ollama). There's no server I could log to, even by accident.
It costs nothing to run, for my friend or for me. No API key, no per-token bill, no account.
I could pick the model size to match the job. A 1B model that loads in a browser for a quick private look, or Gemma 4 E2B on a laptop when you want answers you can lean on. The same prompts and the same verifier work for both, and I could measure both.
The open stack let me fix the hard part. Swapping models, reading their failure modes in results/, and adding a verifier around them is how the "answers from one clause" problem got solved. A closed API would have given me a nicer-sounding answer with the same blind spot.

Not legal advice: Lease Buddy tells you what to ask. For what the law says where you live, a local tenant-rights group is the right call. Next step: going through my friend's real lease with them.

Prize Categories

Best Use of Gemma: Gemma 3 1B runs fully in the browser via Transformers.js (WebGPU/WASM), and Gemma 4 E2B runs locally through Ollama. Both are wrapped in contradiction-aware retrieval and a code verifier, with a measured comparison in the repo.

Fine Print: an agent that knows when the FAQ and the Official Rules disagree

Zhuoli Liang — Sat, 03 Oct 2026 07:51:06 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

Fine Print reads contest rules so you don't miss the clause that disqualifies you.

I enter a lot of hackathons and bounties. The rules for one contest are never on one page. There's a landing page, an FAQ, a contest-rules page, and the platform's general Official Rules, and they drift apart. This very challenge is a good example. Its own pages disagree in ways that matter:

When does the winner paperwork clock start? The FAQ gives you 7 business days "following the date of your first email notification". The MLH Official Rules say "following the date of first attempted notification", and add: "Failure to comply with these deadlines may result in forfeiture of the prize." Under the Official Rules, an email sitting in your spam folder has already started the clock. And the Official Rules govern: "In the event of a conflict between these Official Rules and the applicable Contest Announcement Page, the Official Rules will govern and control."
Who is "Sponsor"? The landing page says "Sponsored by Sanity". The contest-rules page says "Sponsored by Major League Hacking PBC Inc ("Sponsor")". The employee exclusion is written against Sponsor. Can a Sanity employee enter? The rules don't say. Fine Print says exactly that and tells you to ask, instead of guessing.
Can I reuse a project? The FAQ says riffing on prior work is encouraged. The Official Rules require that "development of your Entry was started during, and not prior to, the Entry Period".
"Submissions due: October 04, 2026" is 2:59 PM on October 5 in Beijing.

Tell Fine Print who you are (country, time zone, age, employer, whether you use AI, whether an agent submits for you). Then ask a plain question. The offline results include clause IDs and a tool trace with source quotes and URLs. Model-generated citation correctness is not yet automatically verified.

The local snapshot covers three contests: this one, the DEV Kaggle Benchmarking Challenge, and a Superteam Earn bounty (which turns out to be HUMAN_ONLY, so an agent can draft for you but not submit).

Demo

Live app: https://fine-print-opal.vercel.app
Shareable demo questions:
Studio: https://fine-print.sanity.studio/ (sign-in required; the ⚠️ Open conflicts desk item is the review queue)

Code

https://github.com/Zhuoli/fine-print

Clone it and run npm install && npm run seed:build && npm run dev. It works with no accounts at all: GROQ runs locally via groq-js over the same seed, the Context tools fall back to same-named local shims, and with no LLM key a deterministic planner calls the same tools.

How I Used Sanity

The content model is the product. Rules are prose, but what an entrant needs is computation: comparing instants, ranking sources, checking a country against a list. So every rule lives in Sanity twice: once verbatim (for citation), once normalized (for code).

ruleSource: a page, its kind (officialRules / contestRules / landingPage / listing), a precedenceRank, and the exact sentence that establishes precedence.
clause: a topic, a verbatim quote, a normalized JSON value ({"minAge":18}, {"instant":"2026-10-05T06:59:00Z"}, {"excludedCountries":[…]}), a reference to its source, and contests[]. One clause in the shared MLH Official Rules binds every DEV contest, so I fix it once.
conflict: two or more claims[] (references to clauses) side by side, status, winningClaim, resolvedBy (precedence / precision / needsHuman), and a rationale.
contest: the close instant and the close as written, the source time zone, tracks with winners per path, and a payout timeline (business-day windows with their anchors).
organizer: legal sponsor vs. brand sponsor, because "Sponsor" in the rules isn't the logo on the page.

In the Studio, the first item in the desk structure is ⚠️ Open conflicts, a review queue of contradictions nobody has decided yet.

Sanity Context: one endpoint over the full dataset (no Knowledge Base).

The live app is wired to the project-level Sanity Context MCP endpoint for the whole production dataset, in GROQ mode with dataset embeddings enabled:
https://api.sanity.io/v2026-03-03/context/mcp/9wt4tu94/production?embeddings=true.
The server connects to it with a read-only Viewer token kept in a server-side environment variable. tools/list returns initial_context, groq_query, schema_explorer, and array_field_reader. Embeddings cover { title, name, topic, quote, note, rationale, closesAsWritten }. So semantic ranking works through the MCP: *[_type=="clause"] | score(text::semanticSimilarity("when is the prize money paid")) ranks clause.sanity.prize.rules, clause.tg.payout, and clause.official.notify first. With an LLM configured, the agent gets the schema overview from initial_context and uses groq_query to follow references. A typical call is one contest, its sources ordered by precedence, every clause that references() it, and every conflict with its claims dereferenced.

About Knowledge Bases, honestly: the Knowledge Bases option isn't shown in the dashboard for my Free-plan project, so I couldn't build one. The challenge page says: "Knowledge Bases are in beta and currently index up to 150 documents. Design for that budget, or point your agent at your full dataset through a Context MCP endpoint with embeddings enabled. Both count for Path One." Fine Print takes the second option. kb/knowledge-base.md holds the KB setup (dataset query, website sources, file bundle, standing instructions) that I'd apply once access is available. The app has a knowledge_base_read slot that currently falls back to a local shim over the verbatim snapshots in /sources.

What the public demo actually runs: the deployment has no paid LLM key. So the live demo uses the deterministic planner. It calls the same tools, and they run GROQ directly against the live Content Lake (9wt4tu94/production). The Context MCP is configured and verified, but the planner doesn't call the MCP tools itself. Add ANTHROPIC_API_KEY or OPENAI_API_KEY and the model plans over the MCP tools instead.

What the agent does with it. The model is never allowed to do date math or eligibility logic in its head. It calls five deterministic tools: check_deadline, check_eligibility, cash_timeline, submission_checklist, and source_conflicts. Each runs GROQ against Sanity and returns verdicts carrying clause ids. The system prompt requires a [clause.*] citation on every factual sentence, and requires the agent to say "the rules don't settle this" whenever a conflict is open.

A keyword search would return "October 04" and "seven (7) business days". Answering "I'm in Beijing, when must I submit, and if I win, when is my W-9 due and when does the money arrive?" takes the instant, the zone, the precedence ranks, and the anchored business-day windows. Fine Print converts the submission instant to Beijing time, but refuses to invent a cash date: winner selection is not recorded, paperwork is due 7 business days after first attempted notification, and delivery depends on acknowledged acceptance of completed paperwork.

Conflicts are modeled by hand (no KB build)

Without a Knowledge Base build, there are no machine-raised Issues to compare. The five conflict documents in the dataset are modeled by hand, each with side-by-side claims, a winning claim, and a rationale. Comparing them with a KB build's Issues is the first thing I'd do once access is available.

Honest limitations

The seed is curated: three contests and about 30 clauses. Extracting clauses from a new rules page is still a manual (AI-assisted) step. The next step is a Sanity Function that drafts clause documents from a new ruleSource for a human to approve.
Business-day math skips weekends but not public holidays.
The Kaggle challenge's landing page gives a date with no time. Fine Print says the time is unknown instead of assuming 11:59 PM PDT.
This isn't legal advice. Where the rules are silent, the agent says so.

Sanity Project Details

Project ID: 9wt4tu94
Dataset: production (60 documents, imported from seed/production.ndjson)
Studio: https://fine-print.sanity.studio/
Context MCP: https://api.sanity.io/v2026-03-03/context/mcp/9wt4tu94/production?embeddings=true (dataset embeddings enabled; needs a read token)
The dataset's visibility is public, but document IDs are dotted (clause.official.clock), and Sanity doesn't serve those to anonymous queries. So the app reads with a server-side Viewer token. To browse without an account, use npm run seed:build and read seed/production.ndjson, which is the exact imported content.

Reproducible offline evidence

Run npm test with no credentials. It checks deadline boundaries, unknown eligibility, request validation, payout-anchor uncertainty, seed references, and a counterfactual: keeping the quoted prose unchanged while changing the normalized minimum age changes the verdict. This demonstrates dependence on structured facts, but is not evidence of a live Context call. npm run check:mcp checks the live Context endpoint.

Agent Session

I built this with an AI coding agent. I wrote the brief and the content model decisions, and checked every quoted clause against the source pages. The agent wrote most of the code.