DEV Community: Asuran

Frostline: what to plant this week, from your local frost dates

Asuran — Sat, 10 Oct 2026 16:34:40 +0000

Most garden advice is a fixed calendar. But the right week to sow a carrot in Portland is not the right week in Minneapolis. A calendar does not know that. Your frost dates do.

Frostline asks for your location and tells you exactly what to sow this week, from your own last-spring and first-fall frost dates. The screen is the shortest part. Read the plan, close the laptop, go plant.

Live demo: https://frostline-lime.vercel.app
Code: https://github.com/zkasuran/frostline

What it does

Type a US city or tap "use my location". Frostline shows your two frost dates, then a sow-this-week list drawn from 29 common vegetables and herbs, each with a plain reason like "sow kale now, it takes a light frost and keeps going into the cold". It reads at a glance and needs no sign-in.

Open-source AI is the core, not the garnish

Two open models do the real work.

Frost dates come from TabPFN, an open tabular foundation model from Prior Labs. I trained it offline on 6,784 NOAA weather stations to predict the last-spring and first-fall 32F frost dates from latitude, longitude and elevation. On 1,357 stations held out of training it is accurate to 6.9 days for the last spring frost and 9.7 days for the first fall frost, which beats a nearest-station baseline (8.7 and 10.8 days) on both.

The write-up comes from Gemma, Google's open-weight model. The live site writes it with Gemma 3 served openly through OpenRouter, with the built-in narrator as the instant fallback so the page never blanks. The result is cached per place and day, so a reload is immediate.

Why open matters here, concretely:

It runs cheap. Gemma 3 costs a fraction of a cent per plan, so a free gardening tool stays free.
Your location stays yours. Your coordinates go to the Frostline server only, where they are matched against the baked grid. They are never forwarded to a model vendor and nothing is stored. When Gemma writes the plan it sees a coarse label (your city name or just "your location") and the frost dates, never your exact position.
You can swap the model. Retrain the frost model on your own stations or point the advice at any open model with one environment variable.
It can run offline. Open weights mean the whole pipeline can run on a laptop with no internet, which is the point of a tool you use outside.

How it works

TabPFN needs PyTorch, so it never runs on a request. I bake it once, offline, into a 0.25 degree US grid of 16,353 points and ship that as a small static file. At request time a tiny lookup interpolates your location against the grid, a deterministic planner turns your frost dates and today into the sow-this-week list, then Gemma writes it up. No model sits in the hot path for the frost dates, so the plan is instant and the site cannot blank.

What is real and what is simulated

I will not dress this up:

Frost dates are real TabPFN predictions from NOAA 1991-2020 normals. The error above is measured on held-out stations.
The plan is a real deterministic function of your frost dates and today.
The advice prose is written live by Gemma 3, with the built-in planner as the instant fallback. The app labels which one wrote it.
Frost-free places like Miami and Key West are labelled from the measured NOAA freeze probability of the nearest stations, not from a guessed date, so the app never pretends a frost-free spot has a hard frost. The Reports page shows all of it.

The two partner categories

Best Use of TabPFN: frost-date prediction is a clean small-tabular regression, exactly what TabPFN is for. It beats the obvious baseline. The accuracy is published in the app and reproducible from bake/train_tabpfn.py.
Best Use of Gemma: Gemma writes every plan on the live site, grounded on a strict facts-only prompt so it never invents a date or a crop, with a deterministic fallback so the product is never at the mercy of the model.

Built to be attacked

One command gates the whole thing (./verify.sh): strict types, zero-warning lint, tests, a production build and a clean dependency audit. The site ships a strict nonce Content-Security-Policy, security headers, validated inputs and a route error boundary. The threat model is in the repo.

Try it

Open https://frostline-lime.vercel.app, type your city and see what you could put in the ground this week. Then go outside.

AI assistance (Claude) was used to build Frostline. The design, the model choices and the licence decision are mine. Every number here is measured against the code and the live site so you can check it yourself.

I built my brother a bedtime storyteller that runs on his laptop

Asuran — Sun, 04 Oct 2026 10:55:19 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

My brother and his wife are lovely parents and terrible storytellers. They'd both tell you that. Their kid is four, has just started school and wants a new story every single night. By the third night of "once upon a time there was a... dog?" everyone in that bedroom knows it isn't going well.

So I built them Nightjar.

What I Built

Nightjar is a bedtime storyteller that runs on their laptop.

The kid picks. Eight picture tiles a four year old can tap without reading: Pim the turtle, the moon, snow, a friendly dragon, a sleepy train, teddy, the sea and Big school, which I added for the first weeks of school. Or they can say anything they like. Short story or a long one.
An open model writes it. Gemma 3 1B through Ollama, on the laptop. No account, no cloud.
A guardrail reads every word first. Not a prompt asking nicely. A rule that runs on the output and can't be argued with.
It's read aloud, word by word. An offline voice (Kokoro-82M) reads it with each word lit as it's spoken, so the kid can follow along. A soft lullaby plays underneath, a new one for every story.
Then it stops. "The end. Goodnight." Nothing auto-plays the next one. This is bedtime, not YouTube.
And it learns. A grown-up taps once when the kid is asleep. TabPFN reads those nights and picks tomorrow's length, pace and how calm the story should be.

That last part is the one I couldn't find anywhere else. I checked before building: there are several bedtime story generators on GitHub, a couple even clone a parent's voice. None of them ask whether the story worked.

Demo

Live site: https://nightjar-bedtime.vercel.app

Try this	What happens
Home, tap Voice off to turn it on	A recorded chapter read aloud over a lullaby, each word lit. Tap any word to jump there
Ask for a story, tap Big school	A new story written inside your browser, screened, then read with your device's voice
The tuner, change any "asleep in" number	All 72 story settings rescored, tomorrow's pick moves
Break the guardrail, load "zero width bypass"	A hidden spelling of "monster" refused, with "Python agrees"
The book	The week's chapters as a page-turning book, plus a printable A5 PDF

A fair warning about the in-browser version: it downloads Gemma 3 270M (550 MB without WebGPU) and on a laptop with no GPU a story takes several minutes. It's there so you can try it. The real thing is the laptop box: nightjar serve puts the same page on 127.0.0.1, where a short story is written and voiced in about three minutes and a long one in about four and a half. I timed those in Chrome against the real models, not guessed.

Code

zkasuran / nightjar

Bedtime stories that learn what puts her to sleep. Local Gemma, a hard guardrail, TabPFN.

Live site · DEV post · Try it · How it works · Real vs simulated · Security · Run it

Nightjar is a bedtime storyteller I built for my brother and his wife. Their kid is four and just started school. They would both tell you they are bad at bedtime stories.

So the kid taps a picture or says what tonight's story should be about. An open model on the laptop in their house writes it, short or long. A guardrail outside the model reads every word before anyone hears it. Then an offline voice reads it aloud over a soft lullaby made for that story, with each word lit as it is spoken. One chapter, then "The end. Goodnight."

The part other story generators skip: a grown-up taps once when the kid is asleep. A tabular model picks tomorrow's length, pace and calm level from those minutes…

View on GitHub

How I Built It

tap a picture → tuner picks tonight's settings → series bible adds last night
  → Gemma 3 1B writes (Ollama, local) → guardrail screens, rewrites up to 3×
  → Kokoro narrates with a word clock → lullaby under it → "The end"
  → grown-up taps "asleep in 10" → tomorrow's settings

The guardrail. It normalises text first (NFKC, zero width characters stripped, curly quotes folded) so monster and ｍｏｎｓｔｅｒ are both "monster". Then it checks banned words, the kid's own fear list, the model talking about itself, leaked JSON and length. The title too, because the title gets read aloud.

That title rule exists because of a real bug. My first test asked for "a thunderstorm and a monster in the basement". Gemma handled it nicely and wrote about gentle rain. Then it called the chapter "Thunder and the Moon's Light". Thunder was on the fear list. I'd only been checking the body.

Scary requests aren't refused, by the way. A four year old asking for a dragon that breathes fire deserves a dragon. The request is screened and the prompt asks for a friendly version. The output guardrail still has the final say.

The tuner. One row per night means the data will never be big. Ten rows is a lot for a real family. That's hopeless for anything you train with gradients and it's exactly what TabPFN was built for: in-context learning on tiny tables, no training step. Leave one out on the sample log:

Model	Error predicting a hidden night
Always guess the average	4.99 min
k-NN	2.32 min
TabPFN	0.62 min

I need to be honest about that table. Those nine rows are sample data I wrote with one clean pattern in them: shorter, slower and calmer stories mean faster sleep. A strong model finds a clean pattern. It proves the pipeline works. It doesn't prove Nightjar helps a real kid. Only real nights can. That's what the next few weeks are for.

The voice. My first version used the browser's speech engine. On Linux it was scratchy. The highlighted words fell behind and then restarted from the top every time you paused. So the chapters are narrated offline by Kokoro, which reports when each word starts. The player runs on one clock: the audio's own position when the voice is on, a silent timer when it's off. Pause, resume, tap a word, drag the bar, switch chapters: they all move that one number, so the words can't drift from the sound. A browser test checks the lit word against the audio position in every one of those cases.

On a phone, a story written in the browser has no recording, so it's read with the phone's own voice (it picks Siri or Google voices over the robotic ones), one sentence at a time.

The lullaby is composed live with Web Audio: a soft pad, a music box on a slow five-note scale and a low root note, through a little reverb. The story's title seeds the key, chords, tempo and melody, so every story gets its own tune. Nothing recorded, nothing to license.

Long stories. A 1B model won't write past about 150 words no matter how you ask. So a long story is three parts that continue each other. That took a few tries. The first long story ended, said goodnight, then started again in part two, because the system prompt told it every story ends asleep. Middle parts now get a prompt that forbids ending. They carry the opening so names don't change (the dragon "Sparkle" became "Flicker" two parts later). They're refused if they repeat an earlier sentence.

What I found reading the transcripts back

I recorded eight nights from the real command line and read them the next morning. That taught me more than any test I'd written.

The model ignores length. Asked for 220 words it wrote 77 to 156. "Too short" was the most common refusal.
Two accepted chapters ended with the prompt's own field name read aloud: "Open thread: Maybe tomorrow the moon will show Pim a rainbow." Both passed on the night. That's a rule now. The site still shows both, marked as slipped.
The kid asked for "a dragon that breathes fire" and got Pim wishing for "a tiny flame". "flames" is banned. "flame" isn't. A word list catches words, not ideas. The site says so on its Limits page.
Right after publishing, I asked for "horror" myself and it failed every draft. My prompt said "don't use the word horror" and the retry said "rejected for: horror". A small model repeats what it reads, so it wrote "horror" three times. Now the request is rewritten before the model sees it: horror becomes a cosy story where a shadow turns out to be a kind friend, a zombie becomes a sleepy giant. The scary word never reaches the prompt at all.
With the whole series history in the prompt, the model wrote about the turtle every time the kid asked for a dragon. When there's a request now, only last night's one line summary goes in.

Built to be poked at. One gate, ./verify.sh, runs 65 Python tests (fuzzing, bypass attempts, tampered files, the local server refusing other hosts) and 16 web tests that hold the TypeScript guardrail and tuner to the Python results. The site has a strict CSP and checks a sha256 on every data file and audio file before using it. nightjar serve only answers on 127.0.0.1 and refuses any other Host header.

Why Does Open Innovation Matter?

For this project it isn't a principle. It's four practical problems a closed API couldn't solve.

The kid's data stays home. A four year old's name, fears and the minute they fall asleep every night are about the worst thing to send to someone else's server. Here it's three small files on my brother's laptop.
It's free every night. Around 350 stories a year at zero cost per story. A metered API turns every bedtime into a small decision. Open weights make it a habit.
It works with the wifi down. At 8 pm with a tired kid, "the internet's out" is a broken promise.
I can build around it. A guardrail outside the model, a tuner choosing its settings, an offline voice with word timings, a version pinned so it never changes under them. You can't wrap a closed endpoint that tightly.

Handing it over

This is going onto my brother's laptop this week, with Big school as the first tile. The kid in the live demo is a placeholder called Mira and the sleep log is sample data, both labelled on every page. I'll update this post with what the three of them actually think. I suspect the kid will mostly want the dragon.

Prize Categories

Best Use of Gemma. Gemma 3 1B writes every story locally through Ollama, short or in three parts. Gemma 3 270M and 1B also run in the browser.
Best Use of TabPFN. The TabPFN v2 regressor chooses each night's story settings from minutes to sleep. Built with PriorLabs-TabPFN.

Built with AI coding assistance. I directed it, reviewed the code and ran everything above myself; the numbers come from real runs. Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms. Narration: Kokoro-82M (Apache-2.0).

Signed hackathon results anyone can recompute and verify

Asuran — Sun, 27 Sep 2026 06:09:17 +0000

Every hackathon platform converged on the same features then stopped at the judging. The biggest one cannot weight its criteria. The ones that advertise score normalization do not document how it works. None of them lets you recompute a result, so a ranking is something you trust because the platform says so.

DOGFOOD is a 72-hour event with one product in it, a submission and judging platform that Hackathon Raptors will fork and run. We built ours around a single idea. A result should be something anyone can recompute from the raw scores and verify, with no trust in the host.

Weighted criteria, then a normalization you can defend

Judges score each criterion from 1 to 5. The organizer sets a weight per criterion. We combine them into a weighted composite that stays on the 1 to 5 scale. Then we remove each judge's leniency with an additive model (mu plus a project effect plus a judge bias) and rank by the bias-removed quality after shrinking it toward the mean by n over n plus k. On the event fixture k came out at 0.81, so a project with two reviews keeps about 71 percent of its own signal and one with five keeps about 86 percent. Coverage buys confidence, not rank. A judge who scored everyone the same contributes a bias offset and zero ranking signal, which is correct. It never divides by that judge's spread, so a zero-variance judge is not a crash.

The result is signed and you can recompute it

When results publish, the portal emits a bundle, the raw scores, the rubric and its weights, the method and its parameters, the code commit and the ranking. It is hashed over a canonical encoding and signed with an Ed25519 key the deployment generates on first boot into its own volume. A standalone verify.py recomputes the ranking from the raw scores, checks the hash and verifies the signature. Change one judge's score and it prints NOT VERIFIED. That turns "we normalized the scores" into "here is the record, recompute it yourself".

Two engines, one winner

The strongest evidence that this is real: we built the judging core as an isolated pure-Python module and wired it into the Django app separately. Fed the same raw scores and the same 0.4 0.4 0.2 weights, the two implementations independently produced the same winner, prj_16, at the same final score. Same inputs, same result, two codebases.

Honest about what it does not know

The ranking reports an uncertainty band next to every score. On this sparse fixture 39 of 40 adjacent ranks overlap, so most of the order is a statistical tie and we say so rather than faking precision. The weighted-rubric method and a Bradley-Terry pairwise pass agree on 7 of the top 10 but disagree on the very top seat, which is exactly the signal a pairwise round is for. We publish both.

It runs on a laptop with the network off

docker compose up brings the whole thing up seeded, prints the per-role tokens, then the acceptance suite passes all seven checks. The gallery is public, a closed event refuses a late submission, a judge cannot read a peer's scores over the API, the organizer exports CSV. No cloud account, no hosted database, nothing to visit. It also exports a whole event to JSON and re-imports it into a fresh instance with every project, score and value matching, so an organizer can leave as easily as they arrive.

The repo is github.com/zkasuran/dogfood-portal under MIT. The demo walks one event from create to submit to judge to publish, then runs the verifier on the real bundle and watches it reject a tampered one: youtu.be/igjISVTL2ik

AI assistance (Claude) was used to build this. The design, the maths and the verification were checked by the author. The acceptance suite plus the judging tests pass green.

AI models catch bad code, then cry wolf on the good code

Asuran — Sun, 27 Sep 2026 05:20:55 +0000

This article is a submission for the Kaggle Benchmarking Challenge.

Every data-science tutorial makes a claim: "I split first so nothing leaks," "cross-validation proves it generalizes," "99% accuracy." The code below the claim does not always agree. So I asked one question: when an AI model reads a real Kaggle notebook, does it trust the caption or read the code?

Blog vs Bytecode is a benchmark built entirely from real, public Kaggle notebooks. I pulled 13 popular kernels, cropped 34 short snippets and paired each with the claim its author (or a plausible write-up) makes about it. Half are honest and sound. Half hide a real methodology flaw. The model returns one verdict, OK or PROBLEM, plus its reason. The first thing the benchmark caught was not a model. It was the harness.

The benchmark

34 items. The design is deliberate:

Real code, not toy snippets. 28 items are real notebook code (verbatim or trimmed); 9 take real code and inject exactly one controlled flaw. Every item names the kernel it came from.
Balanced 17 PROBLEM / 17 OK, so a model that reflexively yells "PROBLEM" scores near 50% and gets caught.
Six flaw families: data leakage, train/test contamination, misused cross-validation, metric mismatch, target-leakage/snooping and claim-vs-code divergence.
Matched adversarial pairs that separate understanding from pattern-matching. A RobustScaler inside a pipeline is fine; the same scaler fit once before cross-validation leaks. KFold(shuffle=True) is fine on independent house-price rows but breaks a time series. np.log1p before the split is fine because it fits no parameters.
Genuine flaws from famous notebooks. A top fraud-detection notebook resamples before cross-validating and reports "accuracy" on data that is 0.17% fraud and even labels its own cell "Overfitting Case." A top house-prices notebook fills missing values with a median computed over train and test combined. These are real, not staged.

Three cuts, not one number: overall accuracy, PROBLEM-recall (does it catch the flaw) versus OK-specificity (does it leave sound code alone), plus accuracy on the matched pairs.

Surprise 0: the harness was lying

Kaggle runs the models for you. For several, the run finished but no answer came back: gpt-oss-120b returned nothing on 33 of 34 items, DeepSeek-R1 on 23, Qwen3-Next on 33. Grading those blanks as wrong would rank strong models near zero. A benchmark that does not audit its own capture measures the plumbing, not the model, so the scorer drops uncaptured answers rather than counting them wrong. Everything below is only models whose answers were captured.

Surprise 1: every model catches the flaw, then cries wolf on the clean code

On real notebooks the split is stark. Every captured model scores 100% on PROBLEM-recall. They catch the leak, the resample-before-CV, the accuracy quoted on 0.17% fraud. Where they fall apart is the opposite error: flagging sound code as broken.

Look at the bottom of that chart. Grok with reasoning off cleared zero of the clean notebooks: it called every sound snippet a PROBLEM. Claude Haiku cleared 27%. The expensive mistake on real data is not gullibility. It is the false alarm.

Read the models' own words

The sharpest case is np.log1p applied to the target before the split. It is the textbook exception to "fit transforms inside the fold," because log1p fits no parameters: each row's value depends only on itself. Nothing leaks. Four models flagged it anyway.

Claude Haiku 4.5 argued itself into the false alarm:

Although np.log1p is indeed a fixed elementwise transform with no parameters ... the broader methodological issue is that the transformation is applied before any cross-validation split.

It states the correct fact, then overrides it. Claude Opus 5, on the same snippet, held the line:

log1p is a deterministic, parameter-free transform: each row's transformed value depends only on that row, so no statistic is shared across the fold boundary. This is the standard exception to the 'fit transforms inside the fold' rule.

That gap, knowing the rule versus knowing its exception, is most of the leaderboard.

When the flaw is real, the models are sharp. On the house-prices notebook that fills LotFrontage with a neighborhood median computed over train and test combined, every model caught it. Gemini 3.8 Flash:

The code computes the median LotFrontage per neighborhood on 'all_data' (which concatenates both train and test rows), thus leaking test set information into the imputation rather than computing medians solely from the training rows as claimed.

Surprise 2: reasoning is worth 40 points, all of it specificity

Grok 4.20 scores 94% with reasoning on and 52% with it off. The 42-point gap is entirely on the clean code: OK-specificity falls from 88% to 0%. With reasoning off the model stops reading what a line computes and pattern-matches "this is a notebook, notebooks have bugs, PROBLEM." It flagged the parameter-free log1p too.

Surprise 3: the small model trusts the prose

Gemma 4 31B is the only model that misses the flaws: 35% recall against everyone else's 100%. It reads the confident claim, agrees, then walks past the leak. So the bias flips with scale. Big models over-flag sound code while the small model under-flags real bugs. Same benchmark, opposite failure.

The leaderboard

Scored on the items the proxy captured; models it dropped on most items are left off.

Model	Accuracy	Catches flaws	Clears clean code
Claude Opus 5	94%	100%	88%
Grok 4.20 (reasoning)	94%	100%	88%
GLM-5	93%	100%	85%
Claude Sonnet 5	93%	100%	83%
Gemini 3.1 Pro	93%	100%	82%
Gemini 3.7 Flash	91%	100%	82%
Gemini 3.8 Flash	88%	100%	76%
Gemini 3.5 Flash-Lite	88%	100%	75%
Claude Haiku 4.5	62%	100%	27%
Grok 4.20 (no reasoning)	52%	100%	0%
Gemma 4 31B	47%	35%	59%

Honest limits

Two backends. Kaggle's Model Proxy captured most models; a few (DeepSeek-R1, gpt-oss, some Qwen) it dropped, so those are omitted rather than scored on a handful of items. The numbers above are on captured answers only, so a couple of rows are over a subset.
Injected versus real. 9 of the 17 PROBLEM items are one controlled flaw injected into real code; the rest, plus every OK item, are real notebook code unchanged. The data marks which is which.
Verdict grading. Grades are on OK versus PROBLEM. The written reason is captured too. On the top models it names the right flaw almost every time.

What I would measure next

Localization: not just OK or PROBLEM but which line, scored against the known offending line.
A confidence axis, to see whether the over-flaggers are also the least calibrated.
More time-series and NLP notebooks, where the leakage is subtler than a misplaced scaler.

Reproduce it

The benchmark is public on Kaggle: kaggle.com/benchmarks/tasks/zkasuran/blog-vs-bytecode. Every item, verdict and grader is open. Each model's answer is captured verbatim, so you can read exactly where it went wrong.

Sources and disclosure

Every snippet is derived from a real public Kaggle notebook. Sources include the credit-card-fraud notebook by janiobachmann, the stacked-regressions and house-prices notebooks by serigne and pmarcelino, the Titanic notebooks by startupsci and arthurtok, the IMDB and disaster-tweet NLP notebooks and several Kaggle Learn exercises. Injected flaws are marked in the data and each item names its source kernel.

AI (Claude) helped build the benchmark and draft this post. The design, the items, the grading and the analysis were verified by the author, including reading every snippet and every raw model output.

I vibe-coded a Next.js knowledge base that argues with itself

Asuran — Thu, 24 Sep 2026 01:17:02 +0000

This is a submission for the Sanity Challenge, Path Two: Vibe-Code Something Strange

What I Built

Still True is a knowledge base about Next.js that keeps itself honest. Instead of storing facts as flat pages it stores them as claims tied to sources, with typed relationships between claims. One claim can supersede another or contradict it. The relationship carries the reason it holds. The public board reads that graph and lays it out by topic. Current claims sit on top. Superseded claims are dimmed and struck through. Wherever two sources disagree the conflict shows side by side with an explanation.

The strange part is that the content is built to disagree with itself on purpose and the app treats that as the signal, not as mess to clean up.

Demo

Live board: https://still-true.vercel.app

The same graph grounds an agent at /ask, which is my Path One entry. On the board you can watch the truth change. The fetch-caching topic shows the current "not cached by default" claim above the superseded "cached by default" one. A time slider replays each topic flipping as newer facts supersede old ones. There is an interactive contradiction graph where every claim is a node and amber links are live disagreements. Every claim has its own page with its source, its confidence and the claims it supports or contradicts.

Code

Repo: https://github.com/zkasuran/still-true

Front end is Next.js 16 on the App Router. The Studio in studio/ holds the schema and the workflow. The board reads the dataset with GROQ, following the reference on a claimEdge to both claims it connects. There are 24 unit tests and 4 end-to-end tests running in CI.

My Build Process

I built this with Claude Code as the AI-native IDE. The interesting parts were where the model or the tools pushed back.

The schema was the first real decision. My instinct was to hang a "contradicts" list of references on each claim. That was wrong, because a bare reference has nowhere to store why two claims conflict. The prompt that turned it around was roughly "model the relationship as its own document, not a field", which gave me the claimEdge type: from, to, a relation of supports / contradicts / supersedes and a reason. Once the reason lived on the edge the whole app got simpler, because the board just renders edges.

Sanity Context surprised me. I seeded the Knowledge Base with pairs of documents that disagreed and expected it to flag every pair. It did not. When I labelled the documents by version, Context reconciled them into one timeline-aware entry instead of raising a conflict. That is the behaviour you want most of the time, but it meant my "look, a contradiction" demo produced zero conflicts. The fix was to add one pair that genuinely could not be reconciled, an official default beside a community one that disagree on the same fact with no version to separate them. That rebuild raised exactly one conflict, the one the board and the agent now show.

The Studio schema tooling fought me. sanity schema deploy and sanity schema validate both failed with exports is not defined. They failed on a clean template too, so it was a toolchain issue in this Studio version rather than my schema. sanity build and sanity dev worked fine and the board queries the dataset directly rather than through a deployed schema, so I routed around it and wrote it down instead of fighting it.

Deploying to Vercel hid two gotchas. The first deploy returned 404 on every route because Vercel had not detected the framework as Next.js, so it served the wrong output. The second was a login wall: the project shipped with deployment protection on, which would have shown judges an SSO screen instead of the app. Both were one-line fixes through the Vercel project API, framework: nextjs then ssoProtection: null, followed by a redeploy. I only caught the login wall because I tested the URL logged out, which is the test that actually matters for a submission.

Reaching past the Studio: Workflows and an App SDK app

I modelled the editorial process as data with sanity-plugin-workflow. A claim moves through unverified, sourced and confirmed. The move into confirmed is gated to an administrator role. State lives in a separate metadata document, so the content schema stays clean while the Studio gets a drag-and-drop board, promote and demote actions and colored status badges. A person or an agent can add a claim as unverified and it only becomes confirmed once a human with the right role signs off, the same propose-then-approve shape the challenge suggests.

I also built a real App SDK app in app-sdk/, a contradiction-triage dashboard on @sanity/sdk-react that runs inside the Sanity Dashboard. It reads the open contradictions live and lets a reviewer walk each one, so the "reach past the Studio" bonus is a working custom surface rather than a read-only frontend. The code is in the repo and deploying it to the Dashboard is one command.

Sanity Project Details

Project ID: mx12urdz
Public dataset: production

The dataset holds the claims, sources and typed edges. You can query it directly. Every contradiction with both statements dereferenced:

*[_type == "claimEdge" && relation == "contradicts"]{
  reason, "from": from->statement, "to": to->statement
}

Agent Session

Built end to end with Claude Code: the Sanity schema, the Knowledge Base pipeline and the grounded agent were all generated and verified through it.

AI assistance (Claude Code, Anthropic) was used to build this. The design and the decisions were mine and the build was verified live at the URL above. It runs on MiniMax-M3.

I built an agent that refuses to answer Next.js from stale docs

Asuran — Thu, 24 Sep 2026 01:17:00 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

Next.js changes fast and the old answer keeps ranking. fetch stopped caching by default, params became a Promise you await, next/font replaced @next/font, Turbopack became the default bundler. Ask a plain model "does the App Router cache fetch by default" and it will confidently say yes, because that was true in Next.js 14 and most of its training data still says so.

So I built Still True, an agent that answers Next.js questions only from a Sanity Context Knowledge Base and refuses to give you a stale or contradicted answer. When two sources disagree on the same fact it shows both with their sources and says which one is current. When the base does not cover something it says so instead of guessing.

The part a keyword search cannot do is the whole point. Keyword retrieval returns whichever document mentions "fetch caching" first. This agent returns the claim the structured content marks as current and it surfaces the disagreement instead of burying it.

Demo

Live agent: https://still-true.vercel.app/ask

The home page runs a live head to head: the same MiniMax model with no knowledge base on the left, the grounded agent on the right. Try these.

"Does the App Router cache fetch by default?" The plain model says yes, cached by default, which is the Next.js 14 answer. The grounded agent says no. Not cached by default. Opt in with cache: 'force-cache'. The board shows the two sources that disagree with the reason sitting on the edge between them.
"In Next.js 16 are params and searchParams a Promise I must await?" It answers yes, shows the await code and notes that Next.js 14 was synchronous and is now deprecated.
"Are all my .env.local variables available in the browser?" It answers only the NEXT_PUBLIC_ ones and flags the source that claims otherwise.

Every answer lists the Knowledge Base entries it read, so you can see it is grounded and not improvising.

Code

Repo: https://github.com/zkasuran/still-true

The agent is a small tool-use loop that hands retrieval to the model over three tools on the Sanity Context client.

list_knowledge returns the Knowledge Base outline, every entry path with its title and scope.
read_entries pulls the full cited body for the paths the model picks.
list_conflicts returns the open conflicts Context raised.

Retrieval is structure driven, not a blind top-k. The model reads the outline first, picks the entries that match, then widens only if the evidence is thin. Every answer returns its toolTrace. For the fetch question it is list_knowledge, read_entries(1), list_conflicts, so you can see it read one entry rather than dumping the base. When the outline has nothing on topic the agent abstains with a plain "Not covered" instead of hallucinating. It ships two ways in the repo, a standalone CLI in agent/ and a server route at /api/ask behind the chat UI. It talks to any OpenAI-compatible model through one env var. The public routes are rate limited and same-origin guarded because they cost money to run.

How I Used Sanity

This is built on Sanity Context and a Knowledge Base. The contradiction handling is Context's own feature, not something I bolted on.

I created the Knowledge Base and imported real Next.js source material, deliberately including pairs that disagree: current guidance beside the older guidance it replaced, plus an official recommendation beside a contradicting community one. Sanity Context distilled those sources into a small set of cited entries and did two things I leaned on.

Reconciliation. Where sources described the same fact across versions, Context merged them into one timeline-aware entry, so the agent can say the current way is X while the old way was Y.
Conflict raising. Where two sources asserted the same fact incompatibly with no version to reconcile them, Context raised a conflict into its Issues queue with both claims and their sources. That is the fetch-caching case the demo shows.

The whole pipeline runs headless from the CLI: create the base, import sources, build. The agent reads it through the Sanity Context client (entries.list, entries.get, issues.list), the same content Context serves to an MCP endpoint. It also runs against a live Context MCP URL through one env var. Because the base is small the agent can read every relevant entry, so it never misses the one that matters.

Sanity Project Details

Project ID: mx12urdz
Public dataset: production
Knowledge Base: kbOSaaWFy5yI ("Developer Facts That Drift")

The same project backs the Path Two board, which renders the underlying claim graph. The claims, sources and typed edges live in the public dataset and are queryable over GROQ.

Agent Session

Built end to end with Claude Code: the Sanity schema, the Knowledge Base pipeline and the grounded agent were all generated and verified through it.

AI assistance (Claude Code, Anthropic) was used to build this. The design, the verification and the decisions were mine. The agent runs on MiniMax-M3 and its answers are grounded in the Sanity Knowledge Base, verified live at the URL above.

A charity does not close because its overhead was too high. I checked 328,186 tax returns.

Asuran — Sun, 06 Sep 2026 00:04:16 +0000

This is a submission for Weekend Challenge: Generosity Edition

I have given money to charities for years and I have never once known whether the one I gave to was about to close. The number everybody puts in front of you is overhead, the share of spending that does not go to programmes. Every rating site leads with it. I assumed it meant something.

So this weekend I downloaded the IRS Statistics of Income extracts of Form 990 and Form 990-EZ, which is 3,697,515 filings covering 700,873 organisations across fiscal 2017 to 2024, then asked a narrower question. Not whether a charity is efficient. Whether it is going to still be there.

I took the 328,186 organisations that filed a long-form Form 990 for fiscal 2018 or 2019, then checked which of them appear on any filing for fiscal 2022 or later. 8.88% do not. Then I scored two candidate signals inside each of ten spending deciles, so that nothing I found could turn out to be a story about big charities outliving small ones.

Feeding America spends $4.93 billion a year and holds 1.04 months of it in cash. The Greater Chicago Food Depository spends $261 million and holds half a month. Second Harvest Food Bank of Central Florida holds nine days. None of that is on any rating site, because the number nobody computes for you is the one that decides whether the lights stay on.

What I Built

Keepalive computes months of runway for 579,178 US nonprofits, straight off their own filings. Runway is cash plus savings divided by one month of spending. It is the number a finance director lives by and the number a donor never sees.

Across 1,573,687 long-form filings the median is 5.5 months. 33.5% hold under three months. 15.1% hold under one.

The site does four things with that.

Where does my gift buy the most time. Say what you can give, pick a cause and a state. It ranks real organisations by days of runway your money adds, which is 365 * yearly gift / annual spending. $25 a month is 12 hours of runway at a $260 million food bank and 9.4 days at a $23,000 one. The same money, three orders of magnitude apart in effect. That single line changed how I give.
Look up any of the 579,178 by name or EIN, with seven years of revenue, spending and cash on one chart.
Read the brief. Gemini writes it from a fixed packet of that filer's own figures and is handed nothing else, so there is nothing for it to invent. It has to return a line naming what a tax filing cannot tell you.
Check it. One click publishes the SHA-256 of the figures on screen and the brief just written as a memo on Solana devnet, so what Keepalive said about an organisation on a given date lives somewhere its author cannot quietly edit.

The finding

Both tables below hold inside every one of ten spending deciles.

Runway predicts survival. Thin runway is the worst of the three bands in all ten deciles. Pooled with deciles weighted equally: 86.79% of the under-three-months group was still filing, against 92.61% at three to twelve months and 93.56% above a year. The gap is 10.3 points in the smallest decile and 1.6 in the largest, so it bites hardest exactly where the money is smallest.

Fundraising thrift points the wrong way. The organisations that spent under five cents to raise a dollar survived less than the five-to-fifteen-cent band, in all ten deciles: 91.16% against 94.86%. Spending more on fundraising went with surviving more. In nine of the ten deciles the thriftiest band is the worst of the three outright. Only one of these two numbers is on the rating sites. It is the one that is backwards.

That is the starvation cycle, visible in tax returns. Judge a charity on how little it spends on itself and you reward the ones with no capacity to raise next year's money.

Demo

Live: https://keepalive.zkasuran.dev No sign-in, nothing to install.

Try it in 30 seconds:

Open the site. The two survival tables under The finding are the whole result, drawn from the same query.
In the search box type greater chicago food depository. Open it. The runway card reads 0.5 months on a $261m budget. The peer card shows 95.6% of organisations in its own spending decile and runway band still filing.
Scroll to The honest brief and wait about eight seconds. Gemini writes it live. Read the last line, the one that starts "What these filings cannot tell you".
Press Publish a checkable receipt. About four seconds later you get a Solana devnet signature. Open it. The memo on chain holds the SHA-256 of exactly what you just read.
Back at the top, set the amount to 25 per month, cause Food, agriculture and nutrition, state TN. The list re-ranks by days of runway your $300 a year adds.

Code

zkasuran / keepalive

Months of runway for 579,178 US nonprofits, read off their own IRS filings

Keepalive

A charity does not close because its overhead was too high. It closes because it ran out of months.

Keepalive reads 3,697,515 IRS filings and computes, for 579,178 US nonprofits, the one number no rating site puts in front of you: how many months of spending they hold in cash Then it tells you what your gift does to that number.

Live: https://keepalive.zkasuran.dev

Built for the DEV Weekend Challenge: Generosity Edition 4 to 7 September 2026. Everything in this repository was written inside that window.

The finding

Two candidate signals, measured on the same 328,186 organisations that filed a long-form Form 990 for fiscal 2018 or 2019, scored on whether the same EIN appears on any filing for fiscal 2022 or later. 8.88% do not.

Months of runway predicts survival. Under three months is the worst of the three bands in all ten spending deciles, with no exception.

…

View on GitHub

npm run data reproduces every number: it pulls the IRS extracts, loads 3.7 million filings, prints the finding and writes the static shards the site reads. It needs no credentials and no account.

How I Built It

There is no runtime database. The site is 4,542 static files on a CDN plus four small functions that hold keys a browser cannot have.

Search is a static inverted index. 579,178 names tokenise to 1,910,797 postings, hashed into 2,048 shard files. The browser hashes your query tokens, fetches only the shards it needs and intersects them. Typing "greater chicago food depository" costs three fetches and no server. Each posting list is ordered by spending, which matters more than it sounds: truncate a common token like "food" and the intersection silently stops finding real names. That was my first real bug.

Organisation records are sharded by EIN modulo 1024, so one lookup is one 250 KB fetch that the CDN then holds.

Google AI writes the brief under a JSON schema, from a packet containing every number it is permitted to mention and no others. Absent fields are dropped rather than sent as zero, because "not reported" and "$0" are different claims and a model handed a zero will tell you the charity has no employees. The system prompt forbids computing a new ratio. One measured detail: Gemini 3.5 Flash spends its thinking tokens out of the same maxOutputTokens budget, about 2,200 of them here, so a tight cap truncates the JSON and the parse fails. That looks exactly like a bad model and is not.

Solana is the only place the receipts live. Keepalive keeps no copy. Remove Solana and the app can no longer prove that today's number is the number it computed from the filing, which is the honest answer to "how do I know your AI did not make this up".

The transaction is built against the wire format rather than with a client library. A memo transaction is one signature, two account keys and one instruction. Ed25519 is already in crypto.subtle, so an SDK would have added megabytes to buy nothing. test/attest.wire.test.mjs proves those bytes equal the ones @solana/web3.js produces: message, signature and full serialized transaction, byte for byte, including a memo that crosses the compact-u16 boundary.

One thing I want to state plainly rather than hide. The signature is made on the server, where the key is. The transaction is submitted by your browser. That is not a design flourish. The public devnet RPC answers Cloudflare's egress with 403 Your IP or provider is blocked from this endpoint while a visitor's own connection is fine. That RPC allows any origin. So the key stays server side and the network call happens where the network works. The only thing a client controls is the blockhash, which is length checked and which cannot turn a memo into a transfer.

Snowflake is the warehouse path, in data/snowflake.sql: the same cohort plus one statement that cuts it into deciles then scores both signals inside every decile at once. Cortex does the language work next to the data, with AI_CLASSIFY over names the IRS only gave a letter code and AI_AGG writing one sentence per filer from all of its returns together. There is a licensing line I held to: Marketplace terms do not permit redistributing a dataset, so every per-organisation figure this site serves comes from my own public domain IRS download instead. Only aggregates ever leave a warehouse. Sources and the exact granting sentence for each are in DATA-SOURCES.md.

ElevenLabs reads it out, because the reason a lot of people cannot use a giving tool is that it is a wall of financial text. The free tier is about ten minutes of audio a month, so architecture had to do the work rather than budget: everything that never changes is pre-rendered once at build time by tools/narrate.mjs into static mp3. A visitor who plays the tour or the finding spends nothing. Only a brief about one specific filer is synthesised live, cached in the Worker by a SHA-256 of the text, so the second person to read the same organisation is free too. You can watch it happen: the response carries x-keepalive-audio: fresh the first time and cached after.

The tour is in English, Hindi, Tamil and Spanish. Gemini does the translation with an instruction to keep every figure exactly as stated. ElevenLabs speaks it. The English stays the source of truth. That is the one part of this build I would call generous rather than clever. A US tax filing is hard to read in your first language. The people most likely to be deciding about a small charity back home are not all reading English.

Two things that will bite anyone doing this on a free key. A scoped key answers 401 missing_permissions on GET /v2/voices, so a voice id has to be configured rather than discovered, with a short list of defaults to fall back through. Free accounts also get 402 paid_plan_required on library voices, so a voice id copied out of a blog post is a dead end. The four in the list are ones I verified synthesise on this key.

What is real and what is not

Every figure on an organisation page is that organisation's own reported number, labelled with the fiscal year, computed in your browser from a static file. Nothing is modelled and nothing is imputed.
Form 990-EZ does not break out cash, so EZ filers show no runway rather than a guess.
The survival tables are associations, not causation, controlled for size by decile and for nothing else.
Solana devnet, not mainnet. No money moves anywhere in this project. Keepalive is not a payment processor and never touches a donation.
Keepalive is not a charity rating. It can tell you what an organisation's finances look like. It cannot tell you whether they are good at the work. Neither can any rating site.
AI assistance (Claude) was used in building this. The design, the analysis and the verification are mine. Every number above came out of SQL I can rerun in front of you.

Prize Categories

Best Use of Google AI. Gemini writes every brief under a JSON schema from a packet of that filer's own figures, with a rule that a missing field means not reported. It also writes the narration translations without touching a single figure.
Best Use of Solana. A devnet memo is the only store of record for what this site published. Remove it and no claim on the page is checkable. The transaction is hand built and proved byte-identical to the reference implementation.
Best Use of ElevenLabs. The tour and the finding are pre-rendered at build time in four languages, so listening costs no credits and needs no key. A brief about one organisation is spoken live, then cached by a hash of the text.

The Snowflake path is written and in the repository as data/snowflake.sql, and the live deploy tells you plainly that its credential is not bound: /api/status returns snowflake: false. The finding tables you are reading were produced by the same SQL run locally in DuckDB rather than in a warehouse. I would rather say that than claim a category a judge cannot see working. If it changes before the deadline I will say so here in an edit, with the timestamp.

Thanks for a theme worth building for. The most useful thing I learned is that the kindest gift to a small charity is not the largest one, it is the predictable one. The number to ask about is months.

My checkout said it was ready for a month. Nobody could have paid.

Asuran — Fri, 04 Sep 2026 22:23:29 +0000

The dashboard for my payment provider reads, for the whole life of the account:

Revenue since inception   USD 0.00
Transaction History       Total count: 0
Users                     Total count: 0

Live mode and sandbox mode, same numbers. The easy reading is that nobody wanted it.

Here is the harder reading. Over the eight days my analytics still retains, 21 of 228 real visits reached the pricing page. Not one of those 21 had anything to click. The buy button was there. It was styled, it was enabled, it said "Sign in to start the trial". It had no click handler on it at all.

Three separate defects, none of them visible from reading the code, all three provable from the network tab.

One: my own content security policy blocked the SDK

The policy I shipped was tight on purpose. The whole frontend is first party, so I named exactly what it needed:

default-src 'self';
script-src 'self' 'unsafe-inline' https://esm.sh;
style-src  'self' 'unsafe-inline';
img-src    'self' data:;
connect-src 'self' https://esm.sh https://api.tiun.live https://api-sandbox.tiun.live;

Read that and it looks right. The SDK module comes from esm.sh, so esm.sh is a script source. The SDK talks to the API host, so the API host is a connect source. Done.

An SDK is not one request. Here is what init() actually does, from the shipped bundle:

let r = document.createElement("link");
r.rel = "stylesheet"; r.href = cssUrl;
document.head.appendChild(r);
let a = document.createElement("script");
a.src = scriptUrl; a.async = true;
document.head.appendChild(a);

It appends a stylesheet and a script pointing at the API host. Those are style-src and script-src decisions. The sign in surface renders in a frame, which is frame-src. With no frame-src the browser falls back to default-src 'self'. The one place I named the host was the one place it did not matter.

Driving the page over the DevTools protocol says it plainly:

requestWillBeSent   .../snippets/<id>/background_css
requestWillBeSent   .../snippets/<id>/background_js
Network.loadingFailed
Network.loadingFailed

Both fetch fine with curl. 84 KB of CSS, 506 KB of JS, 200 each. The refusal was mine.

The reason nobody noticed

waitForReady() resolves when the snippet posts back that it initialised. With the runtime bundle refused, that message never arrives, so the promise never settles. My code awaits it at module top level:

await tiun.waitForReady();
// everything below here never ran
for (const button of document.querySelectorAll("[data-buy]")) {
  button.addEventListener("click", () => tiun.checkout({ productId: product.id }));
}

The page did not throw. It did not show an error. It stopped, silently, above the line that wires the buttons. It kept the markup the HTML shipped with. That markup shows the sign in prompt and hides the buy controls, because that is the correct signed out state. A correct looking page with no behaviour behind it.

I had written a checkoutUnavailable() path for exactly this and it never fired, because nothing rejected.

Two: getUser is synchronous

Fixing the policy moved the failure one layer up. From the SDK source:

getUser() { return { isAuthenticated: this._isAuthenticated, user: this._user } }

No promise. My code:

const user = await tiun.getUser().catch(() => null);

.catch on a plain object is a TypeError. It is thrown inside an async function, so it does not crash the page, it becomes a rejected promise nobody is awaiting. That function is the one that decides what the top bar and the buy controls look like. It died on its first line, on every call, for a month. The only trace was a single Uncaught (in promise) line in a console nobody had open.

Three: the purchases are one level down

Same call, look at the shape again. It is { isAuthenticated, user }. The email and the entitlements live on the inner object. I read them off the outer one:

const access = Object.keys(user?.productAccess ?? {}).length > 0;

Always {}. Always false. So the control that hands a paying customer their CLI licence could never appear, for anyone, ever. If someone had pushed through the other two bugs and paid me money, they would have got nothing.

And a fourth, for free

for (const event of ["login", "logout", "checkout:complete", "user:updated"]) {
  tiun.on?.(event, paint);
}

The SDK emits seven event names. checkout:complete is not one of them. user:updated is not one of them. The real names are ready, login, logout, userChange, paywallShow, paywallHide and error. Subscribing to a name that is never emitted is not an error anywhere. It is a listener that waits forever. A completed purchase repainted nothing.

on() is called with ?. too, so even if the method vanished the line would stay quiet.

What I actually got wrong

Not the policy. Policies are fiddly and that mistake is cheap to make.

What I got wrong is that I treated waitForReady() resolving as proof the checkout worked. It is not proof of anything. It is one signal from one layer. Every layer under it can be broken while it still resolves. I built my whole readiness story on a check that could not fail.

I write a tool that finds exactly this in test suites. Tests that call a function and assert nothing. Assertions that cannot be false. Gates no workflow invokes. Every one of them is a signal that stays green because it is not actually connected to the thing it claims to measure. I had one of those in my own shop, on the page where people pay me.

The fix

The policy now names the hosts everywhere the SDK reaches through. Nowhere else:

script-src 'self' 'unsafe-inline' https://esm.sh <api hosts>;
style-src  'self' 'unsafe-inline' <api hosts>;
img-src    'self' data: <api hosts> https://assets.tiun.dev;
font-src   'self' data: <api hosts> https://assets.tiun.dev;
frame-src  <api hosts>;

The font host turned up on the next run, because the stylesheet asks it for one webfont. It gets font-src and img-src and nothing else. A font host has no business being a script source. Nothing is widened to a bare https:, which would allow every host on the internet and is the shortcut I did not want.

Then a readiness check that can fail:

function assertSdkArrived() {
  if (cspBlocked) throw new Error(CSP_BLOCKED);
  const loaded = performance.getEntriesByType("resource")
    .some((r) => r.name.includes("/background_js") && r.responseEnd > 0);
  if (!loaded) throw new Error("tiun runtime bundle never loaded");
  if (typeof tiun.login !== "function" || typeof tiun.checkout !== "function") {
    throw new Error("tiun sdk is missing login or checkout");
  }
}

A resource the policy blocked never enters the Resource Timing buffer, so asking the buffer is a real question with a real answer. Alongside it, a securitypolicyviolation listener records the directive:

const FATAL_DIRECTIVES = new Set(["script-src", "style-src", "frame-src", "connect-src"]);
document.addEventListener("securitypolicyviolation", (e) => {
  if (!String(e.blockedURI).includes("tiun")) return;
  if (FATAL_DIRECTIVES.has(e.effectiveDirective)) cspBlocked = `${e.effectiveDirective} blocked ${e.blockedURI}`;
});

That set matters. My first version treated any blocked provider URL as fatal, so the blocked webfont disabled a checkout that had just started working. A blocked font is ugly. A blocked script is a dead checkout. Only one of those should stop the sale.

Tests, since the whole point is not trusting a green light

19 of them. They pin facts rather than behaviour I cannot exercise headlessly.

One file reads the policy and asserts every directive the SDK needs names both hosts, that the font host is a font source but not a script or frame source, that nothing is a bare scheme, then that the directives with nothing to do with payments still read 'none'.

The other reads the frontend as text and holds it to the SDK: no .catch on getUser(), purchases read from the nested object by name, then every event subscribed to is one of the seven the SDK actually emits. It blanks comments before matching, because the comments describe the old wrong call and would otherwise match themselves.

128 tests pass now. They would all have passed before too, which is the point: none of them was looking here.

Where it stands

The sign in modal opens on margyn.xyz/pricing. Clicking that button did nothing yesterday.

I still have zero sales. That number has not moved. What changed is that it now means something about my product instead of something about my content security policy. Those are very different problems to have. For a month I was working on the wrong one.

The scanner is margyn, free for four checks, no account and no network call. Source is on GitHub.

A test said the server started. I deleted the server. It still passed.

Asuran — Fri, 28 Aug 2026 03:18:57 +0000

Here is a test from a real, well run Node project:

test('server starts', async (t) => {
  const app = build()
  await app.listen({ port: 0 })
  t.assert.ok(true, 'server started')
})

It reads fine in review. It runs green. Now delete the body of build() so the server never comes up. The test is still green, because the only thing it asserts is true. In the same file two more of these caught the error in a catch and asserted true there too, so even the failure path was green.

That is not a made up example. I found it in fastify at a pinned commit and opened a PR to fix it. More on that at the end.

A whole class of tests cannot fail

Once you start looking, the pattern turns up in a few shapes:

A literal: assert.ok(true), expect(1).toBe(1), a snapshot of a constant.
An assertion parked in a catch the happy path never reaches, so nothing is checked when the code works and nothing is checked when it breaks.
A status list that accepts both outcomes: assert.ok([200, 500].includes(res.status)).

Each one runs, counts toward coverage and guards nothing. Coverage is the trap. The line executed, so the tool that counts executed lines is happy. Whether the line would go red on a regression is a different question. It is the one that matters.

Why review misses it

A reviewer reading the diff sees a test called server starts, an await listen and a green tick. The name states intent. The assertion is what actually runs, yet ok(true) does not look like a problem until you stop and ask what would ever turn this test red. A missing check does not show up in a diff the way a wrong line does.

Finding them

I wrote a small scanner for this. No account, no config file, no network call:

npx margyn-scan /path/to/repo

One of its checks is cannot-fail: tests whose assertions hold whatever the code does. It also flags tests that assert nothing at all, files the build reads that git never committed, gates declared in package.json that no workflow invokes and linter exclusions that quietly drop tracked source. Every finding prints the command that reproduces it, so you confirm each one yourself instead of trusting a report.

The fastify case

The two tests above asserted ok(true) right after starting a server. I opened fastify/fastify#6962 to assert the server is actually listening instead. A maintainer reviewed it and suggested tightening it further to t.assert.ok(server.listening) with t.after for teardown. Both are in and the PR is open for review. The point is not that fastify is careless. It is one of the most carefully tested projects in the ecosystem, which is exactly why it matters that this still slips through.

The takeaway

A green suite tells you the tests ran. It does not tell you they would go red if the code they cover regressed. Those are two different claims. Teams merge on the first while believing the second. The gap is worth auditing, by hand or with something like margyn.

I built a scanner that audits your test suite instead of your code

Asuran — Tue, 18 Aug 2026 01:14:39 +0000

A passing test suite is supposed to mean something. Here are three ways it lies. All three stay green forever.

A test that asserts nothing. It calls the function, nothing throws, the runner prints a tick. It counts toward coverage and guards nothing. fastify at commit 6e95cb9 has seven of them.

An assertion that cannot fail. A literal answered inside a catch, a swallowed assertion, a status list that accepts both the success and the failure. The test reads as rigorous and holds whatever the code does. fastify has two.

A gate nobody invokes. A test-cov or verify script sits in package.json, reads as coverage to every reviewer, then no workflow ever calls it. express declares three.

None of that is visible to a code reviewer, to a coverage percentage or to an AI reading the diff, because the absence of a check does not appear in a diff.

So the tool reads the checks rather than the code

npx margyn-scan /path/to/repo

No account, no config file, zero dependencies, nothing uploaded. It answers in about a second. Every finding prints one command that reproduces it on your machine.

Six checks: no-assertion, cannot-fail, ignored-source (a file the build reads that git never committed, so a clean clone cannot build), unrun-check, lint-blindspot (linter exclusions that come from the ignore file rather than the linter's own config) and a mutation proof.

Measured at pinned commits, nothing tuned for the demo and nothing left out because the number was inconvenient:

Repository	Commit	Findings
chalk/chalk	`661317e`	0
sindresorhus/execa	`8017b27`	0
sindresorhus/got	`e3924aa`	1 unrun gate
expressjs/express	`a371447`	3 unrun gates
fastify/fastify	`6e95cb9`	13 (7 assert nothing, 2 cannot fail, 4 unrun)

Proof mode, because a scanner should not be believed either

npx margyn-scan . --prove

It runs each finding's own command, reads the output for the markers that finding predicted, then certifies it REPRODUCED. Anything it cannot reproduce is RETRACTED and dropped, so a CI gate never fails a build on a claim the tool could not show on your own tree. On fastify today: 13 findings, 13 reproduced.

That mode exists because the tool was badly wrong once. An earlier version of ignored-source produced 22 false positives across four real repositories. Proof mode certified every one of them, because the check and its proof had inherited the same wrong premise. I wrote that up separately, fixture included, since it is the more interesting half of building this.

It runs in CI and posts on the pull request

- uses: zkasuran/margyn@v0
  with:
    path: .
    comment: true
    sarif: true

It comments on the request, edits that same comment in place on every push, writes the report to the job summary and uploads the findings to code scanning, all with your own GITHUB_TOKEN and no service of mine in the path.

There is a live run to click rather than a screenshot to trust: pull request 1 plants two faults on purpose, a test that calls the scanner and asserts nothing plus a script no workflow invokes. The suite still reads 109 tests passing. The comment lists both findings with a reproduction each, both appear in the Security tab, then the job exits 1, which is the gate working.

What it does to itself

The mutation proof inverts a line, runs your suite, then reports the suite that stayed green anyway. Over this repository: 40 files tracked as source, 27 carry a mutation it knows how to make, 27 mutated, 27 caught, no survivors, across 108 tests.

Two honest numbers to go with that. It takes 208 seconds, because every mutation runs the whole suite once, which is why the default cap is four rather than every file. And zero survivors is a claim about this suite against seven operators, not a claim that the code is correct.

The first time it ran on itself it found eight survivors, one of them inside the mutation checker. The one that mattered most was in the worker: inverted, the entitlement branch handed a licence to every customer who had not paid. Nothing was watching that line.

What it will not do

It does not test your product, it audits the suite you already have. It does not upload your code, because there is no endpoint that accepts it. It will not tell you your code is correct. It tells you which of your existing checks cannot fail, which is a smaller claim and a checkable one.

The five static checks are free forever. The mutation proof is the paid one, because it runs your suite N times and that is the part with a real cost.

Try it on something you own

npx margyn-scan .

MIT, no telemetry, no account: margyn.xyz and github.com/zkasuran/margyn

The question I would genuinely like answered, because the default is still open: when a tool finds a hollow test in your suite, should it fail the build or only comment on the pull request? Margyn defaults to failing. I am not certain that is right.

Built during the tiun x Microlaunch challenge on hackwithus.dev.

AI assistance (Claude, Anthropic) was used while building this and while writing this post. The design, the review and every number above are mine. Each number ships with the command that re-derives it.

My scanner was wrong 22 times and its own proof certified every one

Asuran — Tue, 18 Aug 2026 00:53:06 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

I built a scanner that audits the verification layer of a repository rather than the code. One of its checks looks for a file the build reads that git never committed: the file is on your disk untracked, so every local run is green while a clean clone or a CI runner cannot read it at all. The diff looks innocent because the defect is an absence.

Because a scanner that only asserts is not much better than the tests it audits, it also has a proof mode. Every finding carries a machine-checkable command. --prove runs that command, reads the output for the markers the finding predicted, then certifies the finding as REPRODUCED. Anything it cannot reproduce is retracted and dropped, so a CI gate never fails a build on a claim the tool could not show on your own tree.

Then I pointed it at five real repositories I had lying around. It produced 22 findings on four of them. Every single one was wrong. Here is the part that still bothers me: the proof certified all 22.

What the false positive looked like

Reduced to the shape it kept taking, on a repository anybody can build in ten seconds:

mkdir -p web/public/tour web/src && git init
echo webm-bytes > web/public/tour/clip.webm
echo 'export const clip = "/tour/clip.webm";' > web/src/app.js
printf 'node_modules\nweb/dist\n' > .gitignore
git add -A && git commit -m "app plus the clip it plays, dist ignored"
npm run build     # or just: mkdir -p web/dist/tour && cp web/public/tour/clip.webm web/dist/tour/

clip.webm is committed. It sits in web/public, which is exactly where a Vite or Next project keeps assets it wants copied verbatim. The build then writes a second copy into web/dist, which is gitignored because a clean clone rebuilds it.

My check walked the source, found "/tour/clip.webm", resolved it against the files on disk, hit web/dist/tour/clip.webm first, saw that git ignores it and reported HIGH: the build reads a file that is not in the commit.

1. web/dist/tour/clip.webm is read by web/src/app.js but git ignores it, so it is not in the commit
   HIGH  ignored-source  web/dist/tour/clip.webm
   ignore rule: .gitignore:2:web/dist
   why: A clean clone or a CI runner cannot read this file.

Read on its own, that finding is defensible. The file it names really is untracked. Every fact in it is true. It is still wrong, because the thing the reader asked for is in the commit. A clean clone answers the request perfectly.

On the real repositories it was the same fault at scale: a Vite app reporting its whole public directory, a Foundry project reporting contracts/out (declared out = "out" in its own foundry.toml), a CLI reporting dist/cli.mjs (declared outdir: "dist" in its own build script). 22 findings, 4 repositories, zero real defects.

The part that actually scared me

Here is the same fixture through proof mode, on the unfixed scanner:

1. web/dist/tour/clip.webm is read by web/src/app.js but git ignores it, so it is not in the commit
   HIGH  ignored-source  web/dist/tour/clip.webm  REPRODUCED
   MARGYN_ABSENT_FROM_HEAD
   MARGYN_PRESENT_ON_DISK

2 findings: 2 reproduced.

REPRODUCED. The proof ran, both markers matched and the tool certified its own mistake with a green badge.

The proof was this:

git archive HEAD | tar -t | grep -qx 'web/dist/tour/clip.webm' || echo 'ABSENT from HEAD: web/dist/tour/clip.webm'
test -f 'web/dist/tour/clip.webm' && echo 'PRESENT on disk: web/dist/tour/clip.webm'

Both lines are correct. web/dist/tour/clip.webm genuinely is absent from HEAD and genuinely is present on disk. The proof passes because it asks the same wrong question the check asked: is this path committed, when the question that decides the bug is does anything in the commit answer the path the reader asked for.

That is the lesson I paid for. A check and its proof can agree, both be internally valid and both be wrong, because they inherited the same premise. Independent verification means asking a different question, not running the assertion twice.

The two fixes

A reader asks for a path, not for a file on your disk. The check now collects every needle a reader names, then reports only when no committed file answers it. web/public/tour/clip.webm in the commit satisfies /tour/clip.webm, whatever the build left in dist. The proof asks that question too, so a finding that slips through retracts itself instead of failing somebody's build over a copy of a committed file.

Regenerated output is not missing source. The check skips output a tool in the repository declares it writes: Foundry's out from foundry.toml, a Vite or Next or Cargo target from the script that runs it, an outdir in a build script or a tsconfig. Read from declarations and resolved to real paths, never from the directory happening to be called dist.

That second rule matters more than it looks. One of the five repositories had vendored real source into vendor/dist, a path no tool there declares. That is exactly the defect this check exists for. A crude "ignore anything named dist" would have deleted the only true positive in the set.

So the fixture keeps both cases. Same tree, one import added, scanner at HEAD:

1. vendor/dist/real.mjs is read by web/src/vendorised.js but git ignores it, so it is not in the commit
   HIGH  ignored-source  vendor/dist/real.mjs
   reproduce:
     git archive HEAD | tar -t | grep -qE '(^|/)dist/real\.mjs$' || echo 'NOTHING in HEAD answers dist/real.mjs'

The build copy is gone. The vendored module is still reported. And look at the new reproduce line: it asks whether anything in HEAD answers the reference, which is the question the fix is built on.

Verification

4 regression tests in test/checks.test.mjs, one per direction: quiet when the commit answers the path, still loud when nothing does, quiet on declared build output, still loud on vendored source under a path that only looks built.
The fixture above, run against the pre-fix commit and against HEAD: 2 findings before, 1 after, then the one that survives is the real one.
Full suite 108 tests passing. The tool's own mutation proof mutates all 27 mutable files of 40 tracked and reports no survivors.

While I was in there, one more thing turned up that nothing was watching: the SARIF helpUri for the mutation rule pointed at /docs#mutation while the docs heading carried id="mutation-check", so every uploaded finding linked to nothing. There is a test now that resolves the helpUri of every check the scanner can emit.

What I would tell my past self

Write the check and the proof from different premises. Otherwise the proof is a spell-check on the check's own reasoning. Mine was, for 22 findings, with a green badge on each.

Code: the fix is one commit, efc5657, in github.com/zkasuran/margyn. The scanner runs with no account and no config: npx margyn-scan /path/to/repo. --prove is the mode that got humbled here.

AI assistance (Claude, Anthropic) was used while writing this post and while implementing the fix. The bug, the reduced fixture, the two rules and every number above were verified by me by running them: the outputs quoted are captured stdout from the pre-fix commit and from HEAD, not retyped.

Pup Card: what's your dog really thinking? (Gemini 3.5 Flash)

Asuran — Fri, 14 Aug 2026 14:10:06 +0000

This is a submission for Weekend Challenge: Dog Days Edition.

What I Built

Pup Card turns one dog photo into a shareable card. Drop in a photo or hit Surprise me for a random dog. Gemini 3.5 Flash reads the picture and hands back:

the likely breed with a confidence score, plus the mix if it is not purebred
the dog's mood and a plain read of its body language
a first-person caption, as if the dog is talking
two or three breed-appropriate care tips
a fun fact

It all renders as a card you can download as a PNG and share. I wanted something that was genuinely fun to use on a Saturday and that leaned on the one thing a multimodal model is really good at: looking at an image and saying something specific and true about it.

Demo

Live demo: https://pupcard.vercel.app

It needs no login and runs on your phone. If you do not have a dog photo on hand, the Surprise me button pulls a random one so you can try it right now.

Code

zkasuran / pupcard

Snap a dog photo, get an AI Pup Card: breed, mood and what they're thinking. Powered by Gemini 3.5 Flash.

🐶 Pup Card

Snap a dog photo and get back a Pup Card: the likely breed, a read on the dog's mood and body language, a first-person line of what they are probably thinking and a few care notes. One multimodal call to Google Gemini 3.5 Flash does the whole read.

Built for the DEV Weekend Challenge: Dog Days, for the Best use of Google AI category.

Live demo: https://pupcard.vercel.app

What it does

Upload a dog photo (or hit Surprise me to pull a random dog from dog.ceo).
Gemini 3.5 Flash looks at the image and returns a structured read: breed guess with a confidence score, likely mix, mood, body language, a playful first-person caption, 2-3 care tips and a fun fact.
The result renders as a card you can download as a PNG and share.

No dog photo on hand? The Surprise me button means anyone can…

View on GitHub

How I Built It

The whole read is a single generateContent call to gemini-3.5-flash. The photo goes in as inline image data next to a prompt that asks for JSON in a fixed shape. A zod schema validates the response so the front end always gets the same fields. If the model returns malformed JSON the server retries once with the parse error fed back. If the primary model is overloaded or out of quota it falls back to gemini-2.5-flash.

The stack is deliberately small:

Gemini 3.5 Flash through @google/genai for the read
TypeScript everywhere, with Hono for the local server
plain Node serverless functions on Vercel for /api/analyze and /api/random-dog, so there is no long-lived server to run
a vanilla HTML, CSS and JS front end with a <canvas> export for the downloadable card, so there is no build step and nothing shipped to the browser but three static files
the free dog.ceo API for the random dog

Two details I liked. Photos are downscaled in the browser before upload, which keeps the request tiny and the read fast. And the caption is where 3.5 Flash shows off: it picks up on the actual photo, the odd-colored eyes, the golden-hour light, the paw draped over a toy, instead of writing something generic.

The Gemini key stays on the server. The public endpoint has no login, so it is guarded by a size cap and a simple per-IP rate limit instead.

Prize Categories

Best Use of Google AI.

Gemini is not a feature bolted onto Pup Card, it is the product. One multimodal Gemini 3.5 Flash call does the breed identification, the mood and body-language read, the writing and the care tips, all returned as structured JSON that the card renders directly. There is no separate model, no rules engine and no template behind it.

Built over the challenge weekend. I used an AI coding assistant to help write it, then designed the prompt, reviewed the code and tested the whole thing end to end against real dog photos with live Gemini calls before shipping.