DEV Community: StarKnight

A company brain that shows its work: multi-hop GraphRAG on FalkorDB, with every hop replayed in 3D

StarKnight — Fri, 16 Oct 2026 18:26:06 +0000

Some questions about a software company are not answered by any single document. "Who should review a change to the vector index code?" "Which pull request fixed this issue, and which release shipped it?" "What is affected if we change this directory?" The answer lives in how people, pull requests, files and releases connect, and text retrieval is a poor tool for following connections.

For WeMakeDevs Graph Hacks I built Threadline, a company brain for an open-source company, with FalkorDB as its only database. An agent answers by walking the graph with typed tools, every hop is replayed in 3D with the exact Cypher it ran, and a second agent checks each citation against a real path in the graph before the answer is marked verified.

Live app: https://threadline-graph.vercel.app (six recorded runs replay instantly with no model call)
Code (MIT): https://github.com/StarKnightt/threadline
Demo video: https://vimeo.com/1236637540

This post covers the graph model, the Cypher that does the work, how the verifier uses the graph, how I measured whether the graph helps, and what did not work.

The data: an open-source organisation as a company

I used the public GitHub history of the FalkorDB organisation: the core database, the browser, QueryWeaver, GraphRAG-SDK, the TypeScript client, the MCP server and the docs. It is public, it has real organisational structure (reviews, closing references, releases, cross-repository mentions), and anyone can check the answers against GitHub.

Two constraints shaped the scope:

Privacy. People are stored by GitHub login and avatar URL only. No names, emails or locations. Bots are flagged and kept out of expertise rankings.
The free tier's 100 MB. FalkorDB Cloud's free instance holds about 100 MB in memory. I kept items updated since January 2023 and used 256-dimension embeddings. The result is about 27,000 nodes and 136,000 relationships, and GRAPH.MEMORY USAGE reports 41 MB, of which text chunks with their vectors take 19 MB.

The graph model

Every brain node carries an :Entity label and a unique key such as pr:FalkorDB/FalkorDB#1461 or person:swilly22. One index on Entity.key lets any tool address any node, and the same keys double as citations in answers, which is what makes them checkable later.

The core relationships come straight from GitHub: AUTHORED, REVIEWED, MERGED, COMMENTED_ON, TOUCHED (PR to file), IN_DIR (file to directory to parent directory), CLOSES, REFERENCES (with a crossRepo flag), SHIPPED_IN and MENTIONS (to a fixed dictionary of about 50 concepts such as vector-index or pagerank, matched without an LLM).

The most useful part turned out to be the derived edges, computed in Cypher after each load and again by a daily sync:

MATCH (p:Person)-[:AUTHORED]->(pr:PullRequest)-[:TOUCHED]->(:File)-[:IN_DIR*1..6]->(d:Directory)
WHERE p.isBot = false AND pr.mergedAt IS NOT NULL
WITH p, d, count(DISTINCT pr) AS authored, max(pr.mergedAt) AS last
MERGE (p)-[k:KNOWS]->(d)
SET k.authored = authored, k.authoredLastAt = last

KNOWS turns thousands of file touches into one weighted edge per person and directory, so "who knows this area" becomes a one-hop question at query time. REVIEWS_FOR (reviewer to author) feeds PageRank, CO_CHANGES (directories changed by the same PRs) feeds label propagation, and CONTRIBUTES_TO feeds betweenness. Precomputing these is the graph equivalent of an index: it moves the expensive walk to load time.

The Cypher that does the work

Who knows an area, now. Expertise decays: someone who touched the index code three years ago is a weaker answer than someone who reviewed it last week.

MATCH (p:Person)-[k:KNOWS]->(d:Directory)
WITH p, d, k, (2 * k.authored + k.reviewed) AS weight, ($now - k.lastAt) / 86400.0 AS ageDays
RETURN p.login, k.authored, k.reviewed, weight * exp(-ageDays / 180.0) AS score
ORDER BY score DESC LIMIT 10

Decision trace. From an issue to the PR that closed it, to the release that shipped it, with the source of each link recorded on the edge:

MATCH (i:Issue {key: $key})
OPTIONAL MATCH (fix:PullRequest)-[c:CLOSES]->(i)
OPTIONAL MATCH (fix)-[:SHIPPED_IN]->(rel:Release)
OPTIONAL MATCH (author:Person)-[:AUTHORED]->(fix)
RETURN i.title, fix.key, author.login, rel.tag, c.source

Impact. A variable-length walk down the directory tree to open pull requests:

MATCH (d)<-[:IN_DIR*1..6]-(f:File)<-[:TOUCHED]-(pr:PullRequest)
WHERE pr.state = 'OPEN'
RETURN pr.key, pr.title, count(DISTINCT f) AS files ORDER BY files DESC

Graph algorithms as tools. algo.pageRank over REVIEWS_FOR finds key reviewers; algo.betweenness over people and repositories finds who bridges teams; algo.labelPropagation over CO_CHANGES finds the subsystems that actually change together, which do not always match the directory layout.

The full catalogue, 27 queries with parameters and measured timings, is generated into docs/CYPHER.md from the same constants the server runs. pnpm verify:cypher runs every one against the live instance; most return in 65 to 160 ms including the round trip from India to the Mumbai region.

Three FalkorDB details I learned the hard way

shortestPath() lives in WITH or RETURN, and it is directed only. My first verifier query put it in MATCH and failed; moving it to WITH then failed on the undirected pattern. allShortestPaths() in MATCH accepts undirected patterns, so the verifier uses that.
Betweenness follows edge direction. People only pointed at repositories, so no shortest path ran through a person and every score was zero. A derived reverse edge (HAS_CONTRIBUTOR) fixed it.
Vector score is a distance. db.idx.vector.queryNodes yields cosine distance, where lower is closer. I had sorted it descending, so entity resolution quietly picked the farthest of the nearest neighbours. A title-only query exposed it: the right chunk came back with a "score" of 0.18 while worse matches scored 0.36. Every vector query now reports 1 - score as similarity.

Typed tools first, free-form Cypher last

The Investigator agent (Vercel AI SDK, temperature 0, at most six steps) gets typed tools: find_entities, who_knows, trace, connect, impact, activity, rank, expand, details, draft_triage, and finally cypher_readonly for questions nothing else fits. Typed tools are easier for a model to call correctly than raw Cypher, and each one is a query I wrote and tested by hand.

The free-form tool runs through GRAPH.RO_QUERY as a separate read-only database user (FalkorDB Cloud's Read role), rejects write clauses and enforces a LIMIT and a 5 second timeout. I checked both guards directly: GRAPH.RO_QUERY refuses writes, and the read-only user cannot call GRAPH.QUERY at all.

Each tool returns rows plus the node keys it touched, the edges it traversed and its Cypher. Those stream to the browser over server-sent events and drive the 3D view, so the visualisation is a direct rendering of what the agent actually did rather than an illustration of it.

A verifier that uses the graph

Language models cite things they never saw. So a Verifier agent, deterministic with no LLM in its check, inspects every citation in an answer:

the cited key must appear in some tool result, and
it must be reachable from the question's entry nodes within four hops over work relationships. Hub edges such as IN_REPO are excluded; otherwise everything is two hops from everything and the check proves nothing.

MATCH (a:Entity {key: $entry}), (b:Entity {key: $cited})
WITH a, b
MATCH p = allShortestPaths((a)-[:AUTHORED|REVIEWED|MERGED|TOUCHED|IN_DIR|CLOSES|REFERENCES|SHIPPED_IN|MENTIONS|KNOWS*..4]-(b))
RETURN length(p) AS hops, [n IN nodes(p) | n.key] AS keys LIMIT 1

If a citation fails, the Verifier hands the claim back to the Investigator once, with the reason. Claims, verdicts (CHECKED_BY {verdict, pathLength}) and handoffs (HANDED_OFF) are written to a shared FalkorDB graph, tl_shared, which is the coordination layer between the two agents.

Here is a real example from a production run during final testing: "Which pull request fixed issue #1003 in the falkordb-browser repository, and in which release did the fix ship?" The first answer cited release:v1.4.0, an abbreviated key that matches no node. The Verifier rejected it, the Investigator rewrote the citation as release:FalkorDB/falkordb-browser@v1.4.0, and the second check verified 2 of 2. The whole exchange took 2.9 seconds and is replayable here. A direct Cypher query agrees: PR #1007 closes the issue through a closing reference and shipped in v1.4.0 according to the release notes.

Building the verifier changed the rest of the system. It caught abbreviated keys like pr:#2527, keys glued to full-width brackets, and answers about org-wide rankings that had no entry node at all. Each case changed how citations are resolved and anchored.

Memory as graphs, one per visitor

FalkorDB holds many isolated graphs on one instance, so each visitor gets their own: tl_mem_<id>. A turn is a node chained to the previous one with NEXT; its tool calls hang off it with edges to the brain nodes they touched, so any past answer can be replayed. Facts come only from explicit statements ("I work on the TypeScript client") and keep a LEARNED_FROM edge to the turn they came from, so the app can show why it believes something. Follow-ups such as "same question, but for QueryWeaver" resolve against the previous turn. Verified tool sequences become shared procedural playbooks with their own vector index, and the next similar question starts from that plan.

Drawing the agent's reasoning

The 3D view is React Three Fiber. The background is a constellation of repositories, active directories, people and concepts, laid out once with d3-force-3d and stored as x, y, z properties on the nodes, so even the layout comes from FalkorDB. When a step arrives, its nodes ignite in their label colour, traversed edges carry a light pulse, the camera eases to the new frontier, and the verifier's evidence path turns gold. Labels are one DOM layer projected every frame with simple collision avoidance, which avoided creating a React root per label.

There is also a light theme. Bloom only works when colours exceed 1.0 against a dark background, so the light stage drops it: nodes switch to darker, saturated variants, faded context fades toward the pale background instead of toward black, and the verified path becomes a deeper gold drawn thicker. An axe-core check passes WCAG AA contrast on every page in both themes.

Acting, not only answering

draft_triage turns an open issue into a ready-to-paste triage comment: likely duplicates by vector similarity, suggested reviewers from the people who changed the same code when similar issues were fixed, and suggested labels. It never posts anything. The same typed tools are served over MCP at /api/mcp, so any MCP client can use the graph directly.

Did the graph actually help?

I wanted a number I could defend, so the evaluation was designed so the graph cannot grade itself. Sixty questions in six categories are generated from templates over the raw GitHub JSON, and their gold answers are computed by a separate script from that JSON, without FalkorDB. Both systems use the same model on a given question, the same 256-dimension embeddings and the same FalkorDB vector index; the vector baseline takes the eight nearest chunks with their document metadata. Scores are set-based F1 over each answer's final ANSWER: line.

Groq's free tier caps tokens per model per day, so questions rotated across three Groq models, and the 29 questions those caps left unfinished were completed on OpenAI gpt-5.6-luna with reasoning off. Every comparison is still paired within a question.

All 60 questions were answered by both systems. Models: qwen/qwen3.8-27b (Groq) on 7 questions, openai/gpt-oss-120b (Groq) on 24 questions, gpt-5.6-luna (OpenAI, reasoning off) on 29 questions.

The questions were split across models: Groq's free daily token caps stopped the rotation part way, so the questions without a finished pair were completed with OpenAI gpt-5.6-luna. Each question was answered by one model, and its graph and vector runs always used that same model, so every comparison is paired within a question. The models received different mixes of categories, so the per-model rows are not a ranking of the models. Which model answered each question is listed in eval/summary.json and on every question at /eval.

Category	Pairs	Graph agent F1	Vector RAG F1
Single-hop lookup (control)	10	1.00	0.60
Expertise (PR to file to directory)	10	0.83	0.00
Decision trace (issue to PR to release)	10	0.85	0.00
Cross-repo connection	10	0.88	0.00
Impact (open PRs under a directory)	10	0.30	0.10
Aggregation (top reviewers or openers)	10	1.00	0.00
All	60	0.81	0.12

Model	Questions	Graph agent F1	Vector RAG F1
qwen/qwen3.8-27b (Groq)	7	1.00	0.29
openai/gpt-oss-120b (Groq)	24	0.81	0.04
gpt-5.6-luna (OpenAI, reasoning off)	29	0.77	0.14

The graph agent used about 6,190 tokens per answer against 1,479 for the baseline. 95 percent of the graph agent's citations were verified by a graph path. Median latency in the eval: 54.4 s graph and 9.4 s vector on qwen/qwen3.8-27b, 19.5 s graph and 1.5 s vector on openai/gpt-oss-120b, 7.2 s graph and 2.4 s vector on gpt-5.6-luna. The Groq runs went back to back under an 8,000 tokens per minute limit, so their graph latencies include waiting for that limit; single live questions took 1.6 to 15 s in testing. The vector baseline does best on single-hop lookup questions (F1 0.60), the control category where the answer sits in one document. Widest gap: aggregation. Every question, gold answer, raw answer and score is in eval/ and on /eval.

How I read these numbers:

The control category behaves as it should. When the answer sits in one document, vector retrieval does reasonably well (0.60). That is the point of a control: the gap on the other categories is about structure, not a crippled baseline.
I gave the baseline a fair chance, and it mattered. For the lookup questions, the right chunk was in the top eight for 1 of 10 questions with a question-first phrasing, 6 of 10 with the title first, and 8 of 10 with the bare title. I used the title-first phrasing.
Aggregation is where retrieval cannot win. "Who reviewed the most PRs in 2024?" is a count over hundreds of edges. No chunk contains it, so the baseline either refuses or guesses; the graph answers with one count().
The graph costs more tokens. About 4 times the baseline per answer, because tool results carry rows and keys. For these questions that trade is worth it, but it is a real cost.

The impact category, and a bug the eval found

Impact scored only 0.30, and the reason is instructive. The graph agent missed seven of its ten questions, and in six of those it answered "none". Checking the misses with Cypher showed the data was there: PR #272 is open and touches files under clients/python/queryweaver_client, exactly as the gold answer says. The problem was the tool. The questions write paths with a trailing slash (clients/python/queryweaver_client/), the impact tool matched them with CONTAINS against directory paths stored without one, and so the query returned nothing. The agent then reported, truthfully, that its tool found no open PRs. Another tool in the same file already stripped slashes; impact did not.

The fix is one line. After it, the impact query itself returns the gold PRs for 9 of the 10 impact questions; the tenth sits under .claude/, which the tool deliberately skips. On production, the exact question the agent missed first now comes back as PR #272 with 1 of 1 citations verified (trace). That is one spot check, not an evaluation: I have not rerun the agent evaluation, so the published numbers above are the measured, pre-fix ones. The lesson for me is that an honest eval is mostly useful for what it tells you about your own tools.

Running on free tiers, and what it costs

The whole project runs on free tiers plus a few cents of OpenAI usage:

FalkorDB Cloud free tier (AWS ap-south-1). It has no persistence and stops after a day without use, so a daily Vercel Cron sync keeps it in use, and pnpm seed rebuilds the graph from a published snapshot in about two minutes. The free tier also has no TLS, so only public GitHub data and least-privilege credentials travel over that connection.
Groq's free tier for the agent (gpt-oss-120b first, then gpt-oss-20b and qwen3.8-27b when rate limited). The caps are per model per day and shared with the evaluation.
A capped OpenAI fallback for live questions only. When every Groq model is capped, a live question continues on OpenAI gpt-5.4-mini, with at most 4 calls of up to 2,000 output tokens per run, 10 runs a day and 25 runs a month across all visitors. At Standard prices ($0.75 input, $0.075 cached input, $4.50 output per million tokens), the token counts of real fallback runs cost about $0.001 to $0.0035 each, a worst-case run is about $0.07, and the monthly cap bounds the fallback at under $2.
Spend so far: embedding the corpus cost about four US cents, and the 29 OpenAI evaluation questions cost about $0.024.

Recorded showcase runs replay from the stored traces in FalkorDB with no model call at all, which is also what makes the demo reliable during judging.

Limitations

The eval pools models. Questions were split across three models because of free-tier caps. Comparisons are paired within each question, but the per-model rows are not a ranking, and a single-model run would be cleaner.
The impact numbers are pre-fix, as described above.
Expertise is inferred from authored and reviewed PRs. Design discussions in chat, calls or other channels are invisible to the graph.
Release attribution falls back to "first release after the merge" when release notes do not list a PR. The source is recorded on each SHIPPED_IN edge, but the fallback can be wrong.
Concept matching uses a fixed dictionary of about 50 FalkorDB terms. It is precise but does not generalise to another organisation without editing.
Concurrent live questions slow down under Groq's 8,000 tokens per minute limit, and past the daily caps the app falls back, then stops and points to the recorded runs.
Decision nodes extracted by an LLM, linked to the comments that justify them, were planned and not built.

What I would do next

Rerun the full evaluation on one model, after the impact fix.
LLM-extracted Decision nodes with JUSTIFIED_BY edges to the comments behind them.
Point it at a second organisation; ingestion only needs an org name, but the concept dictionary needs work.
Persistence beyond the free tier, and a fine-grained token for the daily sync.

AI disclosure

I planned the concept, graph model, query catalogue and evaluation method before the event as notes, which the rules allow. During the event, most of the implementation code was written by an AI coding agent in Cursor (Anthropic Claude) working from that plan under my direction, and this post was drafted with its help and edited by me. Every Cypher query was run and checked against the live FalkorDB instance before it went into the catalogue, and the evaluation's gold answers are computed independently of FalkorDB. At runtime the app uses Groq-hosted open-weight models with a capped OpenAI fallback, plus OpenAI embeddings.

Cashcast: which invoices will be paid late? TabPFN on a laptop, checked month by month

StarKnight — Mon, 12 Oct 2026 19:37:26 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Cashcast tells a small business which unpaid invoices are likely to be paid late, when each payment is likely to arrive, and how much cash to expect over the next eight weeks. It runs on your own laptop, so your client list and amounts stay with you.

It is built for people who send invoices and then wait: freelancers, small agencies, and the bookkeepers who manage receivables for several small clients. For them, late payment is not an abstract metric. It decides whether rent goes out on time and how much of the week goes into writing reminder emails.

Cashcast reads an invoice history (one CSV row per invoice: client, issue date, due date, amount, paid date) and, for every invoice still open, shows:

the chance it will be paid late,
a likely payment date with an 80% range, updated for how long the invoice has already been open,
a suggested next step: contact today, send a reminder now, remind on the due date, or do nothing,
the weekly cash you can expect from invoices already sent.

Who would pay for it, and why. Cloud accounting suites offer payment reminders and cash-flow views, but they expect the whole ledger in their cloud and a monthly subscription. Cashcast is a single-purpose tool that works on the CSV export people already have. A one-time license for freelancers, or a per-client fee for bookkeepers, is a plausible model, because the value is easy to state: fewer surprises in cash flow and less time spent chasing the clients who were going to pay anyway. Running the model locally is what makes a small one-time price workable, since there is no per-prediction API bill behind it.

Demo

Live demo (no install): https://starknightt.github.io/cashcast/

The live page is a replay. It uses IBM's public accounts receivable sample: 2,466 invoices from 100 business clients, 2012 to 2013, 36% of them paid late. Pick a date and you see what Cashcast would have told you that morning, using only invoices that had been paid before that date. Then tick Reveal what happened to put the real payment dates next to the predictions.

Things to try:

Click Sep 1, 2013. 78 invoices are unpaid. Cashcast expects 4,327 to come in over the next four weeks (the sample has no currency). Tick Reveal what happened: 4,241 actually came in.
Click Needs action to see only the invoices it suggests chasing, each with a one-line history computed from the data, for example "Paid 13 of 15 past invoices late, 36 days to pay on average."
Scroll to Does it work? for the month-by-month backtest, including the result where TabPFN did not win.

Using it on your own invoices needs Python 3.10 or newer and works on a CPU. python -m cashcast.serve opens the same page in local mode with a file picker. The file goes from the browser to the Python process on 127.0.0.1 and nowhere else. On the bundled 404-row sample ledger, scoring takes about 7 to 9 seconds on my laptop CPU.

Code

StarKnightt / cashcast

Know which invoices will be paid late, and when the money will arrive. TabPFN on your laptop, client data stays local.

Cashcast

Know which invoices will be paid late, and when the money will arrive, before you chase anyone.

Cashcast is a small receivables tool for freelancers, small agencies and the bookkeepers who manage invoices for them. You give it your invoice history as a CSV. TabPFN, an open tabular foundation model, runs on your own laptop and tells you, for every unpaid invoice:

the chance it will be paid late,
a likely payment date with an 80% range, updated for how long the invoice has already been open,
a suggested next step (contact today, send a reminder now, remind on the due date, or nothing),
how much cash to expect in each of the next eight weeks.

Your client list and amounts never leave your computer.

Live demo: https://starknightt.github.io/cashcast/ (demo mode, no install)

Demo mode

The live page replays IBM's public accounts receivable sample (2,466 invoices from 100 business…

View on GitHub

MIT licensed. The pipeline is about 660 lines of Python plus a static page with no build step:

cashcast/data.py       load the IBM sample or any CSV into one schema
cashcast/features.py   what was known on the day each invoice was sent
cashcast/model.py      TabPFN v2 classifier and regressor
cashcast/forecast.py   late chance, payment range, 8-week cash simulation
cashcast/backtest.py   month-by-month test against four baselines
cashcast/demo.py       replay data for the live page, scored against reality
cashcast/serve.py      local mode (standard library HTTP server)

How I Built It

TabPFN does the predicting

TabPFN is a tabular foundation model from Prior Labs. It was pre-trained on millions of synthetic tables, so instead of training a model on your data, you hand it your table and it predicts in a single forward pass. For a small business with a few hundred or a few thousand invoices, that matters: there is no training loop to tune and nothing to babysit.

Cashcast uses two TabPFN v2 models on the same features:

a classifier for "will this invoice be paid after its due date?",
a regressor for "how many days until it is paid?", asked for 49 quantiles so it returns a full distribution per invoice rather than one number.

Features from the day the invoice was sent

The easiest way to fool yourself with this kind of data is leakage. The IBM sample includes columns such as DaysToSettle and Disputed that are only known after an invoice is settled, and using them makes the task look far easier than it is.

So features.py builds every row from what you knew on the day the invoice went out: how many of that client's invoices had already been paid, how often they were late, their average and most recent days to pay, how many of their invoices were still open, and how this amount compares with their usual. Invoices paid after the issue date are invisible to that row.

From a distribution to a decision

The regressor's quantiles are the useful part. An invoice that is still unpaid after 25 days is a different bet from one sent yesterday, so forecast.py conditions each distribution on the invoice's current age. From that it reads:

the late chance, as P(paid after the due date | still unpaid today),
the 10th, 50th and 90th percentile payment dates,
2,000 simulated payment schedules per date, summed per week, for the cash forecast.

The next-step labels are plain rules on top (overdue: contact today; 50% or more: reminder now; 30% or more: remind on the due date). The history line beside each invoice is computed, not generated, so it cannot state something the data does not support.

Did it work? A month-by-month backtest

For each month from January to October 2013, every model was trained only on invoices paid before that month began, then predicted the invoices sent during the month: 1,075 predictions in total. TabPFN on a CPU accepts up to 1,000 training rows by default, so each month uses the 1,000 most recent paid invoices.

Model	AUC	Brier	Chase recall	Pay day error	80% range hit
TabPFN v2	0.882	0.127	51.0%	5.92 days	79.1%
Logistic / linear regression	0.890	0.122	50.4%	5.81 days	79.3%
Client's own history	0.870	0.135	50.7%	6.54 days	n/a
Gradient-boosted trees	0.864	0.154	49.9%	6.23 days	57.5%
Base rate	0.5	0.218	18.4%	10.11 days	n/a

"Chase recall" answers the business question: if you only chase the riskiest 20% of each month's invoices, what share of the late ones do you catch? Lower Brier and pay day error are better.

The honest reading: on this dataset, a plain logistic and linear regression is as accurate as TabPFN. I expected TabPFN to win clearly, and it did not. It does clearly beat gradient-boosted trees trained on the same rows, whose 80% ranges caught the real payment date only 57.5% of the time.

The probabilities are well calibrated. Of the 586 invoices TabPFN put below 20%, 8.7% were paid late (it predicted 6.2% on average). Of the 116 it put above 80%, 90.5% were (it predicted 87.4%).

Where TabPFN earns its place

A linear model with a residual band gives every invoice the same width of uncertainty. TabPFN gives each invoice its own distribution. To test whether that matters, I split clients into thirds by how much their past payment times varied:

80% range	Steady clients	Middle	Erratic clients
TabPFN	80.5% hit, 16.0 days wide	83.3%, 17.7 days	73.3%, 19.9 days
Linear	83.8% hit, 18.4 days wide	84.2%, 18.5 days	69.7%, 18.4 days

TabPFN narrows the range for reliable payers and widens it for erratic ones, which is what you want, but the effect is modest and both under-cover the erratic third. The distribution is also what lets Cashcast answer "this invoice is 20 days old and still unpaid, now what?" without a separate model.

Checking the replay itself

The numbers on the live page were scored against what really happened (10 dates, 905 unpaid-invoice rows):

Of 296 "send a reminder now" flags, 78% were in fact paid late. Among all not-yet-due invoices on those dates, 43% were.
The real payment date fell inside the 80% range 77.1% of the time, with a typical miss of 5.78 days.
The weekly cash forecast is the weak spot: the actual amount landed inside its 80% range in only 67.5% of 80 weeks. The simulation treats invoices as independent, so the range is too narrow.
Each date takes about 40 seconds on a laptop CPU (classifier plus regressor on 1,000 rows).

What it still cannot do

The sample contains only invoices that were eventually paid, so Cashcast has never seen an invoice that is written off.
Every invoice in the sample has 30-day terms, so the effect of other terms is untested.
It suggests whom to contact; it does not send anything.
The live page is a replay of a public dataset. Before trusting it on your own ledger, run the same backtest on your history (python -m cashcast.backtest shows how).

Why Does Open Innovation Matter?

It keeps the client list at home. An invoice ledger is a list of who pays you, how much and how reliably. With open weights, Cashcast runs on the laptop that already holds that file. The local server binds to 127.0.0.1, and after the one-time weight download, scoring still works with all network traffic blocked (I tested it by routing every request to a dead proxy).

It makes a small, cheap product possible. A one-time-license tool cannot carry a per-prediction API bill. Open weights mean the marginal cost of a forecast is a few seconds of CPU.

The license has to allow selling it. Prior Labs recommends its newest model, TabPFN-3.5, through its API under a free non-commercial license. That is a fine choice for research, but a side hustle is commercial by definition. TabPFN v2 weights are published under the Prior Labs License 1.1 (Apache 2.0 with an attribution requirement, which is why the page says "Built with PriorLabs-TabPFN"), download from Hugging Face without an account, and run offline. For this product, the openness of the license mattered as much as the openness of the weights. The trade-off is that I may be leaving accuracy on the table by not using the newest model.

Openness made the evaluation possible. Because the model, the data and the code are all inspectable, I could run every baseline on exactly the same rows and publish the result where TabPFN did not win. A black-box API would have given me a number to trust instead of a number to check.

Prize Categories

Best Use of TabPFN. TabPFN does all of the predicting in Cashcast. A TabPFN v2 classifier estimates the chance each invoice is paid late, and a TabPFN v2 regressor returns a 49-quantile distribution of days to payment, which Cashcast conditions on invoice age to produce payment ranges, reminder decisions and an eight-week cash simulation. It is tested month by month against four baselines on 1,075 invoices, with calibration and per-client range checks, and the results, including where a linear model matched it, are published in the repository and on the live page.

PageTwins: I Planted One Defect on 100 Landing Pages. Which AI Graders Noticed?

StarKnight — Sun, 11 Oct 2026 06:51:47 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

I build Shipgrade, a tool that grades landing pages. It reads a page the way a script does: the headline, the buttons, the prices. It never looks at the page. That left me with two questions I had never measured. What does a grader miss when it only reads the page? And when a model does get a screenshot, does it notice what is actually broken, or does it give a confident score either way?

To find out, I built PageTwins: 10 original landing pages and 100 "evil twins". Each twin is identical to its original except for one planted defect.

Defect	What changed in the twin
Missing button	every sign-up, buy or download button removed
Hidden price	paid prices replaced with "Contact us"
Contradiction	one claim changed so two parts of the page disagree
Vague headline	the headline no longer says what the product is
Typos	three misspelled words in prominent text
Placeholder text	"Lorem ipsum" or a `{{template_variable}}` left in the copy
Blocking popup	a signup modal covers the first screen
Low contrast	headline faded to about 1.25:1 against its background
Broken image	the hero image fails to load
Cut-off text	the headline box is clipped partway through the first line

All brands, copy and prices are fictional, and the art is SVG drawn by code. Every twin was rendered in headless Chromium and measured, and kept only if the measurement for its own defect changed (for example, the number of sign-up buttons dropped to zero) while every other measurement matched the original. The last three defects are visual by construction: the twin's text is byte-for-byte identical to the original's, so a grader that only reads the text cannot see them.

Each original also has two controls that a careful grader should leave alone: an identical copy, and a copy with a harmless change (a brand color swap at the same contrast, reordered feature cards, or a reworded subheadline).

The benchmark has four tasks, written with the kaggle-benchmarks SDK and pushed from the Kaggle CLI:

Task	The model sees	Score
Pairs, screenshots	two full-page screenshots, shown in both orders	half for defects picked and named in both orders, half for controls answered "same" in both orders
Pairs, text	the same pairs as visible text with element roles	same
Single page, screenshot	one screenshot, no reference	share of twins where the model flags the defect and does not flag it on the original
Single page, text	one page as text	same

Pairs is how a careful reviewer works: compare the new version with the last one. The single-page task is how most graders run in practice, mine included: one page, no reference, "what is wrong with this?". Showing every pair in both orders means a pick only counts if it survives the swap, and it measures position bias for free. In the single-page task a model gets credit only if it flags the defect on the twin and does not flag the same problem on the original, so a model that flags everything scores zero.

Every model gets the same prompt; the screenshot and text versions differ by one sentence that describes the input. Replies are short JSON, and anything unreadable counts as wrong.

Models Tested

Ten models, 34 scored runs, every one on Kaggle's model proxy:

All four tasks (they accept images): Claude Sonnet 5.5 and Claude Haiku 4.5, GPT-6.1 Sol and GPT-6 Luna, Gemini 3.7 Flash and Gemini 3.5 Flash-Lite, and Gemma 4 31B.
The two text tasks (text-only models): GLM-5, DeepSeek-R1 and Qwen 3 235B.

The idea was a stronger and a cheaper model from each of the three big vision providers, so I could see whether "smaller" means "slightly worse" or "a different kind of grader", plus open-weight models, because a grading tool like mine could run one locally. Kaggle gives $10 of model credit per day, so the lineup leaves out the most expensive tiers. The 26 runs I could price came to about $21, estimated from each run's token counts and per-token prices (I had no price for Gemma, GLM-5 and Qwen).

Two planned models are missing, and I would rather say so than quietly drop them:

Grok 4.6 is listed by kaggle b t models, but Kaggle's proxy answered 404 model not found for every xAI model I tried (Grok 4.6, Grok 4.5 and both Grok 4.20 models) from October 7 to October 11, and my runner kept probing until the end. Grok has no results.
gpt-oss-120b hit Kaggle's notebook time limit on two attempts and was dropped.

Every other planned run finished and is on the leaderboard.

Findings

1. The strong models do not need the "before" picture

I expected the reference page to be the big advantage. With screenshots, it barely mattered for the top four: Claude Sonnet 5.5 caught 98% of defects judging a page alone and 100% with the original next to it; GPT-6.1 Sol 98% and 96%; Gemini 3.7 Flash 96% and 98%; Gemma 4 31B 93% and 95%. Sonnet 5.5 got a perfect pairs score: all 100 defects picked and named in both orders, and all 20 controls called "same".

The two small models are where the pair changed things, and not always for the better. Gemini 3.5 Flash-Lite went from 63% to 72% caught with a reference, but it failed to call 9 of its 10 harmless pairs "same" in both orders. Claude Haiku 4.5 also passed only 1 of its 10 harmless pairs, and it showed real position bias: it picked the second screenshot 62% of the time and gave the same letter in both orders on 24% of defect pairs, which is exactly what the swap is there to catch. Identical copies were easy for everyone (10 of 10 for every model in every pairs run); a harmless difference is the actual test.

2. A screenshot is not a superset of the text

This is the result that changed how I think about my own tool. Restricted to the seven defects that are visible in both views, and pooled over the seven models that ran both, the screenshot and the text view tie: 82.9% vs 82.2% caught on the single-page task. But they tie by failing on different things.

Typos: judging one page, the same seven models caught 69 of 70 typo twins from text and 51 of 70 from screenshots. GPT-6 Luna caught 3 of 10 in both screenshot tasks and 9 of 10 from the text view alone. Reading misspellings off pixels is still unreliable.
Blocking popup: 70 of 70 from screenshots, 40 of 70 from text. The popup is in the text view (as a dialog at the end of the page), so this is not a visibility problem. Gemini 3.7 Flash caught 0 of 10 popups from text alone and 10 of 10 from screenshots. Claude Haiku 4.5 often saw the dialog and still did not flag it; one of its replies reads: "the signup dialog that appears on load could be considered slightly intrusive but is dismissible."
The three visual-only defects are 0% from text, as designed, and 70% (low contrast), 80% (broken image) and 90% (cut-off text) from screenshots.

3. Images help strong models and hurt weak ones

On those same seven text-visible defects, the strongest models did better with a screenshot than with the text: Sonnet 5.5 97% vs 90%, GPT-6.1 Sol 97% vs 87%, Gemini 3.7 Flash 94% vs 80%. The two smallest did worse: Gemini 3.5 Flash-Lite 64% vs 76%, Claude Haiku 4.5 46% vs 64%. For a small model, an image is not extra information; it is a harder input to read the same information from.

4. Judging alone is a different skill, and the scores barely move

Without a reference, a vague headline is a matter of standards, and many models had none: Gemini 3.5 Flash-Lite, Claude Haiku 4.5 and Qwen 3 235B caught 0 of 10 vague headlines from text on their own, yet Qwen caught 10 of 10 when it could compare against the original.

The 1 to 10 "launch readiness" score each model also gives answers my second question. Strong models drop it hard on a broken page: Sonnet 5.5 gave originals 9.0 and twins 5.6 on average from screenshots, GPT-6.1 Sol 10.0 and 7.5. Weak ones barely move: Claude Haiku 4.5 gave 8.9 to originals and 8.6 to twins. A score from a weak grader looks just as confident on a broken page.

False alarms were rare for most models on the 20 clean pages per run (originals plus harmless variants), with two exceptions on the text view: DeepSeek-R1 flagged 6 of 20 clean pages and GLM-5 4 of 20, most often with an invented "contradiction" (7 of their 10 false labels). They are also the two models that wrote the longest replies (about 1,500 and 2,200 output tokens per page). Two models are not enough to call that a pattern, but it is the next thing I would test.

5. Price does not predict which cheap model is good

On pairs with screenshots, GPT-6 Luna scored 0.965 for about $0.21 and Gemini 3.7 Flash 0.99 for about $0.71, against 0.98 for about $4.12 (GPT-6.1 Sol) and 1.00 for about $4.44 (Claude Sonnet 5.5). Yet Gemini 3.5 Flash-Lite cost the same $0.21 as GPT-6 Luna and scored 0.635, and Claude Haiku 4.5 cost $0.92 and scored 0.455. "The cheaper model from the same family" ranged from nearly perfect to unreliable as a grader, so for a tool that runs on every deploy, a cheap model has to be tested on the actual task rather than picked by tier.

What this means for Shipgrade

A text-only grader is blind to three of these ten defects by construction, and that is the obvious part. The less obvious part is that switching to screenshots is not a free upgrade: it gains the visual defects and the popups, and gives back typos. The cheapest robust setup this data suggests is both views (a cheap vision model that tested well, such as GPT-6 Luna or Gemini 3.7 Flash here, on the screenshot, plus a text model on the text), and a pair against the last deploy when one exists.

Limitations

The pages are synthetic, with exactly one defect each and a clear control. Real pages have several overlapping problems and no clean original.
10 base designs means 10 samples per defect per model, so one miss moves a category by 10 points. Read the per-category numbers as coarse.
One run per model and one prompt wording. I did not measure run-to-run variance or prompt sensitivity.
The pairs-text task is saturated: three of ten defects are invisible in text, so 70% caught is the ceiling, and four models (Sonnet 5.5, GPT-6.1 Sol, Gemma 4 31B, GLM-5) hit it exactly.
Kaggle's overall leaderboard score averages the tasks each model ran, so the three text-only models are averaged over the two text tasks. Compare them on the text columns.
Grok could not be run at all, and gpt-oss-120b was dropped after two timeouts.
A small local pilot on Groq models was used only to debug the task code; none of the numbers here come from it.

What I would measure next

Subtler variants of the same defects (one typo instead of three, contrast at 2.5:1 instead of 1.25:1, a half-hidden button) to find where each model's threshold is.
Real pages with several defects at once, graded against human reviewers.
Repeated runs per model, to put error bars on these numbers.
Whether asking for less reasoning reduces the invented contradictions.
Mobile widths, where cut-off text and popups behave differently.

My Benchmark

Benchmark on Kaggle: kaggle.com/benchmarks/prasenx/pagetwins
Tasks: pairs, screenshots, pairs, text, single page, screenshot, single page, text
Dataset: kaggle.com/datasets/prasenx/pagetwins (120 pages with HTML, full-page screenshots and text views, CC BY 4.0)
Code, raw answers and charts: github.com/StarKnightt/pagetwins (MIT)

StarKnightt / pagetwins

PageTwins: a Kaggle benchmark of 100 one-defect landing page twins. Do AI graders notice what broke?

PageTwins

A Kaggle benchmark that asks a simple question: when a landing page breaks in one specific way, does an AI grader notice, name the problem, and leave a clean page alone?

Benchmark: https://www.kaggle.com/benchmarks/prasenx/pagetwins
Dataset: https://www.kaggle.com/datasets/prasenx/pagetwins

The dataset

10 original landing pages (SaaS, developer tool, e-commerce tool, consumer products, mobile apps, an online course, a local service) on four layouts Each one has 10 "evil twins" that are identical except for exactly one planted defect, plus two controls: an identical copy and a copy with a harmless change. That makes 120 pages, 100 defect pairs and 20 control pairs. Every brand, product, quote and price is fictional and was written for the dataset.

Defect	What changed in the twin	In the text view
`missing_cta`	every sign-up, buy or download button removed	visible
`hidden_price`	paid prices replaced with "Contact us" or similar	visible
`contradiction`	one claim changed so two parts of

…

View on GitHub

Waymark: a trail guide you listen to, written by Gemma 4 in your browser

StarKnight — Mon, 05 Oct 2026 22:35:41 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Most trail apps ask you to look at them. At a junction you take the phone out, wake the screen and find the blue dot, and for that moment you are looking at a map instead of the trail. This week's theme asks for builds where the screen is the shortest part of the experience, so I built a trail guide that you listen to.

Waymark turns a real trail into a short spoken guide. You pick a trail, or open a GPX file, and Waymark works out the facts from open data: where the junctions are and which way to go at each one, where the climbs start and how steep they are, where the water, bridges, steps and viewpoints are, and how long the walk should take. Gemma 4 E2B, an open-weight model running inside the browser, writes one or two plain sentences for each of those waymarks and a short briefing for the whole walk.

Before you leave, a 3D model of the real terrain previews the walk in about a minute, with every cue shown where it will be spoken. Then you send the guide to your phone and put the phone in your pocket. Pocket mode keeps the screen dark, follows GPS and speaks each cue when you reach that waymark, for example: "Continue straight onto Mist Trail. John Muir Trail on the right is not your route." On a two-hour walk, you look at the screen for about a minute.

It is for walkers and hikers on trails they do not know well, and for anyone who would rather keep their eyes on the path. A group can share one link.

Demo

Live app: https://starknightt.github.io/waymark/

Things to try:

Pick a demo trail. Mist Trail (Yosemite, USA), Lake Agnes Trail (Banff, Canada), Galu Devi to Triund (Himachal Pradesh, India) and the Pen y Fan and Corn Du circular walk (Wales, UK). Their guides were written by Gemma 4 E2B in Chrome and are saved with the site, so nothing large is downloaded.
Preview the walk. The camera flies along the route and slows at each waymark. Tick "Read the cues aloud" to hear the guide.
Show the facts behind each cue. Each cue then lists the facts it was checked against, and what the checker changed, if anything.
Walk it with your phone. Scan the QR code, then press "Try a simulated walk". It replays the route at eight times walking speed and speaks each cue as the simulated walker reaches it. "Start walk" uses real GPS.
Write a guide yourself. On a desktop with Chrome or Edge, press "Write it again on this device" on any demo trail, or "Make a guide for a trail" to search for a mapped route, paste an OpenStreetMap route link or open a GPX file. The first time, the browser downloads the model (2.0 GB) and keeps it for next time.

The phone view during a simulated walk. Each cue is also shown as text when it is spoken:

Gemma writing a guide for the Triund trail in the browser, with nothing mocked (sped up 2 times):

Each cue next to the facts it came from:

Code

StarKnightt / waymark

A trail guide you listen to. Gemma 4 runs in your browser and writes spoken cues for real trails from open map data.

Waymark

A trail guide you listen to. Pick a real trail. Waymark builds the facts from OpenStreetMap and open elevation data, and Gemma 4 E2B, running in your browser on WebGPU, writes a short spoken cue for every junction, climb and landmark. A 3D model of the real terrain previews the walk in about a minute. Then the phone goes in your pocket: it keeps the screen dark and speaks each cue when you reach that waymark.

Live: https://starknightt.github.io/waymark/

Built for the DEV Hacktoberfest Open-Source AI Challenge, Week 1 (Touch Grass). The repository was started on October 5, 2026, inside the challenge window.

What it does

Four demo trails that open instantly: Mist Trail (Yosemite, USA), Lake Agnes Trail (Banff, Canada), Galu Devi to Triund (Himachal Pradesh, India) and the Pen y Fan and Corn Du circular walk (Wales, UK). Their guides were written by Gemma 4 E2B in Chrome…

View on GitHub

TypeScript, Vite, Three.js and LiteRT-LM, under the MIT license. All of the code was written during the challenge window, starting on October 5.

How I Built It

Facts first, then the model

A spoken guide is worse than useless if it says left when the path goes right, and a 2B model cannot do geometry. So the work is split. Code builds the facts:

Junctions come from the OpenStreetMap path network. Wherever other paths meet the route, Waymark measures the bearing of every branch over about 22 m and decides whether the route turns left or right, keeps left or right at a fork, or continues straight, and which of the other ways are not the route. When two ways leave on the same side, it says which one to take.
Climbs and descents come from open elevation tiles: sections steeper than 7% that gain at least 40 m, with their length, gain, average grade and top elevation.
Landmarks near the path: drinking water, springs, waterfalls, viewpoints, summits, huts, bridges, flights of steps, lakes that come into view, and where the path enters or leaves woodland.
Walking time uses Tobler's hiking function on the elevation profile.

These are merged into waymarks, between 5 and 22 on the twelve test trails, each a short list of facts. The start, the finish and every junction where the route turns or moves onto a different path are marked as required. This is what Gemma sees for one waymark on the Mist Trail, wrapped here to fit:

w8 (REQUIRED): junction: continue straight onto "Mist Trail";
  "John Muir Trail" on the right is not your route

Gemma 4 E2B in the browser

Runtime: the JavaScript API of LiteRT-LM from Google AI Edge, on WebGPU. The model file is gemma-4-E2B-it-web.litertlm (2.0 GB). The browser downloads it once into its private file storage (OPFS), and after that it loads in a few seconds (2.3 s on my PC).
One constrained tool call: write_guide({ cues: [{ waymark, text }], briefing }). The waymark field is an enum of the real waymark IDs, so the model cannot invent a waymark, and constrained decoding guarantees the call can be parsed.
Rules in the prompt: use only the facts listed for that waymark, give directions exactly as written, one or two short sentences, and a cue for every required waymark.
Cost of a guide: about 1,400 prompt tokens and 700 output tokens, 18 seconds on my PC's RTX 4060.

A checker that knows what each number means

Every cue is checked against the facts of its own waymark before it is saved:

Numbers must exist in the facts with the same meaning. The words around a number decide its role: "gains 70 m" is a gain, "topping out at 1,980 m" is an elevation and "the next 450 m" is a length. Numbers written as words, such as "an hour and a half", are checked as well.
Names must exist on the route, including names in other scripts.
Directions must match the junction. A wrong side is swapped, an invented turn is removed, and a turn that went missing is restored from the facts. When a waymark holds two junctions close together, both turns are allowed, in order.
Features: a summit, waterfall or bridge can only be mentioned where the facts mention one.

A sentence that fails is removed. If nothing safe is left, the cue is rebuilt from the facts. The checker has 35 unit cases that run in Chrome, and most of them come from real mistakes.

What went wrong along the way

One call instead of many. My first design asked Gemma to call add_cue once per waymark and set_briefing once. It called set_briefing and stopped, in every test. A single write_guide call with an array fixed it.
Every number was right and the sentence was wrong. "Gain 730 meters in 1.5 km." Both numbers were in the facts: 730 m was the length of the climb and 1.5 km was its distance from the start. A checker that only looks for numbers passed it. That is why numbers now carry a role, and why the prompt no longer includes each waymark's distance from the start.
My first checker was wrong more often than the model. In its first full run over 12 trails it changed 12 cues. Four of those changes were right. Five turned a correct cue into a wrong one: on the Dipsea Trail, two junctions 40 m apart share a waymark ("turn left onto Hazel Avenue, then right onto Hale Lane"), and the checker "corrected" the second turn to match the first. Three more removed correct sentences, such as "Find the Sharma store on your right", rejected because "Find the Sharma" looked like an unknown place. Each of these is now a unit case.
Phrases copied between waymarks. Gemma sometimes repeated "the path straight ahead is not your route" at a junction with no such path. The checker now requires every side path it mentions to be listed in that junction's facts.
Map data needs care. I had mapped OpenStreetMap's wetland areas to water, so the bogs around Pen y Fan were drawn as lakes, and the Merced River's water area produced "a lake comes into view on your right" on the Mist Trail. Forest areas also run over 3,000 m ridges, so the 3D view applies a treeline estimated from latitude.
A colour space bug. THREE.Color converts colours to linear light, so my tree placement compared sRGB pixels with linear values and matched nothing. Every mountain was bare until I read the hex value directly.
Map servers. From my network only one of the three public Overpass servers answered, and slowly. The demo trails are built ahead of time from the OpenStreetMap API, the app tries the three servers in turn, and if none answers it still builds a guide from elevation alone.

Results

Twelve trails in seven countries, Chrome on an RTX 4060 desktop GPU, greedy decoding. The second column asks for the same guide as JSON in plain text, without constrained decoding.

	One constrained tool call (shipped)	JSON in plain text
Cues written by Gemma	180	180
Passed every check unchanged	171	170
Changed by the checker	9 (8 of them on Mount Takao)	10
Required waymarks covered	120 of 120	120 of 120
Output that could not be used	0	0
Time per guide	18.2 s	11.4 s
Decoding speed	44 tokens/s	80 tokens/s

Eight of the nine changes were on Mount Takao, where Gemma translated or romanised Japanese path names, for example "Takao Mountain Line" for 高尾山線. The checker replaced them with the names on the map and kept the turns.

The result I did not expect: Gemma 4 E2B followed the JSON format in plain text on every trail, so constrained decoding mainly bought certainty, at a cost of about 45% of the decoding speed. I kept it, because a guide for a trail nobody has tested must always parse. Most of the accuracy came from the facts and the checker rather than from the decoding method.

Two more checks. On the live site, in a fresh browser, the first guide took about 10 minutes, nearly all of it the 2.0 GB download on my connection; writing the guide itself took about 20 seconds. All 62 cues of the four demo guides were also checked line by line against their source facts during development. After the checker, none of them gives a wrong turn, distance or climb, though a few read awkwardly, such as "There is a bridge, Mist Trail.", where the bridge is mapped as part of the Mist Trail. One mistake the checker cannot see is in a briefing: the Pen y Fan briefing says the walk reaches the summit of Pen y Fan, while the map data puts the summit 250 m from the route.

What it cannot do yet

It is not a navigation or safety device. Carry a map and check conditions.
Writing a guide needs WebGPU and about 2 GB of storage, so in practice a desktop or laptop. Any phone can play the guide.
The cues are only as good as the map. An unmapped junction is missed, and English voices read names in other scripts, such as 高尾山線 on Mount Takao, poorly.
The elevation data has a resolution of roughly 10 to 30 m, so sharp summits read lower than their true height.
On a phone the page has to stay open with the screen awake for GPS and speech to keep working. Waymark keeps the screen dark to save power.

Why Does Open Innovation Matter?

Where you walk, and when, is personal. In January 2018, Strava's public heatmap of its users' activity revealed the locations and patrol routes of military bases. A trail plan says where you will be and when you will be away from home. With an open model running in the browser, the route, the guide and your position never go to an AI provider.

It has to work without signal. Trails are often where coverage fails. The guide is written before you leave and fits in a link of about 1.5 KB, so on the trail the phone needs neither a network nor a model.

I could control how the model writes, not only what I ask it. Constrained decoding with my own schema, where the only valid waymarks are the real ones, and a 12-trail evaluation that I could re-run as often as I liked, at no cost. Most of the mistakes described above were found by those repeated runs. With a paid API I would have run the evaluation less often and found fewer of them.

It is open all the way down. OpenStreetMap data, open elevation tiles, an Apache-2.0 model, an Apache-2.0 runtime and MIT code. Anyone can host Waymark as static files, swap in another model, or adjust the facts for the trails where they live.

The trade-off is a 2.0 GB first download and a desktop GPU to write guides. A larger hosted model would write more varied sentences, but for spoken directions being right matters more than style, and that comes from the facts and the checker, which work the same way with an open model on my own machine.

Prize Categories

Best Use of Gemma. Gemma 4 E2B is the only model in Waymark, and it runs entirely in the browser through LiteRT-LM on WebGPU. It writes every spoken cue and every briefing through a constrained tool call whose schema only allows real waymarks. Every demo guide on the site was written by it, and "Write it again on this device" runs it live.

Thanks for reading. If you try Waymark on a trail near you, I would like to hear how the cues held up.

I built my grandmother a tiny island that keeps track of her day, with Gemma running inside the browser

StarKnight — Sun, 04 Oct 2026 11:15:53 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

I built this for my grandmother.

Because of her age, she can't remember a lot of things these days, and keeping track of her daily routine, such as when to take which medicine, has become hard for her. I wanted to give her something calm and simple that shows her day and what is left to do.

Little World is a daily routine tracker shown as a small 3D island. You describe the routine in plain English. Gemma, an open model running inside the browser, turns the description into a schedule. Each task appears on the rim of the island at its time of day, and the sun and moon move across the sky with the real clock, so the current time and the next task are visible at a glance.

This is the example routine I used during development. It is made up, not hers:

Thyroid tablet, 50 mcg, when I wake up, on an empty stomach.
Blood pressure tablet, Amlodipine 5 mg, after breakfast.
Metformin 500 mg before lunch and before dinner.
Eye drops in both eyes at 4 PM.
Calcium tablet after dinner, not with the thyroid tablet.
A 20 minute walk at 6 PM.

When a task is due, its marker lights up and a reminder appears. Pressing Done marks it complete. The Ask tab answers questions such as "What's next?" or "Did I take my eye drops?" using only the routine that was entered.

The routine never leaves the device. There is no account, no server and no analytics, and after the first load the app works offline. For something as personal as a medicine list, that was a firm requirement.

Little World is not a medical tool, and the app says so. It only repeats what it was told. When I asked "Can I take calcium with my thyroid tablet?", it answered that it could not see that in its notes and that I should check with a doctor or pharmacist, which is the behavior I wanted.

Showing it to my grandmother

I showed Little World to my grandmother. Her reaction was "oh wow, this is so good". Then she said "you made something for me", and she was genuinely pleased.

Designed for an older user

After that, I focused on making it easier to use for someone her age:

"What do I do now?" A single button opens a full-screen view with one task in very large type, one large "I've done it" button with an undo option, and the next few tasks below. It can be set as the default screen.
Spoken reminders. Reminders can be read aloud with the browser's built-in speech synthesis, so no external voice service is involved. If a task is still not done, the reminder repeats after 20 and 40 minutes.
Setup by a family member. Gemma needs a computer with a capable GPU, but the person using the routine does not. A family member can enter the routine on a computer, select "Send to another device", and share the QR code or link. The routine is encoded in the part of the link after the #, which browsers do not send to a server. Opening the link on a phone or tablet shows the routine and asks for confirmation before using it.
Readability. Larger text throughout, buttons at least 48 pixels tall, labels that stay readable against the daytime sky, and task states written as words ("Now", "Waiting", "Done") rather than shown only by color. A reduced-motion setting turns on automatically when the device requests it.

Demo

Live app: https://starknightt.github.io/little-world/

"Try the demo island" works on any device, including phones. It loads a routine that Gemma already parsed, so nothing is downloaded. From there, "What do I do now?" opens the large view and "Send to another device" shows the QR code.
"Describe my own routine" requires desktop Chrome or another browser with WebGPU. Gemma 2 2B is downloaded once (about 1.4 GB) and stored by the browser. After that it loads in a few seconds and works offline.

The clip below shows the real model with nothing mocked: I enter a routine, Gemma parses it, I review the preview and save it, then ask two questions. The GIF runs at 2.5x speed; the 27-second video runs at 1.25x.

The same routine at night, and on a phone:

Code

StarKnightt / little-world

A daily routine tracker on a small 3D island. Gemma runs in the browser through WebGPU, and nothing leaves the device.

Little World

A daily routine tracker shown as a small 3D island. Describe a routine in plain English ("Thyroid tablet when I wake up, Metformin before lunch and dinner, eye drops at 4 PM"). Gemma, running entirely in the browser, turns it into a validated schedule. Each task is placed on the rim of the island at its time of day, and the sun and moon follow the real clock. Mark a task as done when it is complete. Ask "What's next?" and Gemma answers using only the routine.

Nothing leaves your device. No account, no server, no analytics. After the first load it works offline.

Live: https://starknightt.github.io/little-world/ (open ?demo for the instant demo, no download)

Day	Night	Phone

Big view	Family setup

Not medical advice. Little World only repeats what you told it. It does not know about medicines, doses or interactions, and it is told to send those…

View on GitHub

Built with Vite, TypeScript, Three.js, WebLLM and zod, under the MIT license. All code in the repository was written during the challenge weekend, starting October 4.

How I Built It

Stack

Gemma 2 2B (gemma-2-2b-it-q4f16_1-MLC), Google's open-weight model, quantized to 4 bits.
WebLLM from MLC, which runs the model on the GPU through WebGPU. It runs in a Web Worker so the 3D scene stays responsive while the model is working.
Three.js for the island, with a custom sky shader, bloom and HTML labels that avoid overlapping each other.
zod to validate every routine before it is saved, and vite-plugin-pwa for offline support.

Pipeline

routine text
  → Gemma 2 2B, constrained by a grammar
  → JSON, one object per task: quoted source text, name, dose, time anchor, food rule, icon
  → quote check (plain code)
  → zod validation (one retry if it fails)
  → time anchors converted to clock times
  → editable preview
  → saved routine

Key decisions

1. A grammar instead of asking for JSON. WebLLM can constrain decoding to a grammar. I wrote a small EBNF grammar for the exact routine format with fixed separators, so the model chooses only the content, never the structure. With it, every output in my test set was valid JSON.

2. The model chooses a meaning, and code computes the time. My first version asked Gemma for an hour and a minute, and it often returned 0:00 for "after breakfast". Now the model chooses from a fixed list of anchors (on_waking, after_breakfast, before_lunch, bedtime and a few others), and code converts them to times such as 8:30, 12:30 and 22:00. The model returns a clock time only when the text contains one, such as "at 4 PM". The food rule is derived from the anchor as well, so "before lunch" cannot become "after food".

3. Quote first, then verify. For each task the model must also copy the exact source text into a said field. A short function then checks that quote with regular expressions. If the quote names exactly one time or food rule and the model chose something else, the quote takes precedence. A quote such as "before lunch and before dinner" is split into two tasks, and a dose containing a number that does not appear in the source text is removed. This check produced the largest accuracy improvement, in about 100 lines of code.

Before anything is saved, the user sees a preview in which every time can be edited and every task can be removed. A 2B model will sometimes be wrong, so the person always has the final say.

Questions follow the same principle. Code first computes the facts: what is done, what is waiting and for how long, and what is next. Gemma only phrases those facts as a short answer, and it is instructed not to give medical advice.

Results

I wrote 20 test routines with known correct answers. They cover medicines, insulin, an inhaler, eye drops, plant care, feeding a pet, explicit clock times and descriptive wording such as "the white tablet with breakfast, the pink tablet at bedtime". I scored Gemma on all of them in headless Chrome on my PC's RTX 4060.

Gemma 2 2B	Valid JSON	Correct task count	Correct times	Correct food rule
Free text, no grammar	14/20 (17/20 after one retry)	13/20	38/54	16/21
Grammar, model alone	20/20	n/a	36/54	11/21
Grammar + quote check (shipped)	20/20	17/20	47/54 (87%)	17/21

Download: 1.4 GB, once, in 42 parts. On the live site, the first run including the download took about 7.7 minutes on my connection. After that, loading from the browser cache takes about 4 to 6 seconds.
Speed: about 40 to 55 tokens per second for generation and 500 to 700 tokens per second for reading the prompt. Short routines take 4 to 6 seconds. The six-line example above is 688 tokens in and 330 out, about 8 seconds.
Questions: answers take about 1.5 to 3 seconds.
Offline: on the live site with a fresh browser profile, I downloaded the model, created a routine, disconnected the network and reloaded. The app loaded from its own cache (0.9 MB), Gemma loaded from the browser cache, and a new routine was created in about 12 seconds.
First paint: under one second on the live site at desktop and phone sizes. The CSS is inlined into the page.
Automated checks: a headless test opens the live site at desktop and phone sizes, covers the large view, the share link round trip and a browser without WebGPU, and fails on any console error. All 28 checks pass.

What did not work

Gemma 3 1B was my first choice. At 537 MB it is a much smaller download. In the WebLLM build I used, its sliding-window attention configuration either failed to load or, once forced, produced unusable output. Without the grammar it produced no valid routines in 3 test cases. With the grammar it produced valid JSON for all 5 cases, but none of the 11 times were correct. I switched to Gemma 2 2B, at the cost of a download nearly three times larger.
JSON schema mode looped in early tests. With a plain JSON schema, the small models sometimes generated whitespace until they reached the token limit. The custom grammar with fixed separators avoids this. In the final evaluation, schema mode was also valid for all 20 cases.
Times were the hardest part. Asking for hours and minutes directly produced many wrong times, often midnight. The anchor list plus the quote check brought it to 47 of 54.
The model still misses tasks. In the example routine above, Gemma leaves out the walk. I kept that in the demo deliberately, because it shows why the editable preview matters. During the offline test it also produced a dose of "500 mg" for a Vitamin D tablet whose dose I had not written, which is why the dose rule exists.
A bug in my own regular expressions. I generated part of the quote checker with a script that turned every \b (word boundary) into a literal backspace character. Nothing matched, and accuracy dropped until I found it.
Testing a 1.4 GB model is slow. Each test with a fresh browser profile includes an eight-minute download. A force-closed Chrome once left the model cache half-written, and the app then waited indefinitely while checking the GPU. That check now times out after 3 seconds.
Not every device can run the model. It needs WebGPU with 16-bit float support and a few GB of GPU memory, and I only verified it on my own GPU. Phones get the full app with the demo routine and can open a routine sent from a computer. When the model cannot run, the first screen says so and suggests that option.

Why Does Open Innovation Matter?

A daily routine, especially one with medicines in it, is private. It shows what a person is treated for, how often, and how consistently they keep up. My grandmother should not have to send that to a company's server to get a reminder or an answer to "Did I take the evening one?"

Because Gemma's weights are open, the whole model can run inside the page. The routine, the completed tasks and the questions stay in one browser on one device. There is no API key to leak, no usage bill, and no dependency on a service that could change its terms or shut down. Once loaded, it keeps working without a network connection.

Open weights also meant I could control how the model behaves, not only what I ask it. WebLLM let me constrain decoding with my own grammar, which raised JSON validity from 14 out of 20 to 20 out of 20. I could switch from Gemma 3 1B to Gemma 2 2B with a one-line change when the first did not work, and I could run the 20-case evaluation repeatedly on my own GPU at no cost. With a closed API, I would have been limited to adjusting the prompt.

The trade-off is a 1.4 GB first download and a WebGPU requirement. For a health routine I think that is the right trade, and smaller open models will reduce it further.

Prize Categories

Gemma: Gemma 2 2B runs entirely in the browser through WebLLM and WebGPU and is the core of the app. It turns a plain-language description into the routine and answers questions using only that routine.

I built a 163-page paycheck calculator that costs $0 a month to run

StarKnight — Thu, 02 Jul 2026 07:39:03 +0000

Earlier this month I shipped PayBrackets, a free calculator that tells Americans what they actually keep from their paycheck after federal tax, state tax, and Social Security take their cut. It covers all 50 states plus D.C., it has no database, no backend, no APIs, and my total running cost is the domain.

I want to walk through how it's built, because the interesting part isn't the code. It's how much you can get out of very little code if the data underneath is right.

Why a tax calculator, of all things

Every month, millions of people type things like "75000 after taxes texas" or "25 an hour is how much a year" into Google. I looked at what they find. Mostly it's ten-year-old calculators buried under popups, or freshly generated AI pages where the numbers are confidently wrong.

That gap felt workable for one person. Salaries and taxes are pure math. The rules are public. Nothing about it needs a server, an account system, or content writers. It needs someone to sit down, get the rules right once, and present them clearly.

So the whole plan was: encode the 2026 tax rules properly, build one calculation engine, and let that engine generate every page.

The unglamorous part was the whole ballgame

There's no tax API behind the site. The source of truth is a set of TypeScript files I wrote by hand, encoding the 2026 federal brackets from the IRS, the Social Security wage base from the SSA, and every state's brackets, deductions, exemptions, and phaseouts from Tax Foundation data. It looks like this:

{
  name: "California",
  slug: "california",
  kind: "graduated",
  topRate: 0.133,
  brackets: {
    single: [
      { rate: 0.01, threshold: 0 },
      { rate: 0.02, threshold: 11_079 },
      // ...8 more brackets up to 13.3% at $1M
    ],
  },
  standardDeduction: { single: 5_540, married: 11_080 },
  payrollTaxes: [{ name: "CA SDI", rate: 0.013 }], // uncapped since 2024
}

Doing this for 51 jurisdictions took me a full day, and it was the most valuable day of the project. States are genuinely weird. Maryland's counties add their own income tax on top of the state's. California's disability insurance tax lost its wage cap in 2024, so it applies to every dollar. Connecticut quietly claws back the benefit of its lower brackets once you earn enough. Nine states don't tax wages at all.

If I had modeled states as "one tax rate" like most quick calculators do, every single page would be wrong in a different way. And where I chose not to model something, like those Maryland county rates, the page says so in a visible note instead of pretending.

One engine, checked against reality

The heart of the site is embarrassingly small. Progressive taxes are just a walk over brackets:

export function marginalTax(taxable: number, brackets: Bracket[]): number {
  let tax = 0;
  for (let i = 0; i < brackets.length; i++) {
    const { rate, threshold } = brackets[i];
    if (taxable <= threshold) break;
    const ceiling = i + 1 < brackets.length ? brackets[i + 1].threshold : Infinity;
    tax += rate * (Math.min(taxable, ceiling) - threshold);
  }
  return tax;
}

Around that sits the composition: federal income tax after the standard deduction, Social Security up to the wage cap, Medicare plus the surtax on high earners, optional 401(k) deferrals, then the state layer with all its exceptions.

Before launch I wrote a small check script with hand-verified salaries. It runs the engine against numbers I worked out independently and fails loudly if anything drifts. A typo in a bracket threshold is invisible in a code review. In a paycheck for a specific salary, it screams.

163 pages from one route

The site is Next.js, fully static. All the calculator pages hang off a single dynamic route, and the slugs are literally the questions people search:

export function allProgrammaticSlugs(): string[] {
  return [
    ...STATES.map((s) => `${s.slug}-paycheck-calculator`),      // 51 pages
    ...HOURLY_RATES.map((r) => `${r}-an-hour-is-how-much-a-year`), // 61 pages
    ...SALARY_AMOUNTS.map((a) => `${a}-after-taxes`),           // 35 pages
  ];
}

At build time, each page computes its own numbers. The "$50,000 after taxes" page shows the exact take-home in all 51 jurisdictions. The Texas page shows what no state income tax is actually worth against California at the same salary. No two pages share their content, which is the difference between programmatic pages that help people and the thin templated stuff Google buries.

The build produces 163 static pages in about half a minute, and the interactive calculator runs the same engine in the browser. Your salary never leaves the page, which was important to me. Nobody should upload their income to a stranger's server to learn what they already earn.

The readers that aren't people

A growing share of these questions never reach Google. People ask ChatGPT or Perplexity instead. So the site treats AI crawlers as an audience of their own.

Every page opens with the question stated plainly and a two-sentence answer with real numbers, before any explanation. There's FAQ markup throughout, a methodology page naming every data source, and a robots.txt that explicitly welcomes the OpenAI, Anthropic, and Perplexity crawlers rather than blocking them.

The bet is simple. Assistants quote pages that are structured like clean answers with checkable numbers. If an assistant answers "what's $70k after taxes in Florida" by citing my site, that citation is the new referral traffic.

The launch video is also code

One more thing that amused me. I wanted a proper product video for the launch, the smooth animated kind startups pay studios for. I don't own After Effects, so I built the video in Remotion instead. Every scene is a React component, the dollar figures in the animation come from the same tax engine as the site, and the final MP4 renders from the command line.

There's something satisfying about a project where even the marketing assets compile.

Where it stands

The site has been live for a few weeks. It's indexing, impressions are climbing, and the first ad review is in progress. I deliberately have no revenue story to tell yet, and I'd rather say that than inflate it. Once there are real numbers, good or bad, I'll write the follow-up.

The site is paybrackets.com if you want to poke at it. Happy to go deeper on the tax data modeling, the static generation, or the video in the comments.

DEV Community: StarKnight

A company brain that shows its work: multi-hop GraphRAG on FalkorDB, with every hop replayed in 3D

The data: an open-source organisation as a company

The graph model

The Cypher that does the work

Three FalkorDB details I learned the hard way

Typed tools first, free-form Cypher last

A verifier that uses the graph

Memory as graphs, one per visitor

Drawing the agent's reasoning

Acting, not only answering

Did the graph actually help?

The impact category, and a bug the eval found

Running on free tiers, and what it costs

Limitations

What I would do next

AI disclosure

Links

Cashcast: which invoices will be paid late? TabPFN on a laptop, checked month by month

What I Built

Demo

Code

StarKnightt / cashcast

Know which invoices will be paid late, and when the money will arrive. TabPFN on your laptop, client data stays local.

Cashcast

Demo mode

How I Built It

TabPFN does the predicting

Features from the day the invoice was sent

From a distribution to a decision

Did it work? A month-by-month backtest

Where TabPFN earns its place

Checking the replay itself

What it still cannot do

Why Does Open Innovation Matter?

Prize Categories

PageTwins: I Planted One Defect on 100 Landing Pages. Which AI Graders Noticed?

What I Benchmarked

Models Tested

Findings

1. The strong models do not need the "before" picture

2. A screenshot is not a superset of the text

3. Images help strong models and hurt weak ones

4. Judging alone is a different skill, and the scores barely move

5. Price does not predict which cheap model is good

What this means for Shipgrade

Limitations

What I would measure next

My Benchmark

StarKnightt / pagetwins

PageTwins: a Kaggle benchmark of 100 one-defect landing page twins. Do AI graders notice what broke?

PageTwins

The dataset

Waymark: a trail guide you listen to, written by Gemma 4 in your browser

What I Built

Demo

Code

StarKnightt / waymark

A trail guide you listen to. Gemma 4 runs in your browser and writes spoken cues for real trails from open map data.

Waymark

What it does

How I Built It

Facts first, then the model

Gemma 4 E2B in the browser

A checker that knows what each number means

What went wrong along the way

Results

What it cannot do yet

Why Does Open Innovation Matter?

Prize Categories

I built my grandmother a tiny island that keeps track of her day, with Gemma running inside the browser

What I Built

Showing it to my grandmother

Designed for an older user

Demo

Code

StarKnightt / little-world

A daily routine tracker on a small 3D island. Gemma runs in the browser through WebGPU, and nothing leaves the device.

Little World

How I Built It