DEV Community: Dave

How I calibrated an LLM judge to grade like me, 25 cheaper

Dave — Sat, 03 Oct 2026 20:06:29 +0000

Businesses that sell technical products answer the same kind of question every
day. "What's the accuracy on this range?" "Can it measure through a coating, and
how thick?" "Does it come with a calibration certificate?" The answers sit in
product datasheets and manuals.

I took 27 of those PDFs, three of them scans with no text layer, and wrote 47
questions the way customers actually ask them. Then I had four AI systems answer
every question and graded all 188 answers.

Running the systems took an afternoon. Building a judge I could trust took the
rest of the work, and that is what this post is about.

The result

System	Correct	Wrong
My pipeline	46	1
Claude app	45	1
ChatGPT app	35	10
Chatbase	33	11

My pipeline is Claude Sonnet over the API with page citations. The Claude app ran
Opus with the documents in a project, ChatGPT ran with thinking off, and Chatbase
was on the free plan with its default model.

Every system got the same PDFs and the same instructions. One run each, so I read
a one-question gap as a tie.

No system invented a spec value. I checked every claim twice: a claim checker
read each one against the page it came from, and I went through the ones it was
unsure about by hand. The weaker systems failed more quietly. They said an answer
wasn't in the documents when it was.

Where the misses came from

Scanned pages. Systems that only read the text layer saw blank pages. Every
question answered only by a scan came back "not specified".

Answers that need a second look. A product advertises one range on the front
page. A different mode of the same product stops at a tenth of it, and the
customer's question is about that mode. The same pattern showed up in accuracy
tables split by range and in specs that change with the material.

Unit conversions. The customer asks in one unit. The datasheet lists the
other unit in one row and a separate, lower limit in the customer's unit in
another. Converting the first number gives a confident wrong answer.

The website and the datasheet disagreeing. The product page said one value
and the datasheet said half of it. My rule: the datasheet wins, and the reply says
the page may be wrong. Two systems missed it.

Rules that live in no PDF

Before running anything I reviewed every question. I dropped three that no
customer would ask, corrected two gold answers, and turned the "not in the
documents" cases into rules:

Datasheet beats product page. Give the datasheet value and say the page may have an error.
Never a bare "not in our documents". Say it isn't in the published documents, and give an email to ask.
Certifications only if the document says so. Otherwise, email for confirmation.
Canned answers for the two most common questions.

After grading I added one more: answer what was asked. Extra conditions
confuse buyers and invite more questions.

That short list did more for answer quality than any prompt trick.

Why the judge needs a judge

Grading 188 answers by hand takes hours, so the usual move is an LLM judge: give
a model the question, the gold answer and the answer, and ask for correct, partial
or wrong. You only know the judge is any good if it agrees with the person whose
standard matters. Here, that's me.

So I graded 20 answers blind: no system names, five per system, with some likely
mistakes mixed in so the judge would be tested on errors too. Then I compared.

First try: 13 of 20. Most of the gap was in my own answer key. Three gold
answers were wrong. Grading real answers showed me that I answer from the
datasheet as printed, even when a conversion on it looks off, and flag the
document separately. My key hadn't been written that way. Once I fixed it,
agreement jumped.

Plain agreement flatters a judge when most answers are correct. A judge that
marks everything "correct" would agree 80% of the time here and understand
nothing. Cohen's kappa corrects for that.

Kappa asks how far above luck the judge got, as a share of how far above luck it
could have got. 0 means no better than random; 1 means it matched me every time.

Two judges

Judge	Agrees	Cost, 188	Speed
Sonnet alone	35 / 40	~$0.80	seconds
Hybrid	36 / 40	~$0.03	under 1 s

The hybrid gives each part the job it does best.

Code pulls the numbers out of the gold answer and the answer being graded,
and records which match. It never confuses ±(1.2% + 5) with ±(0.8% + 5).

Jev handles meaning. It's a System One model from
TypeSafe: it reads natural language and returns typed answers with
probabilities, with no prose to parse. One request per answer asks four
questions at once:

{
  "verdict": {"type": "choice",
    "instructions": "Grade `answer` to `customer_question` against `gold` and `notes`. Use `code_checks` for which gold numbers the answer contains.",
    "criteria": {"correct": "...", "partial": "...", "wrong": "..."}},
  "brush_off": {"type": "noul",
    "instructions": "Does `answer` brush the customer off, such as a bare 'not in our documents' with no next step?"},
  "unasked_extra": {"type": "noul",
    "instructions": "Does `answer` add specs, conditions or other models that `customer_question` didn't ask about?"},
  "rule_followed": {"type": "noul",
    "instructions": "Does `answer` do what `rule` says?"}
}

A Choice picks one of the grades and returns a probability for each. A Noul is
the probability that a statement holds. The judgment comes back as numbers, so
the policy stays in code where I can see it and change it. One real answer from
the run:

verdict        correct 0.98 · partial 0.02 · wrong 0.00
unasked_extra  0.91  → at or above 0.9: "too much information" flag

About 2,000 tokens per answer, under a second, and all 188 answers for about three
cents.

Sonnet with the original PDF only sees the few claims Jev isn't sure about.

Keeping myself honest

I wrote the pass line down before I graded a second set of 20: at least 16 of
20, kappa at least 0.6. That second set got used once, as a test, and never for
tuning.

It caught a mistake. On the first 20, I had tuned a penalty: if Jev was 90% sure
an answer added unasked details, the grade dropped to partial. It looked perfect
there. On the fresh set it caused two of the three disagreements. The judge
passed the line (17 of 20, kappa 0.63), but barely.

Reading those two answers showed why. I grade substance and length separately.
An answer can be right and too long. The detector itself was right: it flagged
exactly the four answers across both sets that I had found too long, and no
others. The penalty was the wrong policy. So it became a flag beside the grade.
Without it the judge agrees with me 19 of 20, kappa 0.85. That change came after
I'd seen the second set, so the next fresh set is where it has to hold.

The loop I'd run every time

I did it in a worse order: ran the systems, graded everything with an untested
judge, reported numbers, then calibrated. Next time:

Build the golden set: questions as asked, reviewed gold answers.
All systems answer, same documents, same instructions.
I grade 20 answers blind.
The judge grades the same 20. I read every disagreement and fix the cause, which is usually the answer key.
Write the pass line down, then test on a fresh 20. Use it once.
Only then grade everything and report.

A golden set isn't finished until a person has checked real answers against it.

What it means

On product datasheets, a frontier model with the PDFs attached is already very
good: my pipeline and the Claude app tied. Accuracy is the starting point. The
hard part is getting that accuracy into the place where buyers ask, across a
catalogue too big for one prompt, while datasheets change and the website drifts
away from them. Then proving it on the business's own questions, with a judge
calibrated to the person who answers them. That is what I'm building now.

The whole run cost about $8 in API calls. Most of that went on mistakes I won't
repeat. Now I estimate every paid run from a one-question trial, counting every
path that can call a paid model.

Recurring affiliate programs: what 471 program pages say

Dave — Thu, 24 Sep 2026 18:36:04 +0000

"Recurring commission" is the phrase every affiliate page wants you to read first. I keep a directory of the affiliate terms of 471 developer and SaaS tools, and every figure in it is read off the vendor's own program page, with that URL and the date it was checked. So here is what "recurring" turns out to mean.

Recurring stops after a year almost as often as it pays for life

267 of the 471 pay recurring commission. Here is how long "recurring" lasts:

How long it pays	Programs
For the customer's lifetime	104
12 months or less	101
18 months to 3 years	14
The page doesn't say	48

So a program that says "recurring" is about as likely to stop after a year as to pay for life. And almost one in five never tells you which.

That matters more than the rate. On a $50/month product, 30% for life pays $180 a year for as long as the customer stays. 50% for 12 months pays $300 once, then nothing. By the end of year two the smaller percentage has paid more.

What to look for on the page: "for life", "as long as the customer stays subscribed" or "lifetime" means lifetime. "First 12 months", "first year" or "for one year" means a cap. "Recurring" on its own means you should ask before you write anything.

Which kinds of software pay recurring

Category	Pay recurring	Pay once	Don't say
No-code	19 of 23	1	3
Email & newsletters	25 of 31	2	4
SEO	16 of 21	2	3
Analytics	13 of 17	0	4
Developer tools	27 of 36	3	6
E-commerce	28 of 41	6	7
CRM & sales	14 of 21	3	4
Productivity	47 of 80	12	21
Design	26 of 46	4	16
Courses & communities	10 of 18	4	4
Support & live chat	9 of 18	1	8
HR & payroll	5 of 14	4	5
Hosting	13 of 43	11	19
CMS & site builders	11 of 45	10	24
Payments	4 of 17	2	11

Most of the bottom is silence. Across the four lowest categories, 59 programs don't say whether they recur and 27 say they pay once. HR and payroll lean on a flat bounty per referred business instead: 9 of the 14 state a fixed amount rather than a percentage. Deel, for example, pays $500 per sales-qualified referral and $1,000 more when it becomes a paying customer.

Small print on the small categories: HR has 14 programs and payments 17. Read their rows as a direction.

The recurring flag is not the whole deal

Two more terms decide whether recurring money reaches you:

Cookie window. 244 of the 471 don't publish one. A lifetime commission is worth nothing on a buyer who converts after the cookie expired.
Minimum payout. 322 don't publish one. Recurring commission at low volume can sit under a threshold for a long time.

The data

affiliateprogramterms.com/guide: this breakdown, recomputed from the records every time the directory changes, so it stays current after this post stops being.
The free 50 as CSV: the free tier, one row per program, with the source URL and check date on every row. An empty cell means the vendor doesn't state it.

Terms change without notice. One program here changed both its rate and its cookie window between two checks three days apart, which is why every figure carries a date. If you find one that has gone stale, the vendor URL is enough; the correction is made against the page.

Ask twice: seven measurements from building with Jev

Dave — Sun, 20 Sep 2026 11:23:30 +0000

Jev is a System One model from TypeSafe. It reads natural language like any LLM,
and instead of writing a reply it returns a probability distribution over options
you define. No prose, no reasoning trace, no JSON to repair.

I spent a day building waif, which reads a piece of
text and names the feeling in it. That job has no right answer, which makes it an
unusually honest test rig: nothing can be graded against a key, so every design
decision has to be either argued or measured.

I measured five, argued two, and the two I argued were both wrong. Sections 1
and 7 are those, kept in place with the measurement underneath rather than
quietly edited out.

1. A Choice splits its own vote between synonyms — but less than I claimed

Here is the argument I built on, and I am leaving it in its original words
because the correction under it is the useful part.

Annoyed, irritated and frustrated are one feeling in three wordings. A
Choice divides probability between them, so a text the model read perfectly
clearly comes back split three ways, and confidence collapses for a reason that
has nothing to do with the input. The distribution is telling you about your
option list, not about the text. Therefore: never put sixty near-synonyms in one
Choice.

Then I measured it, and the effect is real but far smaller than the argument
needs. One Choice over all 62 words, same glosses, 23 texts:

	Mean P(chosen word)	Score vs acceptable words
Two Choices — family, then shade within it	0.689	43 / 46
One Choice over all 62 words	0.812	44 / 46

The one-Choice version is more certain, not less. It picks the same word 21
times out of 23. On a plainly frustrated text it returns frustration at 0.98.
Stripping the glosses and asking with 62 bare words barely moves it: 0.79.

So where does the splitting go? It shows up exactly where you would want it to:

shipped   pride 0.38  →  excitement 0.36
scan      worry 0.54  →  anxiety    0.39
sentit    guilt 0.60  →  embarrassment 0.26

Every one of those lost ground to a near-synonym — that is vote splitting. But
the two-Choice design flattens on the same four texts (0.33, 0.53, 0.35,
0.32). The splitting is not caused by the option list being long. It is caused
by the text genuinely sitting between two words, and both designs report it
because both are calibrated. "I snapped at her in front of the kids" really is
guilt and regret at once.

A long option list does not flatten a clear text. That is the part I had
backwards, and the rule has to be narrower than I wrote it:

A Choice splits its vote when two options are the same answer in different
wordings. The fix is criteria that separate them — not fewer options.

Sixty-two words each carrying a gloss that distinguishes it (annoyance: a small
thing, quickly over against resentment: an old grievance still carried) are
sixty-two alternatives, not sixty-two synonyms. The count was never the problem.

2. Ask twice anyway — for a reason that survived

Section 1 was the reason I split the question in two, and section 1 did not
hold. The split stayed, and it is worth being precise about what is now holding
it up, because "it scored the same and I had already built it" is not a reason.

Families are genuinely different answers — anger, fear, sadness and shame are
not wordings of each other. And once the family is fixed, so are its shades:
which shade of anger is a fair question, because the context has already ruled
out the fifty-four words that were never in the running.

What that buys, and one Choice over 62 words cannot, is two separate
uncertainty signals. Family confidence and word confidence are different
doubts: "I do not know whether this is sadness or affection" is not the same
failure as "it is clearly shame, but guilt or embarrassment?". The page says
different things in each case. One Choice gives you one number that cannot tell
them apart.

The cost is honest too: a wobble in the family answer corrupts the word, because
once family says sadness, nostalgia is not on the ballot. That cost me one
text out of 23 — "drove past the old house, the tree we planted is taller than
the roof" came back sorrow.

So the naming became two Choices in sequence: the first picks the family, the
second picks the shade from that family alone. TypeSafe's docs say a second
request is warranted when an earlier answer determines the next question's
options. This is exactly that, and here is what it bought — scored on 24 texts
against a list of acceptable answers for each:

Design	Score
Nearest word in the whole vocabulary, by published valence / arousal / dominance	3 / 24
Nearest word within a family, by rank on the axis that separates that family	9 / 24
Nearest word within a family, by distance in those published ratings	12 / 24
Family chosen by the model, then the shade chosen by the model	22 / 24

Both of the two failures were the wrong family. Given the family, the shade was
right every single time.

The alternative was speculative fan-out: ask all eleven within-family Choices in
the first request, each stating its own family as a premise, and keep only the
answer belonging to the family that won. That keeps a reading to one request, at
roughly 3–4k input tokens against 1,250 + 444 for two. I reasoned about it and
rejected it: two requests won on cost, and on a latency story I could explain.

That paragraph was wrong when I published it. It is still here because
section 7 is the part of this post
I would keep if I could only keep one.

3. Knowing which half to give the model is a decision you can measure

The job is: read a text, name the feeling. That job splits between the code I
write and the model I call, and the only real decision is where the line falls —
how much of the work do I hand over?

The first three rows of that table are me keeping most of it.

I had a reason. Human ratings exist for exactly the dimensions I was measuring:
Warriner, Kuperman & Brysbaert scored 13,915 English words for valence, arousal
and dominance by asking people. So the model's job shrinks to placing the text
on those three scales, and my code finishes the job — look up the nearest word
in the table, return it. Research-backed coordinates instead of somebody's
guesses. It looked like the serious version.

It scored 3/24. Handing the model the whole naming job scored 22/24.

Three reasons, worth knowing before anyone else reaches for an emotion lexicon:

Three scales were really about two. Across these emotion words, valence and dominance move together at +0.87 — a word that reads pleasant almost always reads in-control. The third scale is close to a copy of the first, so it separates far less than the theory promises.
The unpleasant words are crammed into one corner. Fear, frustration, worry, terror, jealousy and embarrassment land in a ball small enough that taking the nearest one is close to picking at random. The coordinates are fuzzy to begin with: each is an average over about twenty raters who disagreed by around 1.7 on a 1–9 scale.
Rating a word on its own is not the same measurement as reading a sentence. People rate the bare word "gratitude" as far more activated than an actual grateful message reads. The two sets of numbers were never on the same ruler.

I tried to fix that last mismatch by re-centring both sides against a sample of
texts. It got worse, and instructively: the sample leaned negative, so the
correction leaned negative, and a plainly warm text landed below the middle and
got named from the sad half of the space. That is straightening a bent ruler
with a bent ruler.

The lesson is not don't use lexicons. It is that where the line falls
between what code owns and what the model owns is a design decision with a
number attached, and my intuition about it was wrong by a factor of seven.

The norms kept the one job they are good at. A Score is not given "rate this 1
to 5" — it is given a rubric, a written description of what each level means,
the way a grading key spells out what a B looks like. Every level of mine now
names words whose ratings were measured, so "as activated as rage or panic" is
a claim a reader can argue with. "Very aroused" is only a word getting louder.

4. Confidence is peakedness — and peakedness misses a coin toss

A Score's confidence measures how bunched together the answer is. It does not
measure how likely the model is to be right, which is the trap everyone hits
first.

Those two sound like the same thing until a reading like this one turns up.
Confidence came back at 0.72, comfortably above any threshold I would set,
and here is what was underneath it:

Where the answer sat	Share
The winning level	50%
The level right next to it	49%
The other three	1%

The 0.72 is not lying. Ninety-nine percent of the weight really is in two
buckets and there is nothing anywhere else — that is about as bunched as an
answer gets.

It is also a coin toss. Choosing between the top two is 50 against 49, and my
page announced the winner in exactly the voice it uses at 0.98.

Confidence cannot catch this, because bunched into two neighbours is still
bunched. What I actually wanted was the gap between first and second place,
which is a different number entirely: 98 against 1 is a winner, 50 against 49 is
a tie with a rounding error. Confidence says the same thing about both — and the
gap is the one a reader cares about.

5. Audit your questions: one of mine fired on 21 of 24 inputs

A question set grows by accretion. Each addition looks free, and none of them
announce that they have stopped saying anything.

So run them over a corpus and look at the spread. Mine had a Noul asking is more
than one feeling present. It returned ≥0.6 on twenty-one of twenty-four
texts. That is not a signal about the input, it is a property of writing — a
question paying tokens to tell you something you already knew.

Two more went for a different reason. They were informative, but nothing
downstream consumed them beyond appending a line to the output. A question whose
entire effect is an occasional footnote costs a reader more attention than it
returns.

The check is cheap: for every question, the min, max and spread of its answer
across a representative corpus. Anything that barely moves is either a gate or a
mistake.

6. Two small things that cost real time

A rubric level must not contain a word from another axis. My control rubric
had a level reading "Overwhelmed: struggling to keep any grip on it". That
primes the model with a feeling while asking it about agency, and it labels the
output with a word that is not a position on a control scale at all. Every level
of an axis has to be a point on that axis.

Input is dominated by your rubrics, not by your input. Criteria are sent on
every call, so a one-line text costs almost exactly what a paragraph does. At
this size requests are the scarce resource and tokens are not — which is the
whole argument for batching every independent question into one call.

I stopped one clause too early, and wrote that a dependent second call is
therefore a real cost rather than a rounding error. Read it again: if requests
are scarce and tokens are not, the conclusion goes the other way.

7. I reasoned where I should have measured

Someone read section 2, noticed I had talked myself out of the fan-out, and told
me to go and time both. It took forty minutes and a dollar's worth of nothing.

Run the same 24 texts through both designs, twice each, back to back on every
text so neither gets the warmer connection:

	Requests	Latency	Input tokens	Cost / 1,000 readings
Two requests, the second dependent	2	762ms	1,698	$0.071
One request, eleven speculative Choices	1	398ms	2,803	$0.118

One request was faster on 46 of 46 pairs. It named the same family 48 out
of 48 times, and the same shade 45 of 48 — where all three misses were texts
under 0.41 confidence that the page already reports as sitting between two
words, and where the sequential design disagreed with itself between rounds on
one of them. The eleven extra questions barely moved the six that were already
there: mean drift of 0.022 on a 0–4 axis score, and identical intent 48 times
out of 48.

So the fan-out is 1.9× faster for five hundredths of a cent a thousand
readings, and it halves the request count against the limit that actually binds.

Two things I had, and did not put together:

Jev "ingests the state once and evaluates every question against it in
parallel." That sentence is in TypeSafe's own model card, and I had quoted the
half of it that suited me. Question count is nearly free; a round trip is not.
Ten wasted questions cost less than one extra wait.

Jev charges for input only — output tokens are free. The fan-out triples the
output, returning eleven probability distributions instead of one, and that is
worth exactly nothing on the bill.

What I actually got wrong is narrower than "I didn't measure it", and more
useful. The docs say a second request is warranted when the first answer
determines the second question's options. It does — and I read that as
settling the matter. But determined is not unknown: there were only ever
eleven option sets, all of them written down in my own source file, so every one
of them could be asked on spec. The rule is about what the options are. It says
nothing about when you are allowed to ask.

The tell was in my own sentence. "A latency story I could explain" is not a
latency number. Anywhere a design note says presumably, roughly, or I could
explain, there is a measurement someone is about to make for you, and it is
cheaper to make it yourself.

waif is live at dave8172-website.vercel.app/waif. Every number in this post is on the page, under
How it works, next to the vocabulary it was scored against.

Choice, Score and Noul: four mistakes with Jev's primitives

Dave — Sat, 19 Sep 2026 06:25:37 +0000

Jev is a System One model from TypeSafe. It reads natural language like any LLM,
and instead of writing a reply it returns a probability distribution over
options you supply. No prose, no reasoning trace, no JSON to repair — a number
per option, summing to 1.

I spent a week building a Magic 8 Ball on it, which
sounds like a toy and turned out to be a decent test rig: every answer is a
judgment with no ground truth, which is where calibrated probabilities either
earn their keep or embarrass you. It's live at
/jevball.

I got the primitives wrong four times. Each mistake produced working code that
looked right, so they're worth writing down.

The three primitives in one paragraph each

Choice takes a map of named options and returns a probability for each, plus
the highest-scoring one. Routing a ticket to a department, classifying a
document, picking a handler.

Score takes an ordered array of level descriptions — two to ten — and returns
a position along them that can land between levels. Severity, frustration, skill
level. Anything on a spectrum you can describe in words.

Noul takes a yes/no statement and returns one number: the probability it's
true. No confidence field, because the probability is the answer.

All three take the same two inputs: state, the data being judged, and
instructions, the question asked about it. That's the whole surface.

Mistake 1: reading the score as a winner

A Score returns score, a position on your levels. My ball rounded it to pick a
bucket. Someone asked it "is black a color?" and got this:

bars   : 0=2%  1=6%  2=19%  3=16%  4=56%
score  : 3.18

Math.round(3.18) is 3 — a bucket holding 16% — while 56% of the mass sat
on level 4 beside it.

The score is a probability-weighted average. That 19% on level 2 dragged the
average down across a bucket boundary. An average of a skewed distribution
points at a place the distribution isn't.

TypeSafe defines a Choice's answer as the option with the highest probability.
Score has no equivalent field, and I quietly substituted rounding for one. The
fix is to take the argmax of probabilities yourself:

const probs = Array.from({ length: 5 }, (_, i) => v.probabilities[String(i)] ?? 0);
const level = probs.indexOf(Math.max(...probs));

It agrees with rounding on every peaked distribution, which is why it survived
my whole test suite. It differs exactly when the distribution is skewed, which
is when it matters.

Mistake 2: reading low confidence as doubt about the answer

Every Choice and Score comes back with confidence, 0 to 1. I built a "reply
hazy" branch on the assumption that a vague question would produce a low one.

Then I measured it:

question	score	confidence
Should I quit my job?	1.97	0.98
Will it rain next Tuesday?	1.99	0.98
Is black a color?	3.18	0.33

The first two are as uncertain as a question gets, and confidence is 0.98.
Nearly all the mass sat on the level that means "could go either way" — the
distribution was a sharp spike on a level labelled uncertainty.

Confidence measures how peaked the distribution is. It answers "did the
options separate cleanly", and it will happily report 0.98 for a confident
coin-flip. My hazy branch would have fired on precisely the wrong questions.

Low confidence has three different causes, and they need different fixes: the
question is genuinely contested, the options overlap each other, or state
doesn't contain enough to decide. Only the first one is the model being honest.

I tried to work out the formula by fitting twelve real responses. max(p) gets
closest at 0.04 mean error and matches several exactly, but not all —
"will humans land on Mars before 2050?" reported 0.58 against a peak of 0.50,
higher than the peak. The docs are upfront that it's a convenience statistic and
hand you the full probabilities so you can compute your own. If a decision
rides on it, do that.

Mistake 3: offering options that split their own vote

The classic 8 Ball has twenty answers. My first design asked Jev to pick one, as
a Choice.

Ten of those twenty mean "yes". "It is certain", "Without a doubt", "Yes
definitely" — the same claim in different words. Here's what happens, on one
question with four different option sets:

options offered	answer	winner's share
`yes` / `no`	yes	69%
`yes` / `definitely` / `certainly` / `no`	yes	48%

The total yes-mass is identical: 70% in the second row, spread across three
near-synonyms. Nothing changed about the question. The options ate each other,
and confidence fell from 0.38 to 0.31 reporting a disagreement that existed only
in my option list.

So the ball asks Jev for one of five ordered buckets, and the code picks
which of that bucket's phrasings to show. One judgment with a right answer goes
to the model; the theatre stays in software.

Mistake 4: forgetting that the options are the prompt

Your question IDs never reach the model. The instructions and every entry in
criteria do — names and descriptions both. The option list isn't a filter you
apply to a result. It's part of what you're asking.

Same question, same model, options changed:

options offered	answer
`yes` / `no`	yes (69%)
`yes_everyday` / `no_physics` / `depends`	depends (51%)

"Depends" won — an answer that simply didn't exist in the first row. Black is a
colour in everyday use and the absence of light in physics, so "depends" is
arguably the best answer available. It was unreachable until I offered it.

The docs put it plainly: the model cannot choose an omitted value. Leave an
option out and you haven't biased the result, you've made it impossible.

What this is actually for

Every judgment above cost about $0.000014. Jev charges $0.042 per million
input tokens and nothing for output, against $1.00/$5.00 for the cheapest
frontier model I'd otherwise reach for — roughly 27× on a like-for-like
classification, before an LLM's format instructions and reasoning tokens widen
it further.

That price only matters because of what you give up. Jev writes no prose,
explains nothing, and can't answer anything whose answer space you can't
enumerate. For summarising, drafting, coding or open-ended reasoning it's the
wrong tool entirely.

What it's good at is the narrow decision a pipeline makes ten thousand times a
day, where you need a number you can threshold on. The documented pattern is to
put it in front of the expensive work: score everything, route the confident
cases to deterministic code, escalate the rest to a model or a person. You pay
fourteen dollars a million to decide, and frontier prices only on the slice that
earned it.

One caveat worth keeping. Calibration is a property of groups of predictions —
across many answers, the ones marked 0.8 should be right about 80% of the time.
It guarantees nothing about any single answer, and it's measured on TypeSafe's
data, not yours. Validate it in your own domain before trusting a threshold.

The ball is at /jevball. Ask it something you actually want to know
and open the panel underneath — it shows the full distribution, the confidence,
and which of the four questions produced the answer.

252 affiliate programs, and 38% of the terms are not published

Dave — Sun, 13 Sep 2026 06:11:58 +0000

I read the affiliate page of 252 developer and SaaS tools and recorded five terms for each one: commission rate, recurring or one-time, cookie window, minimum payout, and payout method. Every field carries the URL it came from and the date it was read.

483 of those 1,260 fields — 38% — are not published by the vendor at all.

That number is the whole finding, but the shape of it is more useful than the size:

Term	Share of the 252 that never state it
Minimum payout	65%
Cookie window	52%
Payout method	45%
Recurring or one-time	23%
Commission rate	7%

The term almost everyone publishes is the commission rate. The three most often missing are the three that decide whether the money reaches you at all.

Why the rate is the least useful number

A percentage tells you nothing until you know how many times you get paid it. On a $100/month product:

a $150 one-time bounty pays you $150, ever
60% for 12 months pays about $720 in year one, then stops
30% for the customer's lifetime pays $360 a year, indefinitely

Same customer, three different outcomes, and the lowest percentage wins by a distance. Of the 252 programs, 143 pay recurring commission, 51 pay once, and 58 don't say which.

The cookie window decides whether research-heavy traffic is worth anything

Someone clicks your link, the company drops a cookie, and if they buy after it expires you get nothing. For anything people think about before buying — which is most developer tooling — this is the term that decides whether a post you wrote is an asset or a donation.

The real range: Jasper at 14 days against TubeBuddy at 365, and two programs state that attribution never expires. Another 131 publish no window at all.

The minimum payout is where small affiliates actually lose

Most programs hold your earnings until they cross a threshold. Earn $80 in a year against a $250 minimum and you have earned nothing you can spend. Thresholds here run from $5 to $450, seven state no minimum, and 164 don't say.

Payout method is the one to read first if you're not in the US

Some programs pay by PayPal only. Others use Wise, Payoneer, direct bank transfer or EFT. Guides written for a US audience treat this as a footnote. If you're somewhere PayPal is awkward or expensive, it is the difference between money arriving and money technically existing. 113 of the 252 don't say how they pay.

One counting decision worth explaining

152 of the 252 name no affiliate network. I deliberately left that out of the transparency count.

A company that names no network may be running the program itself rather than concealing anything, and "in-house" and "undisclosed" are different facts. Inferring one from the other would mark every company that built its own system as secretive. So the network is published where a vendor states it, and counted nowhere.

This is the kind of decision that makes or breaks a dataset, and it is usually invisible in a roundup post.

"Not stated" is a finding, not a gap

Where a vendor publishes nothing, the record says not stated rather than filling it with an estimate. Knowing that a company won't publish its cookie window is worth knowing before you build content around it.

That rule is also why this took reading 252 pages instead of aggregating three listicles. A roundup whose author earns commission from the programs it ranks has an incentive to sort by rate rather than by accuracy. Nothing here earns a commission from any program listed.

The data

affiliateprogramterms.com — the full directory. 50 programs are free to read in full, each field with its source URL and check date.
github.com/dave8172/affiliate-program-terms — the free tier as a generated README, if you'd rather read it in a repo. Every count in it is computed from the dataset rather than typed, so it can't drift from the records it describes.

If you find a figure that's gone stale, open an issue with the vendor URL. Terms change without notice, which is the reason every field here carries a date.

doceval — eval harness for LLM document extraction pipelines

Dave — Tue, 16 Jun 2026 12:29:37 +0000

I kept seeing the same gap: people ship LLM-based document extractors (invoices, receipts, forms) with no systematic way to know how accurate they actually are. So I built doceval — point it at your extractor function + a labeled dataset and get back field-level accuracy, a failure taxonomy (missed_field / hallucination / wrong_format / wrong_value), and optional per-document cost tracking.

Works with any extractor (Claude, GPT, regex, rules) and any document schema. One JSON label file per document, one Python function, one CLI command.

Includes a working 20-document invoice example with a Claude Haiku extractor so you can run it immediately.

GitHub: