DEV Community: Dave

Local vs remote MCP servers: how each connects, step by step

Dave — Mon, 05 Oct 2026 20:36:01 +0000

An MCP server reaches your AI app in one of two ways. A local server is a program on your own machine. A remote server lives at a URL and signs you in with OAuth. I wanted to know exactly what happens in each case, so I tested it: a local server (doceval's) started by Claude Code, and the three remote servers I use (Upwork, Notion and Todoist). Where something comes from the MCP spec instead of a test, I link the spec.

For what MCP is and how one tool call works, see my previous post.

Local: a child process on two pipes

The app starts the server as a program on your machine and talks to it through two pipes: it writes requests into the server's input and reads replies from its output. I ran doceval's server under Claude Code (claude -p with a test config) and checked the process list every quarter of a second:

Session starts. Claude Code starts the server as its own child process, once, even when the prompt uses no tools.
Every call reuses it. A session that called the tool twice started the server once.
Session ends. Claude stops the server before exiting itself.
Crash. After kill -9 on Claude, which allows no clean-up, the server was gone within two seconds. Its input pipe closed, and it read that as "the session is over".

Each session starts its own copy, so three open sessions run three copies of every local server.

Credentials come from the environment. The MCP spec says local (stdio) servers should take credentials from the environment instead of using OAuth. In practice that's the env block of the app's MCP config, a plain-text file. Claude Code fills in ${VAR} from your shell: a config with "SECRET": "${MY_TEST_SECRET}" handed the server my shell's value, so the secret doesn't have to sit in the file.

Remote: five parties

Who	In my case
You	the person signing in
Browser	where you type your password
Client	Claude Code
MCP server	`mcp.upwork.com/mcp`: runs the tools, checks tokens
Sign-in server	issues tokens. The spec calls it the authorization server

The MCP server and the sign-in server can belong to the same company and still be separate roles. Todoist's MCP server lives at ai.todoist.net, and its sign-in server at todoist.com.

Remote: the first connection, step by step

1. Claude calls the server with no token, and gets a 401. I sent all three servers a request with no token. Each one answered 401 Unauthorized, with a www-authenticate header pointing to a public file. Upwork's points to mcp.upwork.com/.well-known/oauth-protected-resource/mcp.

2. That file names the sign-in server. For Upwork it's https://mcp.upwork.com, and for Todoist it's https://todoist.com.

3. The sign-in server's own public file lists its addresses. These are where you log in and where codes get swapped for tokens. Upwork's login page and token address are on www.upwork.com. All three list PKCE with S256 (more in step 5).

4. Claude identifies itself with a URL. In my stored credentials, the app ID Claude used with all three servers is the same: https://claude.ai/oauth/claude-code-client-metadata. That's a public description of the Claude Code app, which the sign-in server fetches. The spec prefers this method and marks the older automatic registration as deprecated. The description says "token_endpoint_auth_method": "none": Claude Code has no app secret, and every install shares one public ID.

5. Claude makes a one-time secret: PKCE. With no app secret, something else has to prove that the app finishing the login is the one that started it. Claude makes a random verifier, keeps it, and puts only its hash (the challenge) into the login link.

6. You log in on the service's own page. Claude opens your browser at the sign-in server's login page. Per the spec, the link carries the app ID, the return address (Claude's description lists http://localhost/callback), the challenge, and which server the token is for. Your password goes only to the service. Claude never sees it.

7. A one-time code comes back through your browser. After you approve, the sign-in server sends your browser to Claude's local return address with a short-lived, single-use code. The spec also requires the client to check that the response came from the sign-in server it expected.

8. Claude swaps the code for tokens. It sends the code plus the verifier from step 5. The sign-in server hashes the verifier and compares it with the challenge from step 6. Only a match gets tokens, so a stolen code is useless without the verifier. Back come an access token (short-lived, used on every request) and a refresh token (used only to get new access tokens).

9. Claude stores both. On my machine they're in ~/.claude/.credentials.json, an owner-only file, with fields for each server: accessToken, refreshToken, expiresAt, clientId, issuer. My MCP config file holds only each server's type and URL. The tokens are still on disk, readable by anything running as me, but they expire, work for one server, can be revoked, and never contained my password.

Remote: every request after that

The token travels on every request, as Authorization: Bearer <access token>. The spec requires it on every HTTP request, requires the server to check that the token was issued for it, and forbids passing tokens on to anyone else.

How a server checks a token depends on the kind. There are two:

Opaque: a random string that means nothing by itself. The server looks it up in its own database, or asks the sign-in server (a standard call named introspection).
Signed (a JWT): the token carries its own contents and a signature, and the server checks it with a public key.

All three of my access tokens are opaque to me: single strings, not the three dot-separated parts of a JWT, and none of the three sign-in servers publishes a list of public keys. How each service checks its tokens happens on its side, out of view. Notion's sign-in server does publish an introspection address, the standard way to look a token up.

How a signed token is checked (tested locally)

None of my three servers uses JWTs, so I built one on my machine to see how the checking works. A JWT is three parts joined by dots: header.contents.signature.

Signing (by the sign-in server): hash header.contents (anyone can compute a hash; no key needed), then transform the hash with the private key. The result is the signature.
Checking (by the MCP server): open the signature with the public key and compare it with your own hash of the text you received. If they match, the token is genuine and unchanged.
My results: the real token passed. Changing the user ID in the contents failed. Re-signing it with a different key failed too.
Which public key: the header names the key (kid), and the sign-in server publishes its public keys at a fixed address. Several keys can be published at once, so an old key keeps working while tokens signed with it expire. A careful MCP server fetches keys only from the sign-in server it already trusts, never from an address written inside the token.
One key, many tokens: a key pair usually signs tokens for every user for a long time, until it's rotated. What's new on each login is the token.

A JWT is readable by anyone who holds it. The contents are only encoded, so signing stops tampering without hiding anything. Secrecy comes from HTTPS and from where the token is stored.

Refresh, revoke, and no sessions

Refresh: when the access token expires, Claude sends the refresh token to the sign-in server and gets a new access token, with no login.
Revoke: all three sign-in servers publish a revocation address, and a service can also cancel tokens on its own side. After that the next refresh fails, and you're back at step 1.
No sessions: the 2026-07-28 spec retired the initialize handshake and the Mcp-Session-Id header. Each request now carries its own protocol version and client details, so any copy of a server can answer it. The state lives at the edges: tokens on your disk, and token records at the sign-in server.

What the tokens can't protect against

If an attacker steals a sign-in server's private signing key, they can make tokens the MCP server will accept. That happened in 2023, when a group Microsoft calls Storm-0558 forged tokens with a stolen Microsoft signing key to read email. OAuth limits the damage of a leak, with short-lived tokens, tokens bound to one server, revocation and key rotation, but a breach of the service itself is beyond what any token can fix.

Who does what

Piece	Made by	Kept by
App ID (a URL)	the app's maker	public
PKCE verifier	Claude	Claude, until step 8
One-time code	sign-in server	passes through your browser, used once
Access token	sign-in server	Claude, sent on every request
Refresh token	sign-in server	Claude, plus a record at the sign-in server
Your password	you	the sign-in server only
Local server secrets	you	the server's environment

Building and using MCP servers: 4 things I learned

Dave — Mon, 05 Oct 2026 15:38:55 +0000

I use Upwork's official MCP server from Claude Code, and doceval, my open-source eval tool, ships an MCP server of its own. This is how MCP works underneath, and four things I learned along the way. Where I tested something, I say how.

MCP in one paragraph

MCP (Model Context Protocol) is an open standard for connecting AI apps to outside tools and data. A service wraps itself once as an MCP server, and any app that speaks MCP (Claude, ChatGPT, Cursor and many more) can connect to it without a custom integration.

If every app had to write its own integration for every service, 20 apps and 1,000 services would need 20,000 integrations. With MCP each side builds its half once: 20 clients plus 1,000 servers, 1,020 in all. It's the USB-C trick: one port, any charger. Like USB-C, the port says nothing about how good the charger is.

Three roles

Host: the app running the model, like Claude Code, ChatGPT or an agent you build yourself.
Client: a connector inside the host, one per server. It passes JSON back and forth.
Server: wraps one service and offers what it can do as tools.

Usually the host's model decides which tool to call. The server can be plain code (get a request, do the thing, return the result) or run its own model inside. The host can't tell the difference: a request goes in, a result comes out.

I saw this while building upsweep, an open-source agent skill that finds Upwork jobs through Upwork's official MCP server. Claude, running inside Claude Code, reads Upwork's tool list, decides what to call and makes sense of what comes back. My own subscription pays for that model. Upwork pays nothing for it.

One call, step by step

Connect, once. claude mcp add --transport http upwork https://mcp.upwork.com/mcp. Upwork's server needs an account, so the first request is refused with "sign in first", the client finds Upwork's login server, and I sign in with OAuth. The token works only for that server. A server running locally on your own machine usually needs no sign-in.
Discover. The client asks tools/list. The server replies with each tool's name, a plain-English description and the shape of its inputs (a JSON Schema).
Choose. I say "find LLM eval jobs". The model reads the descriptions, picks the search tool and fills in the inputs.
Call. The client sends tools/call with the tool name and arguments. Upwork runs the search.
Use the result. The result goes into the model's context, and the model writes me an answer.

The messages are JSON-RPC: a method name plus parameters.

{
  "jsonrpc": "2.0",
  "id": 7,
  "method": "tools/call",
  "params": {
    "name": "find_jobs",
    "arguments": { "query": "LLM evaluation" }
  }
}

(Argument names simplified.)

1. The description is the whole interface

Step 3 surprised me most. In a normal agent loop, no code decides which tool gets called. The model decides, by reading names and descriptions. It never sees your code.

So write each description the way you'd brief a new colleague: what the tool does, when to use it, what comes back. Tool picking gets harder as the list grows and descriptions overlap. How many is too many depends on the model and the host. PagerDuty's engineers call 20 to 25 the sweet spot, and their own server ships over 20. doceval's needs two. One scores a single extraction against the expected values, the other runs a full eval over a labelled dataset.

Keep results short too. A result goes into the model's context and usually stays there for the rest of the conversation. A tool that returns 50 pages makes the agent slower, dearer and more likely to lose the thread.

2. A local server lives and dies with your session

Upwork's server runs on Upwork's machines. A local server is different: a program on your own computer that the app starts for you. I wanted to know exactly when it starts and stops, so I tested it with doceval's server in Claude Code: a test config passed to claude -p, and the process list checked every quarter of a second.

It starts once, when the session starts. My prompt used no tools, and the server still started, exactly once.
It is the app's child process. The server's parent process was Claude.
It is reused for every call. In a session that called doceval's tool twice, the server started once.
It stops when the session ends. Claude shut it down before exiting itself.
It stops after a crash too. I killed Claude with kill -9, which gives it no chance to clean up. The server was gone within two seconds.

The link between them is a pair of pipes: the app writes requests into the server's input and reads replies from its output. When I closed a running server's input by hand, it exited on its own. That's how it learns the session is over, whether the app quit cleanly or crashed.

Between calls the server just waits, and each session starts its own copy. On a small machine, three open sessions means three copies of every local server in memory.

3. Your MCP config can hold secrets in plain text

A local server often needs a password or an API key, and the usual place for it is the env block in the app's MCP config. That's a plain-text file. Anything running as your user can read it, including a coding agent with file access, and a project's config file can end up in git.

Two things that help:

Reference the secret instead of writing it. Claude Code fills in ${VAR} from your environment. I tested it with a config file passed to claude --mcp-config: a config with "SECRET": "${MY_TEST_SECRET}" handed the server the value from my shell. The config can then be shared, and the secret lives in one place outside it.
Prefer servers that sign you in. Remote servers that use OAuth keep no secret in the config. My entries for Upwork, Notion and Todoist hold a type and a URL, and nothing else.

Stronger options exist, like the operating system's keychain or a secret manager that starts the server for you. I haven't tested those yet, so I'll stop at naming them.

4. What a tool returns is untrusted, so approve the actions

A job post, an email or a web page can contain text written to steer the model. That's prompt injection, and whatever a tool returns goes into the model's context.

So I approve every Upwork proposal myself. Upwork's MCP has a preview built in: the agent shows me the proposal, I confirm, and the preview expires after 15 minutes. Claude Code also asks before running a tool, unless you've allowed it.

I got this wrong at first. I shipped upsweep with a rule that said "drafts only", because I read Upwork's terms as forbidding agent submission. They allow submitting, one approval per proposal, never scripted around. I corrected it in public: Can an AI agent submit Upwork proposals?

Checklist

Describe each tool like a brief to a colleague, with no two that overlap.
Return the smallest result that answers the question.
Expect one copy of every local server per open session.
Keep secrets out of the config file: reference them with ${VAR}, or use servers that sign you in.
Treat what a tool returns as untrusted, and approve anything that changes something.

The model never sees your code, only your tool names, descriptions and schemas. Write them like they are the code.

The tools in this article

doceval: measures extraction accuracy field by field, with an MCP server for coding agents. pip install "doceval[mcp]"
upsweep: finds Upwork jobs that match your filters, through Upwork's official MCP. npx skills add dave8172/upsweep

Both are free and MIT licensed.

How I calibrated an LLM judge to grade like me, 25 cheaper

Dave — Sat, 03 Oct 2026 20:06:29 +0000

Businesses that sell technical products answer the same kind of question every
day. "What's the accuracy on this range?" "Can it measure through a coating, and
how thick?" "Does it come with a calibration certificate?" The answers sit in
product datasheets and manuals.

I took 27 of those PDFs, three of them scans with no text layer, and wrote 47
questions the way customers actually ask them. Then I had four AI systems answer
every question and graded all 188 answers.

Running the systems took an afternoon. Building a judge I could trust took the
rest of the work, and that is what this post is about.

The result

System	Correct	Wrong
My pipeline	46	1
Claude app	45	1
ChatGPT app	35	10
Chatbase	33	11

My pipeline is Claude Sonnet over the API with page citations. The Claude app ran
Opus with the documents in a project, ChatGPT ran with thinking off, and Chatbase
was on the free plan with its default model.

Every system got the same PDFs and the same instructions. One run each, so I read
a one-question gap as a tie.

No system invented a spec value. I checked every claim twice: a claim checker
read each one against the page it came from, and I went through the ones it was
unsure about by hand. The weaker systems failed more quietly. They said an answer
wasn't in the documents when it was.

Where the misses came from

Scanned pages. Systems that only read the text layer saw blank pages. Every
question answered only by a scan came back "not specified".

Answers that need a second look. A product advertises one range on the front
page. A different mode of the same product stops at a tenth of it, and the
customer's question is about that mode. The same pattern showed up in accuracy
tables split by range and in specs that change with the material.

Unit conversions. The customer asks in one unit. The datasheet lists the
other unit in one row and a separate, lower limit in the customer's unit in
another. Converting the first number gives a confident wrong answer.

The website and the datasheet disagreeing. The product page said one value
and the datasheet said half of it. My rule: the datasheet wins, and the reply says
the page may be wrong. Two systems missed it.

Rules that live in no PDF

Before running anything I reviewed every question. I dropped three that no
customer would ask, corrected two gold answers, and turned the "not in the
documents" cases into rules:

Datasheet beats product page. Give the datasheet value and say the page may have an error.
Never a bare "not in our documents". Say it isn't in the published documents, and give an email to ask.
Certifications only if the document says so. Otherwise, email for confirmation.
Canned answers for the two most common questions.

After grading I added one more: answer what was asked. Extra conditions
confuse buyers and invite more questions.

That short list did more for answer quality than any prompt trick.

Why the judge needs a judge

Grading 188 answers by hand takes hours, so the usual move is an LLM judge: give
a model the question, the gold answer and the answer, and ask for correct, partial
or wrong. You only know the judge is any good if it agrees with the person whose
standard matters. Here, that's me.

So I graded 20 answers blind: no system names, five per system, with some likely
mistakes mixed in so the judge would be tested on errors too. Then I compared.

First try: 13 of 20. Most of the gap was in my own answer key. Three gold
answers were wrong. Grading real answers showed me that I answer from the
datasheet as printed, even when a conversion on it looks off, and flag the
document separately. My key hadn't been written that way. Once I fixed it,
agreement jumped.

Plain agreement flatters a judge when most answers are correct. A judge that
marks everything "correct" would agree 80% of the time here and understand
nothing. Cohen's kappa corrects for that.

Kappa asks how far above luck the judge got, as a share of how far above luck it
could have got. 0 means no better than random; 1 means it matched me every time.

Two judges

Judge	Agrees	Cost, 188	Speed
Sonnet alone	35 / 40	~$0.80	seconds
Hybrid	36 / 40	~$0.03	under 1 s

The hybrid gives each part the job it does best.

Code pulls the numbers out of the gold answer and the answer being graded,
and records which match. It never confuses ±(1.2% + 5) with ±(0.8% + 5).

Jev handles meaning. It's a System One model from
TypeSafe: it reads natural language and returns typed answers with
probabilities, with no prose to parse. One request per answer asks four
questions at once:

{
  "verdict": {"type": "choice",
    "instructions": "Grade `answer` to `customer_question` against `gold` and `notes`. Use `code_checks` for which gold numbers the answer contains.",
    "criteria": {"correct": "...", "partial": "...", "wrong": "..."}},
  "brush_off": {"type": "noul",
    "instructions": "Does `answer` brush the customer off, such as a bare 'not in our documents' with no next step?"},
  "unasked_extra": {"type": "noul",
    "instructions": "Does `answer` add specs, conditions or other models that `customer_question` didn't ask about?"},
  "rule_followed": {"type": "noul",
    "instructions": "Does `answer` do what `rule` says?"}
}

A Choice picks one of the grades and returns a probability for each. A Noul is
the probability that a statement holds. The judgment comes back as numbers, so
the policy stays in code where I can see it and change it. One real answer from
the run:

verdict        correct 0.98 · partial 0.02 · wrong 0.00
unasked_extra  0.91  → at or above 0.9: "too much information" flag

About 2,000 tokens per answer, under a second, and all 188 answers for about three
cents.

Sonnet with the original PDF only sees the few claims Jev isn't sure about.

Keeping myself honest

I wrote the pass line down before I graded a second set of 20: at least 16 of
20, kappa at least 0.6. That second set got used once, as a test, and never for
tuning.

It caught a mistake. On the first 20, I had tuned a penalty: if Jev was 90% sure
an answer added unasked details, the grade dropped to partial. It looked perfect
there. On the fresh set it caused two of the three disagreements. The judge
passed the line (17 of 20, kappa 0.63), but barely.

Reading those two answers showed why. I grade substance and length separately.
An answer can be right and too long. The detector itself was right: it flagged
exactly the four answers across both sets that I had found too long, and no
others. The penalty was the wrong policy. So it became a flag beside the grade.
Without it the judge agrees with me 19 of 20, kappa 0.85. That change came after
I'd seen the second set, so the next fresh set is where it has to hold.

The loop I'd run every time

I did it in a worse order: ran the systems, graded everything with an untested
judge, reported numbers, then calibrated. Next time:

Build the golden set: questions as asked, reviewed gold answers.
All systems answer, same documents, same instructions.
I grade 20 answers blind.
The judge grades the same 20. I read every disagreement and fix the cause, which is usually the answer key.
Write the pass line down, then test on a fresh 20. Use it once.
Only then grade everything and report.

A golden set isn't finished until a person has checked real answers against it.

What it means

On product datasheets, a frontier model with the PDFs attached is already very
good: my pipeline and the Claude app tied. Accuracy is the starting point. The
hard part is getting that accuracy into the place where buyers ask, across a
catalogue too big for one prompt, while datasheets change and the website drifts
away from them. Then proving it on the business's own questions, with a judge
calibrated to the person who answers them. That is what I'm building now.

The whole run cost about $8 in API calls. Most of that went on mistakes I won't
repeat. Now I estimate every paid run from a one-question trial, counting every
path that can call a paid model.

Recurring affiliate programs: what 471 program pages say

Dave — Thu, 24 Sep 2026 18:36:04 +0000

"Recurring commission" is the phrase every affiliate page wants you to read first. I keep a directory of the affiliate terms of 471 developer and SaaS tools, and every figure in it is read off the vendor's own program page, with that URL and the date it was checked. So here is what "recurring" turns out to mean.

Recurring stops after a year almost as often as it pays for life

267 of the 471 pay recurring commission. Here is how long "recurring" lasts:

How long it pays	Programs
For the customer's lifetime	104
12 months or less	101
18 months to 3 years	14
The page doesn't say	48

So a program that says "recurring" is about as likely to stop after a year as to pay for life. And almost one in five never tells you which.

That matters more than the rate. On a $50/month product, 30% for life pays $180 a year for as long as the customer stays. 50% for 12 months pays $300 once, then nothing. By the end of year two the smaller percentage has paid more.

What to look for on the page: "for life", "as long as the customer stays subscribed" or "lifetime" means lifetime. "First 12 months", "first year" or "for one year" means a cap. "Recurring" on its own means you should ask before you write anything.

Which kinds of software pay recurring

Category	Pay recurring	Pay once	Don't say
No-code	19 of 23	1	3
Email & newsletters	25 of 31	2	4
SEO	16 of 21	2	3
Analytics	13 of 17	0	4
Developer tools	27 of 36	3	6
E-commerce	28 of 41	6	7
CRM & sales	14 of 21	3	4
Productivity	47 of 80	12	21
Design	26 of 46	4	16
Courses & communities	10 of 18	4	4
Support & live chat	9 of 18	1	8
HR & payroll	5 of 14	4	5
Hosting	13 of 43	11	19
CMS & site builders	11 of 45	10	24
Payments	4 of 17	2	11

Most of the bottom is silence. Across the four lowest categories, 59 programs don't say whether they recur and 27 say they pay once. HR and payroll lean on a flat bounty per referred business instead: 9 of the 14 state a fixed amount rather than a percentage. Deel, for example, pays $500 per sales-qualified referral and $1,000 more when it becomes a paying customer.

Small print on the small categories: HR has 14 programs and payments 17. Read their rows as a direction.

The recurring flag is not the whole deal

Two more terms decide whether recurring money reaches you:

Cookie window. 244 of the 471 don't publish one. A lifetime commission is worth nothing on a buyer who converts after the cookie expired.
Minimum payout. 322 don't publish one. Recurring commission at low volume can sit under a threshold for a long time.

The data

affiliateprogramterms.com/guide: this breakdown, recomputed from the records every time the directory changes, so it stays current after this post stops being.
The free 50 as CSV: the free tier, one row per program, with the source URL and check date on every row. An empty cell means the vendor doesn't state it.

Terms change without notice. One program here changed both its rate and its cookie window between two checks three days apart, which is why every figure carries a date. If you find one that has gone stale, the vendor URL is enough; the correction is made against the page.

Ask twice: seven measurements from building with Jev

Dave — Sun, 20 Sep 2026 11:23:30 +0000

Jev is a System One model from TypeSafe. It reads natural language like any LLM,
and instead of writing a reply it returns a probability distribution over options
you define. No prose, no reasoning trace, no JSON to repair.

I spent a day building waif, which reads a piece of
text and names the feeling in it. That job has no right answer, which makes it an
unusually honest test rig: nothing can be graded against a key, so every design
decision has to be either argued or measured.

I measured five, argued two, and the two I argued were both wrong. Sections 1
and 7 are those, kept in place with the measurement underneath rather than
quietly edited out.

1. A Choice splits its own vote between synonyms — but less than I claimed

Here is the argument I built on, and I am leaving it in its original words
because the correction under it is the useful part.

Annoyed, irritated and frustrated are one feeling in three wordings. A
Choice divides probability between them, so a text the model read perfectly
clearly comes back split three ways, and confidence collapses for a reason that
has nothing to do with the input. The distribution is telling you about your
option list, not about the text. Therefore: never put sixty near-synonyms in one
Choice.

Then I measured it, and the effect is real but far smaller than the argument
needs. One Choice over all 62 words, same glosses, 23 texts:

	Mean P(chosen word)	Score vs acceptable words
Two Choices — family, then shade within it	0.689	43 / 46
One Choice over all 62 words	0.812	44 / 46

The one-Choice version is more certain, not less. It picks the same word 21
times out of 23. On a plainly frustrated text it returns frustration at 0.98.
Stripping the glosses and asking with 62 bare words barely moves it: 0.79.

So where does the splitting go? It shows up exactly where you would want it to:

shipped   pride 0.38  →  excitement 0.36
scan      worry 0.54  →  anxiety    0.39
sentit    guilt 0.60  →  embarrassment 0.26

Every one of those lost ground to a near-synonym — that is vote splitting. But
the two-Choice design flattens on the same four texts (0.33, 0.53, 0.35,
0.32). The splitting is not caused by the option list being long. It is caused
by the text genuinely sitting between two words, and both designs report it
because both are calibrated. "I snapped at her in front of the kids" really is
guilt and regret at once.

A long option list does not flatten a clear text. That is the part I had
backwards, and the rule has to be narrower than I wrote it:

A Choice splits its vote when two options are the same answer in different
wordings. The fix is criteria that separate them — not fewer options.

Sixty-two words each carrying a gloss that distinguishes it (annoyance: a small
thing, quickly over against resentment: an old grievance still carried) are
sixty-two alternatives, not sixty-two synonyms. The count was never the problem.

2. Ask twice anyway — for a reason that survived

Section 1 was the reason I split the question in two, and section 1 did not
hold. The split stayed, and it is worth being precise about what is now holding
it up, because "it scored the same and I had already built it" is not a reason.

Families are genuinely different answers — anger, fear, sadness and shame are
not wordings of each other. And once the family is fixed, so are its shades:
which shade of anger is a fair question, because the context has already ruled
out the fifty-four words that were never in the running.

What that buys, and one Choice over 62 words cannot, is two separate
uncertainty signals. Family confidence and word confidence are different
doubts: "I do not know whether this is sadness or affection" is not the same
failure as "it is clearly shame, but guilt or embarrassment?". The page says
different things in each case. One Choice gives you one number that cannot tell
them apart.

The cost is honest too: a wobble in the family answer corrupts the word, because
once family says sadness, nostalgia is not on the ballot. That cost me one
text out of 23 — "drove past the old house, the tree we planted is taller than
the roof" came back sorrow.

So the naming became two Choices in sequence: the first picks the family, the
second picks the shade from that family alone. TypeSafe's docs say a second
request is warranted when an earlier answer determines the next question's
options. This is exactly that, and here is what it bought — scored on 24 texts
against a list of acceptable answers for each:

Design	Score
Nearest word in the whole vocabulary, by published valence / arousal / dominance	3 / 24
Nearest word within a family, by rank on the axis that separates that family	9 / 24
Nearest word within a family, by distance in those published ratings	12 / 24
Family chosen by the model, then the shade chosen by the model	22 / 24

Both of the two failures were the wrong family. Given the family, the shade was
right every single time.

The alternative was speculative fan-out: ask all eleven within-family Choices in
the first request, each stating its own family as a premise, and keep only the
answer belonging to the family that won. That keeps a reading to one request, at
roughly 3–4k input tokens against 1,250 + 444 for two. I reasoned about it and
rejected it: two requests won on cost, and on a latency story I could explain.

That paragraph was wrong when I published it. It is still here because
section 7 is the part of this post
I would keep if I could only keep one.

3. Knowing which half to give the model is a decision you can measure

The job is: read a text, name the feeling. That job splits between the code I
write and the model I call, and the only real decision is where the line falls —
how much of the work do I hand over?

The first three rows of that table are me keeping most of it.

I had a reason. Human ratings exist for exactly the dimensions I was measuring:
Warriner, Kuperman & Brysbaert scored 13,915 English words for valence, arousal
and dominance by asking people. So the model's job shrinks to placing the text
on those three scales, and my code finishes the job — look up the nearest word
in the table, return it. Research-backed coordinates instead of somebody's
guesses. It looked like the serious version.

It scored 3/24. Handing the model the whole naming job scored 22/24.

Three reasons, worth knowing before anyone else reaches for an emotion lexicon:

Three scales were really about two. Across these emotion words, valence and dominance move together at +0.87 — a word that reads pleasant almost always reads in-control. The third scale is close to a copy of the first, so it separates far less than the theory promises.
The unpleasant words are crammed into one corner. Fear, frustration, worry, terror, jealousy and embarrassment land in a ball small enough that taking the nearest one is close to picking at random. The coordinates are fuzzy to begin with: each is an average over about twenty raters who disagreed by around 1.7 on a 1–9 scale.
Rating a word on its own is not the same measurement as reading a sentence. People rate the bare word "gratitude" as far more activated than an actual grateful message reads. The two sets of numbers were never on the same ruler.

I tried to fix that last mismatch by re-centring both sides against a sample of
texts. It got worse, and instructively: the sample leaned negative, so the
correction leaned negative, and a plainly warm text landed below the middle and
got named from the sad half of the space. That is straightening a bent ruler
with a bent ruler.

The lesson is not don't use lexicons. It is that where the line falls
between what code owns and what the model owns is a design decision with a
number attached, and my intuition about it was wrong by a factor of seven.

The norms kept the one job they are good at. A Score is not given "rate this 1
to 5" — it is given a rubric, a written description of what each level means,
the way a grading key spells out what a B looks like. Every level of mine now
names words whose ratings were measured, so "as activated as rage or panic" is
a claim a reader can argue with. "Very aroused" is only a word getting louder.

4. Confidence is peakedness — and peakedness misses a coin toss

A Score's confidence measures how bunched together the answer is. It does not
measure how likely the model is to be right, which is the trap everyone hits
first.

Those two sound like the same thing until a reading like this one turns up.
Confidence came back at 0.72, comfortably above any threshold I would set,
and here is what was underneath it:

Where the answer sat	Share
The winning level	50%
The level right next to it	49%
The other three	1%

The 0.72 is not lying. Ninety-nine percent of the weight really is in two
buckets and there is nothing anywhere else — that is about as bunched as an
answer gets.

It is also a coin toss. Choosing between the top two is 50 against 49, and my
page announced the winner in exactly the voice it uses at 0.98.

Confidence cannot catch this, because bunched into two neighbours is still
bunched. What I actually wanted was the gap between first and second place,
which is a different number entirely: 98 against 1 is a winner, 50 against 49 is
a tie with a rounding error. Confidence says the same thing about both — and the
gap is the one a reader cares about.

5. Audit your questions: one of mine fired on 21 of 24 inputs

A question set grows by accretion. Each addition looks free, and none of them
announce that they have stopped saying anything.

So run them over a corpus and look at the spread. Mine had a Noul asking is more
than one feeling present. It returned ≥0.6 on twenty-one of twenty-four
texts. That is not a signal about the input, it is a property of writing — a
question paying tokens to tell you something you already knew.

Two more went for a different reason. They were informative, but nothing
downstream consumed them beyond appending a line to the output. A question whose
entire effect is an occasional footnote costs a reader more attention than it
returns.

The check is cheap: for every question, the min, max and spread of its answer
across a representative corpus. Anything that barely moves is either a gate or a
mistake.

6. Two small things that cost real time

A rubric level must not contain a word from another axis. My control rubric
had a level reading "Overwhelmed: struggling to keep any grip on it". That
primes the model with a feeling while asking it about agency, and it labels the
output with a word that is not a position on a control scale at all. Every level
of an axis has to be a point on that axis.

Input is dominated by your rubrics, not by your input. Criteria are sent on
every call, so a one-line text costs almost exactly what a paragraph does. At
this size requests are the scarce resource and tokens are not — which is the
whole argument for batching every independent question into one call.

I stopped one clause too early, and wrote that a dependent second call is
therefore a real cost rather than a rounding error. Read it again: if requests
are scarce and tokens are not, the conclusion goes the other way.

7. I reasoned where I should have measured

Someone read section 2, noticed I had talked myself out of the fan-out, and told
me to go and time both. It took forty minutes and a dollar's worth of nothing.

Run the same 24 texts through both designs, twice each, back to back on every
text so neither gets the warmer connection:

	Requests	Latency	Input tokens	Cost / 1,000 readings
Two requests, the second dependent	2	762ms	1,698	$0.071
One request, eleven speculative Choices	1	398ms	2,803	$0.118

One request was faster on 46 of 46 pairs. It named the same family 48 out
of 48 times, and the same shade 45 of 48 — where all three misses were texts
under 0.41 confidence that the page already reports as sitting between two
words, and where the sequential design disagreed with itself between rounds on
one of them. The eleven extra questions barely moved the six that were already
there: mean drift of 0.022 on a 0–4 axis score, and identical intent 48 times
out of 48.

So the fan-out is 1.9× faster for five hundredths of a cent a thousand
readings, and it halves the request count against the limit that actually binds.

Two things I had, and did not put together:

Jev "ingests the state once and evaluates every question against it in
parallel." That sentence is in TypeSafe's own model card, and I had quoted the
half of it that suited me. Question count is nearly free; a round trip is not.
Ten wasted questions cost less than one extra wait.

Jev charges for input only — output tokens are free. The fan-out triples the
output, returning eleven probability distributions instead of one, and that is
worth exactly nothing on the bill.

What I actually got wrong is narrower than "I didn't measure it", and more
useful. The docs say a second request is warranted when the first answer
determines the second question's options. It does — and I read that as
settling the matter. But determined is not unknown: there were only ever
eleven option sets, all of them written down in my own source file, so every one
of them could be asked on spec. The rule is about what the options are. It says
nothing about when you are allowed to ask.

The tell was in my own sentence. "A latency story I could explain" is not a
latency number. Anywhere a design note says presumably, roughly, or I could
explain, there is a measurement someone is about to make for you, and it is
cheaper to make it yourself.

waif is live at dave8172-website.vercel.app/waif. Every number in this post is on the page, under
How it works, next to the vocabulary it was scored against.

Choice, Score and Noul: four mistakes with Jev's primitives

Dave — Sat, 19 Sep 2026 06:25:37 +0000

Jev is a System One model from TypeSafe. It reads natural language like any LLM,
and instead of writing a reply it returns a probability distribution over
options you supply. No prose, no reasoning trace, no JSON to repair — a number
per option, summing to 1.

I spent a week building a Magic 8 Ball on it, which
sounds like a toy and turned out to be a decent test rig: every answer is a
judgment with no ground truth, which is where calibrated probabilities either
earn their keep or embarrass you. It's live at
/jevball.

I got the primitives wrong four times. Each mistake produced working code that
looked right, so they're worth writing down.

The three primitives in one paragraph each

Choice takes a map of named options and returns a probability for each, plus
the highest-scoring one. Routing a ticket to a department, classifying a
document, picking a handler.

Score takes an ordered array of level descriptions — two to ten — and returns
a position along them that can land between levels. Severity, frustration, skill
level. Anything on a spectrum you can describe in words.

Noul takes a yes/no statement and returns one number: the probability it's
true. No confidence field, because the probability is the answer.

All three take the same two inputs: state, the data being judged, and
instructions, the question asked about it. That's the whole surface.

Mistake 1: reading the score as a winner

A Score returns score, a position on your levels. My ball rounded it to pick a
bucket. Someone asked it "is black a color?" and got this:

bars   : 0=2%  1=6%  2=19%  3=16%  4=56%
score  : 3.18

Math.round(3.18) is 3 — a bucket holding 16% — while 56% of the mass sat
on level 4 beside it.

The score is a probability-weighted average. That 19% on level 2 dragged the
average down across a bucket boundary. An average of a skewed distribution
points at a place the distribution isn't.

TypeSafe defines a Choice's answer as the option with the highest probability.
Score has no equivalent field, and I quietly substituted rounding for one. The
fix is to take the argmax of probabilities yourself:

const probs = Array.from({ length: 5 }, (_, i) => v.probabilities[String(i)] ?? 0);
const level = probs.indexOf(Math.max(...probs));

It agrees with rounding on every peaked distribution, which is why it survived
my whole test suite. It differs exactly when the distribution is skewed, which
is when it matters.

Mistake 2: reading low confidence as doubt about the answer

Every Choice and Score comes back with confidence, 0 to 1. I built a "reply
hazy" branch on the assumption that a vague question would produce a low one.

Then I measured it:

question	score	confidence
Should I quit my job?	1.97	0.98
Will it rain next Tuesday?	1.99	0.98
Is black a color?	3.18	0.33

The first two are as uncertain as a question gets, and confidence is 0.98.
Nearly all the mass sat on the level that means "could go either way" — the
distribution was a sharp spike on a level labelled uncertainty.

Confidence measures how peaked the distribution is. It answers "did the
options separate cleanly", and it will happily report 0.98 for a confident
coin-flip. My hazy branch would have fired on precisely the wrong questions.

Low confidence has three different causes, and they need different fixes: the
question is genuinely contested, the options overlap each other, or state
doesn't contain enough to decide. Only the first one is the model being honest.

I tried to work out the formula by fitting twelve real responses. max(p) gets
closest at 0.04 mean error and matches several exactly, but not all —
"will humans land on Mars before 2050?" reported 0.58 against a peak of 0.50,
higher than the peak. The docs are upfront that it's a convenience statistic and
hand you the full probabilities so you can compute your own. If a decision
rides on it, do that.

Mistake 3: offering options that split their own vote

The classic 8 Ball has twenty answers. My first design asked Jev to pick one, as
a Choice.

Ten of those twenty mean "yes". "It is certain", "Without a doubt", "Yes
definitely" — the same claim in different words. Here's what happens, on one
question with four different option sets:

options offered	answer	winner's share
`yes` / `no`	yes	69%
`yes` / `definitely` / `certainly` / `no`	yes	48%

The total yes-mass is identical: 70% in the second row, spread across three
near-synonyms. Nothing changed about the question. The options ate each other,
and confidence fell from 0.38 to 0.31 reporting a disagreement that existed only
in my option list.

So the ball asks Jev for one of five ordered buckets, and the code picks
which of that bucket's phrasings to show. One judgment with a right answer goes
to the model; the theatre stays in software.

Mistake 4: forgetting that the options are the prompt

Your question IDs never reach the model. The instructions and every entry in
criteria do — names and descriptions both. The option list isn't a filter you
apply to a result. It's part of what you're asking.

Same question, same model, options changed:

options offered	answer
`yes` / `no`	yes (69%)
`yes_everyday` / `no_physics` / `depends`	depends (51%)

"Depends" won — an answer that simply didn't exist in the first row. Black is a
colour in everyday use and the absence of light in physics, so "depends" is
arguably the best answer available. It was unreachable until I offered it.

The docs put it plainly: the model cannot choose an omitted value. Leave an
option out and you haven't biased the result, you've made it impossible.

What this is actually for

Every judgment above cost about $0.000014. Jev charges $0.042 per million
input tokens and nothing for output, against $1.00/$5.00 for the cheapest
frontier model I'd otherwise reach for — roughly 27× on a like-for-like
classification, before an LLM's format instructions and reasoning tokens widen
it further.

That price only matters because of what you give up. Jev writes no prose,
explains nothing, and can't answer anything whose answer space you can't
enumerate. For summarising, drafting, coding or open-ended reasoning it's the
wrong tool entirely.

What it's good at is the narrow decision a pipeline makes ten thousand times a
day, where you need a number you can threshold on. The documented pattern is to
put it in front of the expensive work: score everything, route the confident
cases to deterministic code, escalate the rest to a model or a person. You pay
fourteen dollars a million to decide, and frontier prices only on the slice that
earned it.

One caveat worth keeping. Calibration is a property of groups of predictions —
across many answers, the ones marked 0.8 should be right about 80% of the time.
It guarantees nothing about any single answer, and it's measured on TypeSafe's
data, not yours. Validate it in your own domain before trusting a threshold.

The ball is at /jevball. Ask it something you actually want to know
and open the panel underneath — it shows the full distribution, the confidence,
and which of the four questions produced the answer.

252 affiliate programs, and 38% of the terms are not published

Dave — Sun, 13 Sep 2026 06:11:58 +0000

I read the affiliate page of 252 developer and SaaS tools and recorded five terms for each one: commission rate, recurring or one-time, cookie window, minimum payout, and payout method. Every field carries the URL it came from and the date it was read.

483 of those 1,260 fields — 38% — are not published by the vendor at all.

That number is the whole finding, but the shape of it is more useful than the size:

Term	Share of the 252 that never state it
Minimum payout	65%
Cookie window	52%
Payout method	45%
Recurring or one-time	23%
Commission rate	7%

The term almost everyone publishes is the commission rate. The three most often missing are the three that decide whether the money reaches you at all.

Why the rate is the least useful number

A percentage tells you nothing until you know how many times you get paid it. On a $100/month product:

a $150 one-time bounty pays you $150, ever
60% for 12 months pays about $720 in year one, then stops
30% for the customer's lifetime pays $360 a year, indefinitely

Same customer, three different outcomes, and the lowest percentage wins by a distance. Of the 252 programs, 143 pay recurring commission, 51 pay once, and 58 don't say which.

The cookie window decides whether research-heavy traffic is worth anything

Someone clicks your link, the company drops a cookie, and if they buy after it expires you get nothing. For anything people think about before buying — which is most developer tooling — this is the term that decides whether a post you wrote is an asset or a donation.

The real range: Jasper at 14 days against TubeBuddy at 365, and two programs state that attribution never expires. Another 131 publish no window at all.

The minimum payout is where small affiliates actually lose

Most programs hold your earnings until they cross a threshold. Earn $80 in a year against a $250 minimum and you have earned nothing you can spend. Thresholds here run from $5 to $450, seven state no minimum, and 164 don't say.

Payout method is the one to read first if you're not in the US

Some programs pay by PayPal only. Others use Wise, Payoneer, direct bank transfer or EFT. Guides written for a US audience treat this as a footnote. If you're somewhere PayPal is awkward or expensive, it is the difference between money arriving and money technically existing. 113 of the 252 don't say how they pay.

One counting decision worth explaining

152 of the 252 name no affiliate network. I deliberately left that out of the transparency count.

A company that names no network may be running the program itself rather than concealing anything, and "in-house" and "undisclosed" are different facts. Inferring one from the other would mark every company that built its own system as secretive. So the network is published where a vendor states it, and counted nowhere.

This is the kind of decision that makes or breaks a dataset, and it is usually invisible in a roundup post.

"Not stated" is a finding, not a gap

Where a vendor publishes nothing, the record says not stated rather than filling it with an estimate. Knowing that a company won't publish its cookie window is worth knowing before you build content around it.

That rule is also why this took reading 252 pages instead of aggregating three listicles. A roundup whose author earns commission from the programs it ranks has an incentive to sort by rate rather than by accuracy. Nothing here earns a commission from any program listed.

The data

affiliateprogramterms.com — the full directory. 50 programs are free to read in full, each field with its source URL and check date.
github.com/dave8172/affiliate-program-terms — the free tier as a generated README, if you'd rather read it in a repo. Every count in it is computed from the dataset rather than typed, so it can't drift from the records it describes.

If you find a figure that's gone stale, open an issue with the vendor URL. Terms change without notice, which is the reason every field here carries a date.

doceval — eval harness for LLM document extraction pipelines

Dave — Tue, 16 Jun 2026 12:29:37 +0000

I kept seeing the same gap: people ship LLM-based document extractors (invoices, receipts, forms) with no systematic way to know how accurate they actually are. So I built doceval — point it at your extractor function + a labeled dataset and get back field-level accuracy, a failure taxonomy (missed_field / hallucination / wrong_format / wrong_value), and optional per-document cost tracking.

Works with any extractor (Claude, GPT, regex, rules) and any document schema. One JSON label file per document, one Python function, one CLI command.

Includes a working 20-document invoice example with a Claude Haiku extractor so you can run it immediately.

GitHub: