Jev is a System One model from TypeSafe. It reads natural language like any LLM,
and instead of writing a reply it returns a probability distribution over options
you define. No prose, no reasoning trace, no JSON to repair.
I spent a day building waif, which reads a piece of
text and names the feeling in it. That job has no right answer, which makes it an
unusually honest test rig: nothing can be graded against a key, so every design
decision has to be either argued or measured.
I measured five, argued two, and the two I argued were both wrong. Sections 1
and 7 are those, kept in place with the measurement underneath rather than
quietly edited out.
1. A Choice splits its own vote between synonyms — but less than I claimed
Here is the argument I built on, and I am leaving it in its original words
because the correction under it is the useful part.
Annoyed, irritated and frustrated are one feeling in three wordings. A
Choice divides probability between them, so a text the model read perfectly
clearly comes back split three ways, and confidence collapses for a reason that
has nothing to do with the input. The distribution is telling you about your
option list, not about the text. Therefore: never put sixty near-synonyms in one
Choice.
Then I measured it, and the effect is real but far smaller than the argument
needs. One Choice over all 62 words, same glosses, 23 texts:
| Mean P(chosen word) | Score vs acceptable words | |
|---|---|---|
| Two Choices — family, then shade within it | 0.689 | 43 / 46 |
| One Choice over all 62 words | 0.812 | 44 / 46 |
The one-Choice version is more certain, not less. It picks the same word 21
times out of 23. On a plainly frustrated text it returns frustration at 0.98.
Stripping the glosses and asking with 62 bare words barely moves it: 0.79.
So where does the splitting go? It shows up exactly where you would want it to:
shipped pride 0.38 → excitement 0.36
scan worry 0.54 → anxiety 0.39
sentit guilt 0.60 → embarrassment 0.26
Every one of those lost ground to a near-synonym — that is vote splitting. But
the two-Choice design flattens on the same four texts (0.33, 0.53, 0.35,
0.32). The splitting is not caused by the option list being long. It is caused
by the text genuinely sitting between two words, and both designs report it
because both are calibrated. "I snapped at her in front of the kids" really is
guilt and regret at once.
A long option list does not flatten a clear text. That is the part I had
backwards, and the rule has to be narrower than I wrote it:
A Choice splits its vote when two options are the same answer in different
wordings. The fix is criteria that separate them — not fewer options.
Sixty-two words each carrying a gloss that distinguishes it (annoyance: a small
thing, quickly over against resentment: an old grievance still carried) are
sixty-two alternatives, not sixty-two synonyms. The count was never the problem.
2. Ask twice anyway — for a reason that survived
Section 1 was the reason I split the question in two, and section 1 did not
hold. The split stayed, and it is worth being precise about what is now holding
it up, because "it scored the same and I had already built it" is not a reason.
Families are genuinely different answers — anger, fear, sadness and shame are
not wordings of each other. And once the family is fixed, so are its shades:
which shade of anger is a fair question, because the context has already ruled
out the fifty-four words that were never in the running.
What that buys, and one Choice over 62 words cannot, is two separate
uncertainty signals. Family confidence and word confidence are different
doubts: "I do not know whether this is sadness or affection" is not the same
failure as "it is clearly shame, but guilt or embarrassment?". The page says
different things in each case. One Choice gives you one number that cannot tell
them apart.
The cost is honest too: a wobble in the family answer corrupts the word, because
once family says sadness, nostalgia is not on the ballot. That cost me one
text out of 23 — "drove past the old house, the tree we planted is taller than
the roof" came back sorrow.
So the naming became two Choices in sequence: the first picks the family, the
second picks the shade from that family alone. TypeSafe's docs say a second
request is warranted when an earlier answer determines the next question's
options. This is exactly that, and here is what it bought — scored on 24 texts
against a list of acceptable answers for each:
| Design | Score |
|---|---|
| Nearest word in the whole vocabulary, by published valence / arousal / dominance | 3 / 24 |
| Nearest word within a family, by rank on the axis that separates that family | 9 / 24 |
| Nearest word within a family, by distance in those published ratings | 12 / 24 |
| Family chosen by the model, then the shade chosen by the model | 22 / 24 |
Both of the two failures were the wrong family. Given the family, the shade was
right every single time.
The alternative was speculative fan-out: ask all eleven within-family Choices in
the first request, each stating its own family as a premise, and keep only the
answer belonging to the family that won. That keeps a reading to one request, at
roughly 3–4k input tokens against 1,250 + 444 for two. I reasoned about it and
rejected it: two requests won on cost, and on a latency story I could explain.
That paragraph was wrong when I published it. It is still here because
section 7 is the part of this post
I would keep if I could only keep one.
3. Knowing which half to give the model is a decision you can measure
The job is: read a text, name the feeling. That job splits between the code I
write and the model I call, and the only real decision is where the line falls —
how much of the work do I hand over?
The first three rows of that table are me keeping most of it.
I had a reason. Human ratings exist for exactly the dimensions I was measuring:
Warriner, Kuperman & Brysbaert scored 13,915 English words for valence, arousal
and dominance by asking people. So the model's job shrinks to placing the text
on those three scales, and my code finishes the job — look up the nearest word
in the table, return it. Research-backed coordinates instead of somebody's
guesses. It looked like the serious version.
It scored 3/24. Handing the model the whole naming job scored 22/24.
Three reasons, worth knowing before anyone else reaches for an emotion lexicon:
- Three scales were really about two. Across these emotion words, valence and dominance move together at +0.87 — a word that reads pleasant almost always reads in-control. The third scale is close to a copy of the first, so it separates far less than the theory promises.
- The unpleasant words are crammed into one corner. Fear, frustration, worry, terror, jealousy and embarrassment land in a ball small enough that taking the nearest one is close to picking at random. The coordinates are fuzzy to begin with: each is an average over about twenty raters who disagreed by around 1.7 on a 1–9 scale.
- Rating a word on its own is not the same measurement as reading a sentence. People rate the bare word "gratitude" as far more activated than an actual grateful message reads. The two sets of numbers were never on the same ruler.
I tried to fix that last mismatch by re-centring both sides against a sample of
texts. It got worse, and instructively: the sample leaned negative, so the
correction leaned negative, and a plainly warm text landed below the middle and
got named from the sad half of the space. That is straightening a bent ruler
with a bent ruler.
The lesson is not don't use lexicons. It is that where the line falls
between what code owns and what the model owns is a design decision with a
number attached, and my intuition about it was wrong by a factor of seven.
The norms kept the one job they are good at. A Score is not given "rate this 1
to 5" — it is given a rubric, a written description of what each level means,
the way a grading key spells out what a B looks like. Every level of mine now
names words whose ratings were measured, so "as activated as rage or panic" is
a claim a reader can argue with. "Very aroused" is only a word getting louder.
4. Confidence is peakedness — and peakedness misses a coin toss
A Score's confidence measures how bunched together the answer is. It does not
measure how likely the model is to be right, which is the trap everyone hits
first.
Those two sound like the same thing until a reading like this one turns up.
Confidence came back at 0.72, comfortably above any threshold I would set,
and here is what was underneath it:
| Where the answer sat | Share |
|---|---|
| The winning level | 50% |
| The level right next to it | 49% |
| The other three | 1% |
The 0.72 is not lying. Ninety-nine percent of the weight really is in two
buckets and there is nothing anywhere else — that is about as bunched as an
answer gets.
It is also a coin toss. Choosing between the top two is 50 against 49, and my
page announced the winner in exactly the voice it uses at 0.98.
Confidence cannot catch this, because bunched into two neighbours is still
bunched. What I actually wanted was the gap between first and second place,
which is a different number entirely: 98 against 1 is a winner, 50 against 49 is
a tie with a rounding error. Confidence says the same thing about both — and the
gap is the one a reader cares about.
5. Audit your questions: one of mine fired on 21 of 24 inputs
A question set grows by accretion. Each addition looks free, and none of them
announce that they have stopped saying anything.
So run them over a corpus and look at the spread. Mine had a Noul asking is more
than one feeling present. It returned ≥0.6 on twenty-one of twenty-four
texts. That is not a signal about the input, it is a property of writing — a
question paying tokens to tell you something you already knew.
Two more went for a different reason. They were informative, but nothing
downstream consumed them beyond appending a line to the output. A question whose
entire effect is an occasional footnote costs a reader more attention than it
returns.
The check is cheap: for every question, the min, max and spread of its answer
across a representative corpus. Anything that barely moves is either a gate or a
mistake.
6. Two small things that cost real time
A rubric level must not contain a word from another axis. My control rubric
had a level reading "Overwhelmed: struggling to keep any grip on it". That
primes the model with a feeling while asking it about agency, and it labels the
output with a word that is not a position on a control scale at all. Every level
of an axis has to be a point on that axis.
Input is dominated by your rubrics, not by your input. Criteria are sent on
every call, so a one-line text costs almost exactly what a paragraph does. At
this size requests are the scarce resource and tokens are not — which is the
whole argument for batching every independent question into one call.
I stopped one clause too early, and wrote that a dependent second call is
therefore a real cost rather than a rounding error. Read it again: if requests
are scarce and tokens are not, the conclusion goes the other way.
7. I reasoned where I should have measured
Someone read section 2, noticed I had talked myself out of the fan-out, and told
me to go and time both. It took forty minutes and a dollar's worth of nothing.
Run the same 24 texts through both designs, twice each, back to back on every
text so neither gets the warmer connection:
| Requests | Latency | Input tokens | Cost / 1,000 readings | |
|---|---|---|---|---|
| Two requests, the second dependent | 2 | 762ms | 1,698 | $0.071 |
| One request, eleven speculative Choices | 1 | 398ms | 2,803 | $0.118 |
One request was faster on 46 of 46 pairs. It named the same family 48 out
of 48 times, and the same shade 45 of 48 — where all three misses were texts
under 0.41 confidence that the page already reports as sitting between two
words, and where the sequential design disagreed with itself between rounds on
one of them. The eleven extra questions barely moved the six that were already
there: mean drift of 0.022 on a 0–4 axis score, and identical intent 48 times
out of 48.
So the fan-out is 1.9× faster for five hundredths of a cent a thousand
readings, and it halves the request count against the limit that actually binds.
Two things I had, and did not put together:
Jev "ingests the state once and evaluates every question against it in
parallel." That sentence is in TypeSafe's own model card, and I had quoted the
half of it that suited me. Question count is nearly free; a round trip is not.
Ten wasted questions cost less than one extra wait.
Jev charges for input only — output tokens are free. The fan-out triples the
output, returning eleven probability distributions instead of one, and that is
worth exactly nothing on the bill.
What I actually got wrong is narrower than "I didn't measure it", and more
useful. The docs say a second request is warranted when the first answer
determines the second question's options. It does — and I read that as
settling the matter. But determined is not unknown: there were only ever
eleven option sets, all of them written down in my own source file, so every one
of them could be asked on spec. The rule is about what the options are. It says
nothing about when you are allowed to ask.
The tell was in my own sentence. "A latency story I could explain" is not a
latency number. Anywhere a design note says presumably, roughly, or I could
explain, there is a measurement someone is about to make for you, and it is
cheaper to make it yourself.
waif is live at dave8172-website.vercel.app/waif. Every number in this post is on the page, under
How it works, next to the vocabulary it was scored against.
Top comments (1)
The split-vote problem with synonyms is underappreciated — you can prompt-engineer around a lot of things, but you can't fix a structurally bad option list by rewording the instructions. The 22/24 vs 9/24 jump from "family then shade" vs. "within-family by axis rank" makes the case clearly.
The confidence-as-peakedness insight in section 4 is the one I'd clip. 0.72 confidence with a 50/49 first-vs-second margin is a genuinely tricky failure mode — the distribution IS concentrated, so any confidence threshold passes it, and you'd never know it's effectively a coin toss without checking the margin separately. That's the kind of bug that ships quietly and shows up as mysterious output variance rather than an obvious error.
The question audit idea (run every question over a corpus, check spread, kill anything that barely moves) is also worth generalizing beyond emotion labeling. Any multi-question evaluation prompt accumulates dead questions the same way. Good write-up — the scored table across all five designs makes the argument concrete rather than anecdotal.