A language model does not have to write a sentence to be useful. Sometimes an application already knows the possible actions. It needs to decide which one fits the current state, and perhaps how much uncertainty to attach to each option.
That is the idea behind NobodyWho’s small Python demonstration. It feeds a local Qwen model an email and three labels—legitimate, spam, and phishing—then reads the model’s next-token scores instead of asking it to generate a reply. The example is deliberately provocative: its author presents it as a compact answer to TypeSafe’s Jev, a specialized service for structured, probabilistic decisions. It is also explicitly a parody. The interesting technical question is what this short program actually demonstrates, and what still needs to be built around it.
The task is a choice, not a conversation
Imagine a message claiming to be from payroll and directing an employee to enter a password on an unfamiliar sign-in page. A conversational assistant might explain the danger in several sentences. A mail filter needs an output that code can use: Legitimate, Spam, or Phishing.
The local demonstration loads a quantized Qwen3 0.6B GGUF model with llama-cpp-python. Its prompt supplies the email and the three possible answers, represented by A, B, and C. It ends immediately before the assistant’s answer. The model performs a prompt evaluation, and the program inspects the scores for those three label tokens at that position.
That distinction matters. A normal generation loop repeatedly selects a token, appends it to the context, and runs the model again. Here the application needs only the first decision position. It can read a distribution without waiting for an explanation or JSON object to be spelled out token by token.
How the 25-line idea works
The original program has three essential stages. First it loads the model with Llama.from_pretrained, selecting the GGUF file and a 512-token context. It enables logits_all=True, because the example reads the wrapper’s stored score array after evaluating the prompt. The binding’s API reference documents how prompt evaluation and score storage work. The exact Python interface can change, so a production implementation should pin and test its dependency version.
Second, it formats an instruction and an email alongside three labeled answers. The assistant prefix includes an empty Qwen thinking block, steering the model to answer at the next token rather than begin an extended reasoning trace. The code tokenizes that prompt with special tokens enabled and passes the token IDs to model.eval(...). This is a model-specific prompt format; another model needs its own chat template.
Third, it takes the last row of logits and selects only the token IDs for A, B, and C. A logit is an unnormalized score. It has no useful probability interpretation on its own. The program subtracts a log-sum-exp over those three scores and exponentiates the results. Mathematically, for a permitted choice, p_i = exp(z_i) / (exp(z_A) + exp(z_B) + exp(z_C)), where z_i is the model’s next-token logit for that label. Using log-sum-exp avoids numerical overflow when scores are large. In the article’s sample run, the resulting values are about 3.1% legitimate, 8.4% spam, and 88.5% phishing. These are an illustration from one prompt and one model run, not a measured reliability guarantee.
A minimal version of the readout, once a model and prompt are prepared, looks like this:
model.eval(model.tokenize(prompt.encode(), add_bos=False, special=True))
last_scores = model.scores[model.n_tokens - 1]
ids = [model.tokenize(x.encode(), add_bos=False)[0] for x in ("A", "B", "C")]
selected = numpy.array([last_scores[token_id] for token_id in ids])
weights = numpy.exp(selected - selected.max())
probabilities = weights / weights.sum()
The max subtraction is another stable form of softmax. In real code, check that every label is exactly one token in the answer context, that the three IDs differ, and that the prompt fits the context window. Otherwise a “choice score” could actually score only the first token of a longer label. Also reset or manage model state between independent requests; the demonstration is a one-shot example, not a request-serving design.
What those probabilities mean
The three numbers add to one because the code renormalizes only across the three displayed labels. That does not mean the model assigned all its probability mass to those labels. It might have preferred a newline, a refusal, or a different answer token. The code discards that outside mass and asks, “Assuming the answer is one of these three labels, how are they ranked?”
Nor does 88.5% automatically mean that 88.5% of messages assigned that score will truly be phishing. That latter statement is calibration. It requires a representative labeled test set and a comparison of predicted confidence against observed correctness. SemIf’s browser experiment, which compares direct readout with generated JSON on the same local model, makes this limit explicit: its direct scores are conditional choice probabilities, not calibrated confidence.
This distinction changes how a system should use the result. Suppose phishing has the highest score. A mail client might show a warning, but automatically deleting the message would need a much stronger evidence base. A responsible deployment would measure false positives and false negatives on its own mail, set thresholds for different actions, and send ambiguous cases to review. It would also test how often benign changes to option order, wording, punctuation, and email format alter the distribution.
The tiny example is useful precisely because it exposes these assumptions. Once the answer is a fixed set of labels, an application can log each distribution, compare model versions, and draw a reliability curve. A generated explanation may be helpful to a human, but it is not a substitute for these measurements.
What the comparison with Jev does and does not establish
TypeSafe describes Jev as a model built around structured program state and typed decisions. Its launch post says the system uses a different architecture, a parallel sampler, and training it calls Reinforcement Learning for Calibrated Decisions. It also says Jev can produce multiple structured outputs in one query and is optimized for short-latency automation. Those are claims about a specialized model and service, not features created by applying softmax to three Qwen logits.
The local Python program does establish a narrower, valuable point: a general language model can act as a fixed-choice scorer without generating answer text. For one small classification task, the entire path can run locally, with no input sent to an inference API after the model has been downloaded. It can produce a valid distribution over the offered labels and let ordinary application code decide what to do next.
It does not establish equivalent accuracy, calibration, speed, cost, or behavior on many simultaneous questions. The demonstration supplies no head-to-head benchmark. TypeSafe’s own performance figures come from its chosen workflows and comparison harness; its launch post acknowledges that these workflows were made by its model-capabilities team and that its headline gains may sit toward the high end of real deployments. Neither side’s marketing numbers should replace a test on the workload you actually have.
Open implementations show the space between a toy example and a full service. The openjev-sglang project implements a Jev-compatible HTTP interface using a larger Qwen model and SGLang. It validates request schemas, handles multiple questions, scores answer-token probabilities, and documents batching, caching, limits, and failure behavior. That extra machinery is not ornamental: it is what turns a clever scoring trick into something that can sit behind an application endpoint.
When to use the pattern
Direct choice scoring fits problems where the output space is known in advance: routing a support ticket, choosing a moderation category, ranking a small set of next actions, or deciding whether to escalate a case. It can be especially attractive when local processing is a requirement or when generating prose would add latency and parsing failures without adding value.
The pattern is less natural when the application needs a novel answer, extraction of an arbitrary name or value, a long explanation, or a choice set too large to score cleanly as distinct single tokens. Even in its sweet spot, the system still needs conventional engineering: input validation, a stable prompt, labeled evaluation data, calibrated thresholds, monitoring for distribution shifts, and a safe path for uncertain cases.
The short script’s lasting lesson is not that specialized decision models are unnecessary. It is that a model’s most useful output can be the scores it already computes before it writes a word. Expose those scores, state exactly what they are conditional on, and test whether they support the decision your software is about to make.
Top comments (0)