How to label text with a local LLM so it only answers with your labels

Labelling rows with a language model looks like the easiest job you can give it. Send each row with an instruction such as "complaint, praise or question, only the label", keep what comes back, count. The Run Model action in the Datasets screen of the osFoundry desktop app works this way: BatchInferenceDialog.tsx sends each row on its own as the instruction, a blank line and the row's text, and stores the answer as a new column. Below, that loop is rebuilt in Python with mlx-lm and run on 312 shop reviews whose correct labels are known, to see what actually comes back. A 3B model got almost every review right. It also wrote the labels six different ways.

The setup

• Data: 312 short reviews for a small pottery shop, generated by a script from phrase lists, so every review's label is known in advance: 181 praise, 87 complaint, 44 question. One English set and one Japanese set with the same mix. They are synthetic, which makes them easier than real reviews; the point here is the shape of the answers, not a benchmark.
• Models: Qwen2.5 3B Instruct and Qwen2.5 0.5B Instruct, both 4-bit MLX builds from mlx-community.
• Machine: Apple M1 with 8 GB of memory, mlx-lm 0.32.2, mlx 0.32.4, greedy decoding.

The loop is the same as the app's: one user message per row, instruction first.

import csv
from mlx_lm import load, generate

PROMPT = "Label this customer review as complaint, praise or question. Output only the label."
model, tok = load("mlx-community/Qwen2.5-3B-Instruct-4bit")

def prompt_for(text):
    msgs = [{"role": "user", "content": PROMPT + "\n\n" + text}]
    return tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)

rows = list(csv.DictReader(open("reviews.csv")))
answers = [generate(model, tok, prompt_for(r["review"]), max_tokens=16) for r in rows]

Right answer, six spellings

The 3B model's answers for the English set, counted as written:

praise      138
Complaint    48
Praise       42
complaint    39
question     26
Question     19

Six distinct values. Compared exactly with the true labels, 202 of 312 are right. Lowercased, 311 are right; the one miss is "Perfect. My coffee tastes better in it. Probably. Will be back.", which it called a question. So the model understood the task almost perfectly and still produced a column that a plain count or a filter on complaint gets wrong by more than a third. Nothing in the instruction was ambiguous: it listed the labels in lowercase and asked for the label only. The capital letters come from the model's habit of starting an answer with one, and it does that on some rows and not others.

The cheap check is the number of distinct values in the new column. It should equal the number of labels. If it is higher, look at the extra values before counting anything.

The cheap fix is to normalise before comparing:

def norm(s):
    return s.strip().strip("\"'「」.。:：*` ").lower()

That is enough for capitals and full stops. It is not enough for the Japanese set.

A label that isn't a word

With the Japanese instruction (不満・満足・質問のどれか, ラベルだけ), the same 3B model gave four distinct values: 満足 178, 不満 87, 質問 45, and 不溝 twice. 不溝 is not a label, and not a word anyone would write here. Normalising can't map it to anything.

The tokenizer explains where it comes from. Qwen2.5's tokenizer works on bytes, and 満 is not a single token:

不満  -> [16530, 35017, 222]    '不' + two byte pieces of 満
満足  -> [35017, 222, 99336]    two byte pieces of 満 + '足'
不溝  -> [16530, 35017, 251]    '不' + the same first piece + a different last byte

満 and 溝 share their first UTF-8 bytes, so the model wrote 不, then the shared byte piece, then picked the wrong final byte. Both rows were praise in the true labels, and both start with プレゼント用のはずが、自分で使っています ("bought it as a present, ended up using it myself"), so the model was already unsure.

Make it choose instead of write

If the answer has to be one of three strings, there is no need to let the model write. Run the prompt once, look at the log-probabilities it assigns to the next token, and compare only the first token of each label. Here the three first tokens are different (com, pr, question in English; 不, the 満 byte piece and 質 in Japanese), so one step decides:

from mlx_lm import stream_generate

LABELS = ["complaint", "praise", "question"]
first = [tok.encode(l)[0] for l in LABELS]
assert len(set(first)) == len(LABELS)

def choose(text):
    step = next(stream_generate(model, tok, prompt_for(text), max_tokens=1))
    scores = [step.logprobs[t].item() for t in first]
    return LABELS[scores.index(max(scores))], scores

If two labels share a first token, score the whole label instead: append each label's tokens to the prompt and sum their log-probabilities.

Both versions were run back to back for each language, on the same machine:

3B model	distinct values	right as written	right after norm	time for 312 rows
English, generate	6	202	311	113.2 s
English, choose	3	311	311	94.5 s
Japanese, generate	4	308	308	146.4 s
Japanese, choose	3	308	308	117.5 s

Choosing gives exactly three values every time, needs no clean-up, and is a little faster because it stops after one step instead of generating the label and the end-of-turn token. It does not make the model any smarter: 311 and 308 right either way.

What choosing hides

The two 不溝 rows show the cost. In the generate run they were visibly broken. In the choose run the first token was 不, so both became 不満: wrong, but indistinguishable from the 87 real complaints. An invalid answer was a free warning, and forcing a valid one removes it.

The scores bring the warning back. The gap between the best and second-best label's log-probability says how close the call was, and sorting by it puts the doubtful rows first:

label, s = choose(text)
top, second = sorted(s, reverse=True)[:2]
margin = top - second   # small = the model nearly picked another label

In the English run, the one wrong row had the smallest margin of all 312 (0.97; the next smallest was 3.06). In the Japanese run, 9 rows had a margin under 2, and 3 of the 4 wrong rows were among them. Reading 9 rows out of 312 is a reasonable amount of checking for a batch job; reading all 312 is what the model was supposed to save you from.

A small model doesn't get rescued

The 0.5B model shows the limits of all this. On the English set it answered "Complaint" for all 312 reviews; choosing gave "complaint" for all 312. On the Japanese set, generating gave 206 distinct values, mostly sentences about the review rather than a label, and choosing gave only 不満 and 満足, 110 of 312 right. One distinct value is as strong a warning as six: a label column with a single value has not sorted anything.

Back in the app

The full script, with the counting and a confusion matrix, is under 70 lines. In the osFoundry app, Run Model in the Datasets screen does the batch part with a model on your computer, and Show Quality Stats on the result shows the distinct-value count for each column, which is the first check above.

Running the models themselves is covered on the page about local LLM inference in osFoundry, and the desktop app is on the osFoundry download page.