I tested three ways to do zero shot classification on 200 real customer messages.
This article covers how each of the three methods work, how they compare on accuracy, speed, and cost, and which one best fits your application.
What is zero shot classification?
Zero shot classification is when you give an AI model an input and a list of label names, and it picks the best fit with no training data and no examples.
Say a customer writes "Do you know the closest ATM?" and your labels are atm support, lost or stolen card, and request refund. A zero-shot classification model picks atm support without ever seeing a labelled example.
I personally work on zero shot classification every day building AI agents for online stores, and it's far from a waste of time. Online shoppers who experience a high quality chat experience spend significantly more money.
3 ways to do zero shot classification
There are three common ways to do zero-shot text classification today: small open-source models, large language models (LLMs), and System One models. All three take the same input (a piece of text and a list of labels). System One models like Jev are the new kids on the block.
1. A Hugging Face zero shot classification model
The Hugging Face zero-shot classification pipeline with facebook/bart-large-mnli is what most existing tutorials use. It turns each label into a sentence ("This example is atm support.") and checks whether the text backs that sentence up. It's free and runs on your own machine - which is a major drawcard.
import { pipeline } from '@huggingface/transformers';
const classify = await pipeline('zero-shot-classification', 'Xenova/bart-large-mnli');
const result = await classify(message, labels);
console.log(result.labels[0]); // the top label
2. Zero shot classification with an LLM
Zero-shot classification with an LLM is a prompt: here is the message, here are the labels, pick one. I tested Claude Opus 5.5 and Claude Haiku 5.5, and limited the answer to the label list so the model could not invent a new label. It's worth mentioning that this method doesn't always work as LLMs tend to get chatty, especially when something goes wrong.
const response = await client.messages.create({
model: 'claude-haiku-5-5',
max_tokens: 2000,
system: `Pick the one label that best fits the message.\n${labels.join('\n')}`,
messages: [{ role: 'user', content: message }],
output_config: { format: { type: 'json_schema', schema: labelSchema } },
});
3. A System One model
A System One model is built to make fast decisions instead of writing text. Typesafe's Jev takes the message and one multiple-choice question, then returns the chosen label with a confidence score (from zero to one) for every option.
// options = { 'atm support': null, 'request refund': null, ... }
const { answers } = await client.systemOne({
state: message,
questions: { label: choice('Which label best fits this message?', options) },
});
console.log(answers.label.choice);
Testing zero shot classification
I tested all three methods on the same 200 customer messages from Banking77, a public set of banking support messages. I picked 10 labels at random, took 20 messages for each, and gave every model the label names and nothing else.
It's worth mentioning that I'm no machine learning engineer. I first learned to code by building a Next.js app with no experience. Each setup here is one you can easily copy yourself.
| Method | Accuracy | Median time | Cost per 1,000 messages |
|---|---|---|---|
| bart-large-mnli | 73.5% | 1,264 ms | Free (own machine) |
| Claude Haiku 5.5 | 93.0% | 1,211 ms | US$0.05 |
| Claude Opus 5.5 | 95.0% | 1,926 ms | US$1.96 |
| Jev | 92.0% | 226 ms | US$0.016 |
Three things stood out:
- The LLMs were the most accurate. Claude Opus 5.5 got 190 of 200 right, and Claude Haiku 5.5 got 186
- Jev was the fastest and the cheapest to call. It answered in 226 ms, about five times faster than Haiku and eight times faster than Opus, while scoring 92%
- The tutorial model trailed by about 20 points. bart-large-mnli scored 73.5% and took 1.3 seconds a message on my own machine
I also ran every message twice to see whether the answers held. bart-large-mnli changed none of its 200 answers, Jev and Opus changed one each, and Haiku changed three.
Limitations of zero shot classification
The main limitation of zero-shot classification is that the model only has your label names to go on. When two labels overlap, or a name is vague, the model guesses.
This exact limitation showed up in two places during my tests:
-
Overlapping labels. bart-large-mnli got all 20
atm supportmessages wrong. It filed "Which ATMs accept this card?" undercard acceptance, which is a fair reading of those two names -
Labels that hide their meaning. Every model put "Are my friends able to top up my account?" under
receiving money. The dataset files it undertopping up by card. The same six messages caught Opus, Haiku, and Jev
Neither is really the model's fault, and both are fixable. To prove this I ran a single tuning round on Jev. I checked the misses, wrote a one-line description for each label, and tested it on 200 fresh messages it hadn't seen yet.
Jev went from 93% to 99%, reaching LLM-level accuracy at a fraction of the time and cost, after one round of tuning.
How to choose the right zero shot classification method for your application
Choose your zero-shot classification method by what your application needs most: privacy, accuracy, or speed.
- Use bart-large-mnli when the data cannot leave your machine or the budget is zero. Expect to check its work
- Use an LLM when accuracy matters most and a second or two per message is fine. Claude Haiku 5.5 scored 93% for five US cents per 1,000 messages
- Use a System One model like Jev when classification sits in front of every request, like routing live chat. Allow for one or two tuning rounds on your labels
- Train your own model when you have thousands of labelled examples and the labels never change. This comparison of zero-shot vs fine-tuned models covers when that is worth it
Whichever you choose, test it on your own messages first. Two hundred is enough to show where your labels overlap.
UPDATE: Google released their own decisions API the day after I wrote this article. Here's what happened when I compared the Google Decisions API vs Jev.
Top comments (1)
Google just released their own decision model API: blog.google/innovation-and-ai/tech...