DEV Community: Matt Duffin

Google Decisions API vs Jev: Which is Better?

Matt Duffin — Thu, 12 Nov 2026 23:42:49 +0000

Google just launched a direct rival to the fastest model in my zero shot classification test.

Considering Google's track record with AI (not the greatest), I wanted to see how their new "System One" model stacked up.

So in this article, I ran the new Google Decisions API through the same 200 customer messages as my previous article.

TLDR: They're about the same, but Jev wins on speed.

What is the Google Decisions API?

The Google Decisions API is a Gemini API endpoint that answers fixed questions about an input instead of writing text. You send text, images, audio, video, or a PDF with a list of questions, and it returns an answer and a confidence score for each one.

It launched on 11 November 2026 and runs on a new model, Gemini Decision 1 (gemini-decision-1).

It matters to me because of my work building chatbots and agents for online stores. When every shopper message gets sorted before anything else happens, a model that only makes decisions (and makes them fast) is usually the right tool for that step.

How does the Google Decisions API work?

The Google Decisions API takes one shared input and up to 1,000 questions, and scores every question in a single pass. Each question is one of three types:

noul is a yes or no question. It returns the probability that the answer is yes.
choice picks one option from a list of 2 to 255. It returns the winner and a probability for every option.
score rates the input on an ordered scale. It returns a weighted score and a probability for each level.

If those names look familiar, Jev uses exactly the same three question types with the same names.

Zero shot classification is one choice question. This is the call I used in the test:

import { GoogleGenAI } from '@google/genai';

const ai = new GoogleGenAI({ apiVersion: 'v1beta' });
const result = await ai.decisions.create({
  model: 'gemini-decision-1',
  input: message,
  questions: {
    label: {
      type: 'choice',
      instructions: 'Which label best fits this customer message to a bank?',
      criteria: { 'atm support': '', 'request refund': '' }, // one key per label
    },
  },
});
console.log(result.answers.label.choice);

Google Decisions API pricing

Google Decisions API pricing is US$0.10 per million input tokens, with no charge for output tokens. There's also a no-cost tier in Google AI Studio for prototyping.

Here is how Gemini Decision 1 and Jev compare on paper, from Google's Decisions API documentation and TypeSafe's model page:

	Gemini Decision 1	Jev 1.14
Price per million input tokens	US$0.10	US$0.042
Output tokens	Free	Free
Question types	noul, choice, score	noul, choice, score, box
Inputs	Text, images, audio, video, PDFs	Text, images
Context per request	262,144 tokens	64,000 tokens (262,000 with long context on)

Jev is less than half the price per token, but the two APIs count tokens differently. For the same message and the same ten labels, Google billed 152 tokens a request and TypeSafe billed 375 (I have no idea why). Both came to about $0.015 per 1,000 messages.

Testing Google Decisions API vs Jev

I tested the Google Decisions API on the same 200 customer messages as last time: banking support messages from the public Banking77 set, 10 labels, 20 messages each. Every model got the label names and nothing else.

Model	Accuracy	Median time	Cost per 1,000 messages
Gemini Decision 1	92.5%	511 ms	US$0.015
Jev	92.0%	226 ms	US$0.016
Claude Haiku 5.5	93.0%	1,211 ms	US$0.05
Claude Opus 5.5	95.0%	1,926 ms	US$1.96
bart-large-mnli	73.5%	1,264 ms	Free (own machine)

Three things stood out:

Accuracy was a tie. Gemini Decision 1 got 185 of 200 right and Jev got 184. Twelve of Gemini's 15 misses were the same messages Jev missed
Jev was more than twice as fast. Jev answered in 226 ms and Gemini Decision 1 in 511 ms. Google's article quotes 150 to 250 ms
Both left the LLMs behind on speed and cost. Claude Haiku 5.5 took 1.2 seconds, and Claude Opus 5.5 cost over 100 times more per message

I ran every message twice. Gemini Decision 1 gave the same answer both times on all 200, and Jev changed one.

Then I gave both models one tuning round. I wrote a one-line description for each label and tested on 200 fresh messages neither model had seen.

Both models finished on 99%. Gemini Decision 1 went from 93.5% to 99%, and Jev went from 93% to 99%.

Limitations of the Google Decisions API

The main limitations of the Google Decisions API are that it is literally brand new, and that a choice question always picks something. Google's documentation lists these as well:

It is in beta. The Decisions API is only on the Gemini Developer API (v1beta), with Google Cloud support still to come
A choice question never says "none of these". Probability is always spread across your options, so you need a confidence threshold or an "other" option
No batch mode or context caching. Repeated input is billed in full on every request
Questions are text only. You can send media as the input, but not inside a question

In my test, its misses came from overlapping label names. It filed "Which ATM's will accept my card?" under card acceptance when the dataset says atm support.

The confidence score caught most of them. Twelve of its 15 misses had a confidence below 0.8, and it was right on 145 of the 148 answers at 0.8 or higher.

Should you use the Google Decisions API or Jev?

Use the Google Decisions API or Jev based on speed and input type, because accuracy and cost were level in this test.

Use the Google Decisions API when you already build on Gemini, when you need to classify audio, video, or PDFs, or when you want the no-cost tier.
Use Jev when response time matters most. It was more than twice as fast from Australia.
Use an LLM when you need the highest accuracy from label names alone and can wait a second or two.

Whichever you choose, write a description for every label. That one change took both decision models from 93% to 99%.

Update: Google just released their very own Decisions API. I will have to check how it goes against Jev for zero-shot work. https://blog.google/innovation-and-ai/technology/developers-tools/gemini-decisions-api/

Matt Duffin — Thu, 12 Nov 2026 08:59:07 +0000

Matt Duffin

Nov 12

Zero Shot Classification: Jev vs LLM vs Hugging Face

#zeroshotclassification #jev #systemone #typesafe

5 min read

Gemini Decisions API: Fast multimodal decisions

Meet the Gemini Decisions API, powered by our state-of-the-art model Gemini Decision for fast, calibrated multimodal decisions.

blog.google

Zero Shot Classification: Jev vs LLM vs Hugging Face

Matt Duffin — Thu, 12 Nov 2026 00:03:33 +0000

I tested three ways to do zero shot classification on 200 real customer messages.

This article covers how each of the three methods work, how they compare on accuracy, speed, and cost, and which one best fits your application.

What is zero shot classification?

Zero shot classification is when you give an AI model an input and a list of label names, and it picks the best fit with no training data and no examples.

Say a customer writes "Do you know the closest ATM?" and your labels are atm support, lost or stolen card, and request refund. A zero-shot classification model picks atm support without ever seeing a labelled example.

I personally work on zero shot classification every day building AI agents for online stores, and it's far from a waste of time. Online shoppers who experience a high quality chat experience spend significantly more money.

3 ways to do zero shot classification

There are three common ways to do zero-shot text classification today: small open-source models, large language models (LLMs), and System One models. All three take the same input (a piece of text and a list of labels). System One models like Jev are the new kids on the block.

1. A Hugging Face zero shot classification model

The Hugging Face zero-shot classification pipeline with facebook/bart-large-mnli is what most existing tutorials use. It turns each label into a sentence ("This example is atm support.") and checks whether the text backs that sentence up. It's free and runs on your own machine - which is a major drawcard.

import { pipeline } from '@huggingface/transformers';

const classify = await pipeline('zero-shot-classification', 'Xenova/bart-large-mnli');
const result = await classify(message, labels);
console.log(result.labels[0]); // the top label

2. Zero shot classification with an LLM

Zero-shot classification with an LLM is a prompt: here is the message, here are the labels, pick one. I tested Claude Opus 5.5 and Claude Haiku 5.5, and limited the answer to the label list so the model could not invent a new label. It's worth mentioning that this method doesn't always work as LLMs tend to get chatty, especially when something goes wrong.

const response = await client.messages.create({
  model: 'claude-haiku-5-5',
  max_tokens: 2000,
  system: `Pick the one label that best fits the message.\n${labels.join('\n')}`,
  messages: [{ role: 'user', content: message }],
  output_config: { format: { type: 'json_schema', schema: labelSchema } },
});

3. A System One model

A System One model is built to make fast decisions instead of writing text. Typesafe's Jev takes the message and one multiple-choice question, then returns the chosen label with a confidence score (from zero to one) for every option.

// options = { 'atm support': null, 'request refund': null, ... }
const { answers } = await client.systemOne({
  state: message,
  questions: { label: choice('Which label best fits this message?', options) },
});
console.log(answers.label.choice);

Testing zero shot classification

I tested all three methods on the same 200 customer messages from Banking77, a public set of banking support messages. I picked 10 labels at random, took 20 messages for each, and gave every model the label names and nothing else.

It's worth mentioning that I'm no machine learning engineer. I first learned to code by building a Next.js app with no experience. Each setup here is one you can easily copy yourself.

Method	Accuracy	Median time	Cost per 1,000 messages
bart-large-mnli	73.5%	1,264 ms	Free (own machine)
Claude Haiku 5.5	93.0%	1,211 ms	US$0.05
Claude Opus 5.5	95.0%	1,926 ms	US$1.96
Jev	92.0%	226 ms	US$0.016

Three things stood out:

The LLMs were the most accurate. Claude Opus 5.5 got 190 of 200 right, and Claude Haiku 5.5 got 186
Jev was the fastest and the cheapest to call. It answered in 226 ms, about five times faster than Haiku and eight times faster than Opus, while scoring 92%
The tutorial model trailed by about 20 points. bart-large-mnli scored 73.5% and took 1.3 seconds a message on my own machine

I also ran every message twice to see whether the answers held. bart-large-mnli changed none of its 200 answers, Jev and Opus changed one each, and Haiku changed three.

Limitations of zero shot classification

The main limitation of zero-shot classification is that the model only has your label names to go on. When two labels overlap, or a name is vague, the model guesses.

This exact limitation showed up in two places during my tests:

Overlapping labels. bart-large-mnli got all 20 atm support messages wrong. It filed "Which ATMs accept this card?" under card acceptance, which is a fair reading of those two names
Labels that hide their meaning. Every model put "Are my friends able to top up my account?" under receiving money. The dataset files it under topping up by card. The same six messages caught Opus, Haiku, and Jev

Neither is really the model's fault, and both are fixable. To prove this I ran a single tuning round on Jev. I checked the misses, wrote a one-line description for each label, and tested it on 200 fresh messages it hadn't seen yet.

Jev went from 93% to 99%, reaching LLM-level accuracy at a fraction of the time and cost, after one round of tuning.

How to choose the right zero shot classification method for your application

Choose your zero-shot classification method by what your application needs most: privacy, accuracy, or speed.

Use bart-large-mnli when the data cannot leave your machine or the budget is zero. Expect to check its work
Use an LLM when accuracy matters most and a second or two per message is fine. Claude Haiku 5.5 scored 93% for five US cents per 1,000 messages
Use a System One model like Jev when classification sits in front of every request, like routing live chat. Allow for one or two tuning rounds on your labels
Train your own model when you have thousands of labelled examples and the labels never change. This comparison of zero-shot vs fine-tuned models covers when that is worth it

Whichever you choose, test it on your own messages first. Two hundred is enough to show where your labels overlap.

UPDATE: Google released their own decisions API the day after I wrote this article. Here's what happened when I compared the Google Decisions API vs Jev.