Five Labelled Examples Match Jev. Then I Tried Tanglish.

"Online la oru item order panni romba naal aachu, aana innum varave illa. Enakku podhum! En money-a thirupi kudunga, indha aalungaluku naan pay panna maaten."

(I ordered something online ages ago and it still hasn't come. I've had enough! Give me my money back, I'm not paying these people.)

That's a Banking77 refund request, rewritten in Tanglish (Tamil and English in one message) by a Claude agent, the way a lot of us actually type to a bank. Cosine similarity filed it under supported_cards_and_currencies. A classifier trained on 35 English examples per intent got it wrong in 2 of 3 runs. TypeSafe's Jev, a model that picks one option from a list you give it, filed it under request_refund with confidence 0.98.

I expected Jev to win everywhere. It didn't. On clean English, five labelled examples per category let a free classifier match Jev. On Hinglish, Tanglish and trick questions, 35 English examples didn't close the gap, even with a multilingual embedder.

I couldn't find a write-up that tests Jev on how Indian users actually type, so this is one: the numbers, a router you can copy, and a rule for when five examples are enough. Every answer is in the repo.

The contest in 30 seconds

The job: route a bank customer's message to 1 of 77 intents from Banking77 (card_arrival, declined_card_payment, request_refund, ‚Ķ). Five contenders:

Method	Compares the message with	Labelled examples
Cosine	each intent's one-line description, both as embeddings (vectors where similar meanings sit close)	0
Jev	the same 77 descriptions, as a list of options	0
Cross-encoder	each description, read together with the message	0
Cosine voting	past labelled messages; the 5 nearest vote	1 to 35 per intent
Classifier	logistic regression on the same past messages	1 to 35 per intent

Think of two clerks: a fast, free one (cosine, voting, the classifier) and a careful one, Jev, who reads every option. When is the fast one enough?

Jev returns one option and a probability for every option, not text, so there's nothing to parse. The whole call:

from typesafe_sdk import Choice, RetryPolicy, TypeSafeClient

client = TypeSafeClient(api_key="public", base_url="https://opencode.ai/zen",  # free route
                        retry=RetryPolicy(max_retries=0))  # retries count against its daily cap
question = Choice(
    instructions="Which banking support intent does the customer_message express? "
                 "Pick the single best match.",
    criteria=descriptions,  # {"card_arrival": "The customer ordered a card and it has not arrived yet, ...", ...} x 77
)
response = client.system_one({"customer_message": text}, {"intent": question}, model="jev-1.13-free")
answer = response.choices["intent"]
answer.choice, answer.confidence  # ("request_refund", 0.98)

Every call sends all 77 descriptions, about 2,400 input tokens.

Round 1 · Clean English: five labelled examples match Jev

With zero examples, Jev beats cosine by 10 points across 1,001 test messages.

A logistic-regression classifier on the embeddings matches Jev at 5 examples per category: 385 labelled messages in total. Its 0.9-point edge is inside Jev's margin of error, so call it a tie. Cosine voting needs 10.

One example is worse than one sentence. A single labelled message per category scores 63%. A one-line description scores 73%.

The cross-encoder, which reads message and description together, did no better than cosine. Pasting the same 5 examples into Jev's descriptions reached 90.4% on 249 messages: a borderline lead over the classifier, at 3.7 times the tokens.

On clean English, the fast clerk catches up with five examples. Then I changed the language.

Round 2 · Hinglish and Tanglish: the embeddings fall apart

Your users won't type like your test set. Hinglish mixes Hindi and English, Tanglish mixes Tamil and English, both in Latin letters. Claude agents rewrote 144 test messages (all 77 intents) as natural chat in each, keeping words like card, top up and PIN in English, the way people type them. Other agents checked every rewrite; no native speaker has yet.

	Message	Cosine	Classifier, 35 examples (3 random draws)	Jev
English	"I ordered something ages ago online but it's simply not arriving… Give me back my money…"	✗ card_arrival	✓	✓ 0.99
Hinglish	"Maine kaafi time pehle online kuch order kiya tha… Mere paise wapas do…"	✗ getting_virtual_card	✗ in all 3 draws	✓ 0.99
Tanglish	"Online la oru item order panni romba naal aachu… En money-a thirupi kudunga…"	✗ supported_cards_and_currencies	✗ in 2 of 3 draws	✓ 0.98

One message, picked to show the pattern. All 144:

"Isn't that just your English-only embedding model?" Fair: bge-small-en is English-only. So I re-ran cosine voting and the classifier with bge-m3, a multilingual model about 17 times bigger (the dashed line, the better of the two). It helps, by 3 to 6 points. Jev stays clearly ahead on Hinglish; on Tanglish, 144 messages make the gap borderline. What I haven't tested is training on Hinglish or Tanglish examples. That's next.

Round 3 · Trick questions: more examples don't help

Next, 105 new messages built to fool keyword matching. Three agents wrote them and three more labelled each one blind; all 105 passed, which may mean the check was lenient.

Message (and the right answer)	Cosine	Classifier, 35 examples	Jev
Negation: "It's not contactless that's the problem, tapping my card works fine. It's Google Pay that refuses to add my card." → apple_pay_or_google_pay	✗ contactless_not_working	✗ same, 86–91% sure	✓
Decoy: "An ATM in Lisbon swallowed my card last week, but that's been dealt with and a replacement is ordered. How long does delivery usually take?" ‚Üí card_delivery_estimate	‚úó card_swallowed	‚úó same, all 3 draws	‚úì
Indirect: "The four numbers I tap in at shops are my old house number and I'd like something different. How do I swap them?" ‚Üí change_pin	‚úó edit_personal_details	‚úó same, all 3 draws	‚úì

Jev got all 105 right. These were written to fool surface matching, so read 100% as "Jev doesn't fall for this", not "Jev is perfect".

The classifier stayed at 64% with 5 examples per intent or 35, and reached 76.5% with the multilingual bge-m3. Cosine took the planted bait 23 times.

My favourite detail: the bug was in my description

Jev's worst category on English was get_physical_card: 0 out of 13. Cosine's too.

As a fairness rule, I wrote every description from the label name alone. Mine said: "The customer asks how to get their physical card." Then I read the messages:

"Where is my PIN number located?" · "Is PIN delivered separately?" · "I just got my new card but am not sure how to check its PIN."

In Banking77, get_physical_card really means "where's my PIN?" (105 of its 106 training messages mention one). Jev sent all 13 to change_pin, 9 of them at confidence 0.95 or higher. I haven't re-run it fixed: the cosine, cross-encoder and Jev numbers carry the bug, while voting and the classifier never read descriptions.

Your descriptions are your prompt. Read five training messages per category before you write one.

When to trust an answer, and when to ask Jev

Confidence works as a filter. On English, Jev's answers at 1.0 (half of them) are 98% right, and the classifier's own probability works just as well. But on Hinglish and Tanglish only about 1 in 10 classifier answers clear 0.9: it knows it's lost. So the fast clerk answers, and passes the odd letter to the careful one:

Let the classifier answer. When its confidence is below 0.7, ask Jev.

from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
embed = lambda texts: model.encode(texts, normalize_embeddings=True)
clf = LogisticRegression(C=100, max_iter=5000).fit(embed(examples), labels)  # 35 per intent; C=1 underfits

def route(text: str) -> str:
    probs = clf.predict_proba(embed([text]))[0]
    if probs.max() >= 0.7:
        return clf.classes_[probs.argmax()]
    answer = client.system_one({"customer_message": text}, {"intent": question},
                               model="jev-1.13-free").choices["intent"]
    return answer.choice  # client and question: from the Jev call above

Measured on Round 2's 144 messages and Round 3's 105, with 35 examples per intent. The English column is the 144 originals, so Jev scores 86.1% there, not 83.3%:

	English (144)	Hinglish (144)	Tanglish (144)	Trick (105)
Jev alone	86.1%	85.4%	77.1%	100%
Classifier alone	93.1%	56.2%	52.8%	64.1%
Classifier first, Jev below 0.7	92.6%	84.5%	78.7%	91.7%
Messages sent to Jev	14%	73%	75%	54%

On English and code-mixed text it stays within about a point of the better single method, or above it on Tanglish. That saves 86% of Jev calls on English but only about a quarter on Hinglish and Tanglish, where the classifier is rarely sure. Trick questions are the exception: the classifier was 86–91% sure of its wrong Google Pay answer, so the 0.7 gate let it through.

I didn't tune 0.7; it's the middle of a fixed 0.9 / 0.7 / 0.5 grid. Your classifier's probabilities will differ, so sweep 0.5 to 0.9 on a few hundred held-out messages, as TRIAGE.md does. It also shows a simpler rule (trust answers when voting and the classifier agree) that wins on clean English and breaks on Hinglish.

So which one should you use?

flowchart TD
  Q1{"Labelled messages<br/>per intent?"} -- "fewer than 5" --> J["Jev<br/>(cosine on descriptions if it must be free)"]
  Q1 -- "5 or more" --> Q2{"Only clean English?"}
  Q2 -- yes --> E["Classifier on embeddings"]
  Q2 -- "no, or not sure" --> H["Classifier first;<br/>ask Jev below 0.7<br/>(measured with 35 per intent)"]
  J --> P["Jev's answer below 0.95?<br/>Send it to a person<br/>(Jev alone: 3 in 10 English, 6 in 10 Tanglish)"]
  H --> P

For scale: the classifier answers in 7 ms on a laptop CPU (43 ms with bge-m3), for $0. Jev takes 0.76 s a message, median round trip from India, and about $0.10 per 1,000 messages at TypeSafe's list price of $0.042 per million input tokens.

What's missing, honestly

• The classifier only saw English examples; training on code-mixed examples is untested.
• The probe sets are small: 105 to 144 messages, roughly ¬±6 to ¬±9 points.
• Banking77 is public, so Jev may have seen it in training.
• One cross-encoder, trained for search; larger rerankers may do much better.
• Jev went through a free third-party route: checked against TypeSafe's published answers but not proven identical, capped near 250 calls a day, and down for about four days in late October. TypeSafe's own signups were paused when I checked. Latency is from one laptop in India.

Steal this, then run it

Needs Python 3.12 or newer; the install pulls in PyTorch.

git clone https://github.com/Madheshvivekanandan/jev-vs-cosine && cd jev-vs-cosine
python3.12 -m venv .venv && .venv/bin/pip install -e .
.venv/bin/python -m app.main triage && cat results/TRIAGE.md   # no key, no API calls

Every Jev answer is committed, so the reports rebuild for free. To call Jev yourself, .env.example points at OpenCode Zen's free route, a third party that verify-route checks against TypeSafe's published answers before you spend quota. Install only the official typesafe-sdk; lookalike packages exist.

If you take three things from this post:

1. Write descriptions from real messages, not label names. One wrong sentence zeroed a whole category.
2. Test on how your users actually type. English said five examples were enough. Tanglish disagreed.
3. Route on confidence, and pick the threshold on your own data. Let the cheap model answer what it's sure of; escalate the rest to Jev or to a person.

Run it on your own users' messages and tell me where Jev breaks. If you speak Hindi or Tamil and a rewrite reads wrong, I genuinely want the issue.

---

Previously: The Coding Agent That Can't Grade Its Own Homework, on why the agent that writes the code shouldn't be the one that grades it.

All illustrations in this post were generated with ChatGPT; the charts were drawn from the repo's result files.

Banking77 is by PolyAI (CC BY 4.0). Not affiliated with TypeSafe AI.