🚦 Auto-routing Claude models with Jev prompt classification

Originally published in Japanese on Zenn: Jevでプロンプトを仕分けして、Claudeのモデルを自動で切り替える

Goal 🎯

• Stop burning Opus on "hello" and Haiku on design decisions
• Let Jev (TypeSafe AI's classification model) classify each prompt, then automatically switch Claude Code's model and effort every turn

I built a Claude Code plugin that sends every prompt to Jev, picks a suitable model and effort, and applies them for that turn.

What is Jev? 🧩

You pass it text plus "descriptions of the choices," and it returns which choice fits, with probabilities.
You supply both the label names and the description text on every request. Those descriptions are all Jev uses to decide.

Input : state (text to classify) + criteria (label → description of that label)
Output: choice (selected label) + confidence + probabilities (for every label)

There are three players:

Player	Role
Jev	Picks one label from the descriptions you passed
Custom plugin	Builds the request, calls Jev, reads the response, sets the model
Claude Code	Sends the model request with the model and effort the plugin chose

⚠️ Note: Jev only provides a classification API. Callers have to hit the raw API themselves.

Unlike /model, this plugin does not touch conversation history or memory files. The only thing that changes is the model name and effort on each request.

How 🛠️

1. What we send to Jev

Here is the request the plugin builds , shown as curl:

curl -s https://api.typesafe.ai/v1/systemone \
  -H "Authorization: Bearer $TYPESAFE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Fix the typo in the README",
    "model": "jev-latest",
    "questions": {
      "tier": {
        "type": "choice",
        "instructions": "How much model capability does answering this coding-assistant prompt need",
        "criteria": {
          "micro":    "A greeting, a yes/no, a one-line factual question, or a short follow-up that needs no code reading",
          "low":      "A small well-defined edit or lookup in one or two files, a rename, a simple command to run, formatting",
          "medium":   "A feature or bug fix across several files, writing tests, explaining how part of a system works",
          "high":     "Debugging that needs investigation across modules, a refactor touching many files, planning a multi-step change, a normal code review",
          "critical": "Architecture or design decisions, migrations, state-machine changes, ambiguous multi-step tasks with trade-offs",
          "max":      "Fund-safety or security code, adversarial review, cryptography or key handling, anything where a subtle mistake loses money or leaks secrets"
        }
      }
    }
  }'

2. What comes back

This is the real response for the request above (state = "Fix the typo in the README"). HTTP 200, 0.17s, called once.

{
  "model": "jev-1.14.0",
  "answers": {
    "tier": {
      "type": "choice",
      "choice": "low",
      "confidence": 0.9,
      "probabilities": { "micro": 0.08, "low": 0.92, "medium": 0.0, "high": 0.0, "critical": 0.0, "max": 0.0 }
    }
  },
  "usage": { "input_tokens": 504, "output_tokens": 59 }
}

• choice: the label Jev picked. The plugin reads only answers.tier.choice; if the JSON is broken, it treats it as undefined.
• probabilities: probabilities for every label. Here low is 0.92, micro is 0.08, and the rest are 0.
• confidence and usage: the plugin ignores these.

Because Jev returned low, the plugin looks up ROUTES.low and sets haiku with effort high.

3. Deciding after the response

The label Jev returns is tier. The plugin maps tier through ROUTES to get the model and effort. Jev does not choose the model — the plugin does.

export const ROUTES = {
  micro:    { model: 'haiku',            effort: 'medium' },
  low:      { model: 'haiku',            effort: 'high'   },
  medium:   { model: 'sonnet',           effort: 'high'   },
  high:     { model: 'opus',             effort: 'medium' },
  critical: { model: 'opus',             effort: 'high'   },
  max:      { model: 'claude-fable-5-1', effort: 'high'   },
}

tier (label from Jev)	When Jev picks it (summary of criteria)	Model the plugin uses (ROUTES[tier])	effort
micro	Greeting, yes/no, one-line answer	haiku	medium
low	Small edit in 1–2 files, rename, formatting	haiku	high
medium	Multi-file feature or bugfix, writing tests	sonnet	high
high	Cross-module investigation, large refactor, normal review	opus	medium
critical	Design decisions, migrations, state-machine changes	opus	high
max	Fund/security, adversarial review, crypto and keys	claude-fable-5-1	high

const route = ROUTES[tier]    // e.g. "high" → { model: 'opus', effort: 'medium' }

// One exception: if we would stay on Haiku past 150k context tokens, bump to Sonnet
if (route.model === 'haiku' && contextTokens > 150_000) return ROUTES.medium
return route                   // rewrite this turn's request to route.model / route.effort

If the label is missing from the table, or the Jev call fails, the plugin changes nothing — same behaviour as with no plugin at all.

Result ✅

The routing result shows up under the prompt almost instantly, and it feels great. It doesn't interrupt the UX at all.
The accuracy seems solid.
Simple tasks now land on Haiku or Sonnet, so the stress of picking a model yourself goes away and the workflow feels more efficient.

Going forward, I also think this kind of classification could help with things like prompt-injection defense and other security use cases.

As a side note, I also drew on the conversation in mozaic.fm ep.218.

Comments

[Cristian K]

Nice separation between classification and the route table. A practical guardrail is to log the selected tier/model and fallback reason (without prompt text), and keep the current model when confidence is below a configurable threshold; that makes tuning easier without silently downgrading hard tasks. Have you tried a confidence threshold yet?

[Toshiya Matsumoto]

Thanks for your advice. None of them were prepared yet because I had created it casually but both do make sense for real use case and improvement.
I'll add them for sure.

[Pierre-Laurent Medori]

Really like that the classifier's whole config is basically six lines of plain English. You want different behaviour, you edit a line, no training set.

I'd try one case though: plan a migration on Opus, then type "ok, go ahead". If state is only the current prompt, Jev can't see the plan behind it, so that could read as your micro ("a short follow-up that needs no code reading") and the turn that does all the work lands on Haiku. Do you pass anything from the previous turn, or has it not bitten you yet?