DEV Community: Anushka Shukla

Building an LLM Router with Jev: A Practical Approach

Anushka Shukla — Sun, 11 Oct 2026 16:19:58 +0000

When building applications with LLMs, using one model for everything is a reasonable starting point.

But not every request needs the same level of reasoning.

Converting a list into JSON is very different from debugging a race condition in a distributed system. Sending both requests to an expensive model might work, but it may not be the most efficient approach.

What if we could evaluate a request first and then choose an appropriate model?

That is the idea behind LLM routing with Jev.

In this article, we'll explore a practical architecture for routing requests between model tiers, implementing fallback logic in Python, and measuring whether the approach actually improves costs.

1. The architecture

Imagine an application with three model tiers:

Route	Best suited for	Example
`small`	Simple, repetitive tasks	JSON formatting, field extraction
`standard`	Everyday requests	Summaries, editing, general questions
`advanced`	Complex reasoning	Difficult debugging, system design

The routing pipeline looks like this:

User Request
     |
     v
Jev Evaluation
     |
     v
Application Routing Policy
     |
     v
Selected LLM
     |
     v
Response

The important part is the separation of responsibilities.

Jev evaluates the request and provides structured decision outputs.
The application decides how to interpret those outputs, applies routing rules, checks availability, and handles fallbacks.
The selected LLM performs the actual task.

This separation makes the routing policy easier to test and modify independently of the model provider.

2. Understanding the routing decision

Jev supports typed questions such as choice, score, and noul. These can be used to represent decision-making logic with a shared state and defined options or rubrics.

For the application example below, imagine we normalize the evaluation result into this structure:

{
  "winner": "small",
  "distribution": {
    "small": 0.92,
    "standard": 0.06,
    "advanced": 0.02
  }
}

Note: This JSON is an illustrative application-level representation, not a claim about Jev's exact SDK response format. Map the actual documented output into your application's own schema.

In this example:

winner is the leading route.
distribution contains the normalized scores associated with the available choices.
The application's routing policy uses the winning option's score to decide whether to accept the result.

Conceptually:

routing_score = distribution[winner]

A threshold can help the application decide when to accept a routing decision.

For example, a threshold of 0.80 means the policy accepts a winning score of at least 0.80.

It does not mean the selected LLM has an 80% chance of answering correctly. That interpretation would require empirical calibration and validation.

3. Implementing the routing policy in Python

Let's assume the application has already normalized the evaluation result into the structure shown above.

We also maintain a dictionary of configured model routes:

available_routes = {
    "small": "provider-small-model",
    "standard": "provider-standard-model",
    "advanced": "provider-advanced-model",
}

The model identifiers are placeholders. Replace them with the identifiers supported by your LLM provider.

Now, let's implement route selection with explicit fallback handling.

from typing import Any

ROUTING_THRESHOLD = 0.80
DEFAULT_FALLBACK_ROUTE = "standard"


def choose_route(
    decision: dict[str, Any],
    available_routes: dict[str, str],
    fallback_route: str = DEFAULT_FALLBACK_ROUTE,
) -> str:
    """Select a configured route from normalized decision output."""

    if not available_routes:
        raise RuntimeError("No LLM routes are configured.")

    if fallback_route not in available_routes:
        raise RuntimeError(
            f"Configured fallback route '{fallback_route}' is unavailable."
        )

    winner = decision.get("winner")
    distribution = decision.get("distribution", {})

    if not isinstance(distribution, dict):
        return fallback_route

    score = (
        distribution.get(winner, 0.0)
        if isinstance(winner, str)
        else 0.0
    )

    valid_score = (
        isinstance(score, (int, float))
        and not isinstance(score, bool)
        and 0.0 <= score <= 1.0
    )

    if (
        valid_score
        and score >= ROUTING_THRESHOLD
        and winner in available_routes
    ):
        return winner

    return fallback_route

Here's what this implementation does:

Checks that at least one model route is configured.
Verifies that the designated fallback exists.
Accepts the winning route only when its score meets the threshold and the route is available in the configuration.
Uses the configured fallback when the decision is missing, invalid, or below the threshold.

Notice that the function does not silently select an arbitrary model if the fallback is missing. It raises an error instead.

In production, you should also validate the entire normalized input schema and handle provider health, rate limits, timeouts, user permissions, and model-specific constraints.

4. What happens when the decision is ambiguous?

Consider two illustrative cases.

Case A: No route meets the threshold

small:     0.20
standard:  0.35
advanced:  0.45

The leading option is advanced, but its score is below the configured threshold of 0.80.

Under our example policy, the application selects the configured fallback.

If the fallback is standard, the request goes to standard. If advanced is explicitly configured as the fallback, it goes to advanced.

The choice is determined by the application's policy, not by silently assuming that the leading option is always suitable.

Case B: One route clearly leads

small:     0.94
standard:  0.05
advanced:  0.01

Here, small meets the threshold. The application selects it if the route is configured.

These examples illustrate the mechanics of the routing policy. They do not establish how accurately Jev classifies real requests. That needs to be measured using representative evaluation data.

5. Connecting the policy to an application

The orchestration layer can remain separate from the routing function.

def handle_request(user_prompt, available_routes, model_client):
    # Application-specific integration:
    # Normalize the documented Jev output first.
    decision = evaluate_with_jev(user_prompt)

    route = choose_route(
        decision=decision,
        available_routes=available_routes,
        fallback_route="standard",
    )

    model_id = available_routes[route]

    return model_client.generate(
        model=model_id,
        prompt=user_prompt,
    )

evaluate_with_jev and model_client.generate are placeholders for your actual integration. This is an architectural example, not a drop-in implementation for a specific SDK.

Keeping evaluation, routing, and dispatch separate has a practical benefit: changing the model provider or adjusting the routing policy does not require rewriting the entire request-handling pipeline.

6. Does LLM routing actually save money?

Not necessarily.

The routing step introduces additional computation and latency. A smaller model may also produce an inadequate answer, leading to retries or escalation to a larger model.

To evaluate the trade-off, compare three approaches using the same representative dataset:

Approach	What it measures
Always use the advanced model	Baseline quality, cost, and latency
Use a simple rule-based router	Whether basic routing is sufficient
Use Jev-based routing	Whether decision-based routing improves the trade-off

Track these metrics:

Total cost: Include the routing step, model calls, retries, and escalations.
Task success rate: Measure whether the response meets task-specific correctness criteria.
End-to-end latency: Measure the full request lifecycle, not just the final LLM call.
Fallback rate: Measure how often the routing policy falls back.
Quality by task category: Ensure that savings on simple tasks do not come at the expense of difficult ones.

A useful first step is to run the router in shadow mode. Continue serving requests through the existing pipeline while evaluating proposed routes in the background. Compare the decisions against actual task outcomes before introducing automated routing into live traffic.

The metric I'd pay most attention to is cost per successfully completed task, rather than cost per token alone.

Conclusion

LLM routing offers a way to match model capability to request complexity. But the router is only one part of the system.

A robust implementation needs a clearly defined routing policy, explicit fallback behavior, endpoint validation, and benchmarks that include both cost and output quality.

Jev provides an interesting approach to structured decision-making in this architecture. Whether it produces meaningful savings depends on the workload and the results of the evaluation.

I'd benchmark it against a single-model baseline and a simple rule-based router before making any claims about cost reduction.

# Where Jev Fits in an Application

Anushka Shukla — Wed, 30 Sep 2026 19:06:56 +0000

Suppose you’re building a service that receives customer support tickets. Before assigning a ticket, your code needs to answer two questions:

Which team should handle it?
Does it need urgent attention?

You could ask a language model to write an assessment and then parse its response. Jev, TypeSafe AI’s decision model, offers another interface: send it the ticket as state, define the questions and their answer types, and receive structured answers.

For example, you might define the team as a Choice between billing, technical, and sales. Urgency could be a Noul, Jev’s probability-based answer to a yes-or-no question. Jev also supports Score questions for placing something on a defined scale.

The flow looks like this:

Ticket + defined questions
           ↓
          Jev
           ↓
Choice and probability-based answers
           ↓
Your application decides what to do

Your application still owns the rules. It might assign a ticket automatically when the team choice is clear, or put it in a review queue when the result is uncertain. Jev supplies a judgment; your code decides how much to trust it.

That separation is useful when the inputs vary too much for a few if statements, but the possible actions are known in advance. Think ticket routing, document classification, content review, or deciding which tool an AI agent should call next.

There’s a limit to keep in mind: structured output guarantees a usable shape, not a correct judgment. I would test it against real examples, track mistakes, and keep human review for decisions where errors matter.

TypeSafe AI’s quickstart shows the request format and a working example if you want to try it.

LLM Wiki: A Smarter Alternative to RAG

Anushka Shukla — Tue, 09 Jun 2026 14:56:16 +0000

Every developer I know has the same problem.
You read a great article. You save it. You take notes. You bookmark three more links. A month later, you need that knowledge again and you're starting from scratch, re-reading the same things, rediscovering what you already knew.
So you do what every sensible developer does: you build a RAG pipeline.
You chunk your documents. You embed them. You set up a vector database. You wire up a retrieval layer. Three days later, you have a system that answers questions from your documents sort of. Sometimes the chunks are too small. Sometimes the wrong document gets retrieved. Sometimes the answer is technically correct but misses the point entirely.
And the worst part? The system learns nothing. Every query starts from zero.
There's a better way. Andrej Karpathy co-founder of OpenAI, former Director of AI at Tesla published a deceptively simple idea in April 2026 that's been quietly spreading through the developer community ever since.
He calls it the LLM Wiki.

What's Wrong with RAG?
Before I explain LLM Wiki, let me be fair to RAG. It's not bad. For large enterprise document stores thousands of PDFs, legal contracts, customer records RAG is the right tool. It scales in ways that other approaches can't.
But most developers aren't building enterprise search engines. They're trying to get an AI to work intelligently with their knowledge their notes, their research, their team's documentation.
And for that use case, RAG has a fundamental flaw: it's stateless.
Every time you ask a question, the system re reads your raw documents from scratch. It retrieves chunks. It answers. It forgets. The next question, same thing. There's no accumulation. No synthesis. No memory of what it figured out last time.
It's like hiring a brilliant researcher who reads your entire library before answering every single question and then immediately forgets everything when you ask the next one.

The LLM Wiki Pattern
Karpathy's insight is almost embarrassingly simple once you hear it.
Instead of querying raw documents over and over, you have an LLM compile your sources into a structured wiki of plain Markdown files once. From then on, you query the compiled wiki, not the raw sources.
Think of it like this:

"Obsidian is the IDE, the LLM is the programmer, the wiki is the codebase."
— Andrej Karpathy

Your raw documents are source material. The LLM reads them and produces clean, structured, interlinked pages one per concept. Like Wikipedia, but for your personal knowledge.
When you have a new source, you add it and run the compilation again. The LLM updates the relevant pages, adds new ones, and notes any contradictions. The wiki grows. The knowledge compounds.

What a Wiki Page Looks Like
Here's an example of what the LLM produces a page for the concept "vector embeddings":
markdown# Vector Embeddings

Type: Concept

Related: [[RAG]], [[Semantic Search]], [[Neural Networks]]

Summary

Vector embeddings are numerical representations of text that capture semantic
meaning. Similar concepts produce vectors that are close together in space.

How They Work

Text is passed through a model which outputs a list of numbers the embedding.
These numbers encode the meaning of the text in a way that allows mathematical
comparison.

Where They're Used

RAG pipelines (retrieving relevant document chunks)
Semantic search engines
Recommendation systems

Limitations

Embeddings from different models aren't compatible
Quality degrades on very long texts
Require a vector database to query at scale

Sources

[[Attention Is All You Need Vaswani et al.]]
[[OpenAI Embeddings Documentation]] Clean. Structured. Cross-referenced. Written once, queryable forever.

RAG vs LLM Wiki: A Direct Comparison
RAGLLM WikiLearns over time? No YesInfrastructure neededVector DB + embeddings + retrieval pipelineA text editorQuery costHigh re reads docs every timeLow reads structured wikiCross-source synthesisWeakStrongBest forMillions of documentsPersonal / team knowledge

How to Try It in 30 Minutes
You don't need to write any code to try this. Here's the simplest possible version:
Step 1 — Gather your sources
Create a folder. Drop in 3–5 Markdown or text files articles you've saved, notes you've taken, documentation you reference often.
Step 2 — Compile the wiki
Open Claude.ai, upload your files, and paste this prompt:
Read these documents and create structured Markdown wiki pages for the key
concepts. For each concept, write a summary, list related concepts using
[[wikilinks]], and note the source. Output each page as a separate code block.
Step 3 — Save and query
Copy the generated pages into a folder. Open it in Obsidian (free). Now ask questions against your wiki instead of your raw documents.
That's it. No vector database. No embedding model. No retrieval pipeline. Just Markdown files and an LLM.

When to Stick with RAG
LLM Wiki isn't a replacement for RAG in every situation. Use RAG when:

You have thousands of documents the wiki would become too large for a context window
Your documents update constantly real time data needs real time retrieval
You need exact citation RAG can point to the precise chunk; a wiki synthesizes across sources

For everything else personal research, team knowledge bases, project documentation, learning notes the LLM Wiki pattern is simpler, cheaper, and produces better answers.

The Bigger Idea
What Karpathy identified isn't just a technical pattern. It's a shift in how we think about AI and knowledge.
Most AI tools treat your documents as static inputs things to be searched. The LLM Wiki treats your knowledge as something alive something that grows, connects, and compounds every time you add to it.
RAG answers questions. LLM Wiki builds understanding.
And honestly? That's the kind of AI tool I actually want to use.

Have you tried building a personal knowledge base with LLMs? Drop a comment below I'd love to hear what's working for you.

Want a deeper dive?
I wrote a more detailed version of this on Medium covering the full history behind the idea (including Vannevar Bush's 1945 Memex concept) and a step by step walkthrough for three different implementation paths.

Read the full article on Medium

If you find it useful, a follow there helps me write more content like this.