DEV Community

Rinat Amanbekov
Rinat Amanbekov

Posted on

An LLM as an if statement: self-hosted, Jev-style decisions on vLLM

#ai #python #llm #opensource

A request lands in a support queue:

"I cannot log in to my account after changing my password."

Before anything else happens, the pipeline has to answer one small question: which queue gets it?

curl -s http://127.0.0.1:8000/semantic-if/v1/decide \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": {
      "client": "Acme LLC",
      "request": "I cannot log in to my account after changing my password"
    },
    "question": "Which queue should handle this request?",
    "options": [
      {"id": "access", "description": "Account access"},
      {"id": "billing", "description": "Payments and invoices"},
      {"id": "other", "description": "Other or not enough information"}
    ]
  }'
Enter fullscreen mode Exit fullscreen mode

The answer, trimmed:

{
  "choice": "access",
  "confidence": 0.9958,
  "margin": 0.9926,
  "low_confidence": false,
  "probabilities_by_option": {
    "access": 0.9958,
    "billing": 0.0010,
    "other": 0.0032
  },
  "input_tokens": 150,
  "request_seconds": 0.172
}
Enter fullscreen mode Exit fullscreen mode

0.17 seconds on a 27B model. The model didn't write a single word. There is no JSON to parse or repair, and if the model had been unsure, the option probabilities would have shown it.

This is semantic-if. You give it a state (text or JSON), one question and 2–16 mutually exclusive options. It returns the chosen option and a probability distribution over the options. MIT licensed, runs on your own GPU: https://github.com/rinat-amanbekov/semantic-if

Why I built it

On September 15, TypeSafe announced Jev, a model that doesn't generate text at all. You define the question and the allowed answers at call time, and it returns a typed answer with probabilities. That is the exact shape of many decisions inside the agents and pipelines I build. But Jev is closed and hosted, while a lot of my work runs on local models.

SemIf showed that a similar interface works with open models: instead of generating an answer, read the probabilities of the option labels directly from the next-token logprobs. SemIf is a research project, with scripts that load the model on a local GPU. I needed something I could deploy: behind the vLLM and LiteLLM infrastructure we already run, callable from agents over MCP, and shared between teams where every caller uses their own key.

So semantic-if is an independent implementation of SemIf's direct method, packaged as a Python library, a CLI, a REST API and an MCP server. It is not affiliated with TypeSafe and does not reproduce Jev's model.

How it works

  1. The decision {state, question, options} is rendered into SemIf's direct-options-v1 prompt, byte for byte (a SHA-256 check guards it). The options are labeled A, B, C…, and thinking is disabled.
  2. The fast path is one request to vLLM's /v1/completions: 
{
  "model": "qwen-3-8-27b",
  "prompt": "<the rendered ChatML prompt>",
  "max_tokens": 1,
  "temperature": 0,
  "logprobs": 32,
  "allowed_token_ids": [32, 33, 34],
  "return_tokens_as_token_ids": true
}
Enter fullscreen mode Exit fullscreen mode
  1. The raw logprobs of the labels go through a float64 softmax over the candidate options. That's the result: a probability distribution over the provided answers, plus confidence, margin and low_confidence.

In the common case that's one prefill, one decoding step and nothing to parse.

The method itself is almost trivial. Most of the implementation work went into the edges:

  • Labels are matched by token ID, never by token text, so two tokens that render the same way can't be confused.
  • If a label falls outside the returned top-k, semantic-if retrieves it with a follow-up request instead of silently treating its probability as zero.
  • The prompt is never truncated: a prompt that's too long is an error. The server's usage.prompt_tokens must equal the local tokenizer's count, so a tokenizer mismatch fails loudly instead of quietly changing the decision.
  • Input containing model control tokens such as <|im_end|> or <think> is rejected, so a caller can't forge a turn boundary.
  • Every input row returns either a decision or an explicit error, in order. Nothing is silently dropped.

Why not just ask for JSON?

You can, and it works. But generation is sequential: every token of {"queue": "access"} is another decoding step. And you don't get useful option probabilities for free. You can ask the model to add "confidence": 0.96 to its JSON, but that number is just more generated text.

SemIf measured the difference on a 4B model: generating a 21-value JSON array took about five times longer than reading the same 21 decisions directly from the logits. For a lot of agent decisions I don't need the model to explain itself. I need it to choose.

Using it

As a library:

from semantic_if import Decision, classify

row = {
    "id": "route-1",
    "state": "I can't log in after changing my password",
    "question": "Which queue should handle this request?",
    "options": [
        {"id": "access", "description": "Account access"},
        {"id": "billing", "description": "Payments and invoices"},
        {"id": "other", "description": "Other or not enough information"},
    ],
}

for result in classify([row], "qwen-3-8-27b"):
    if isinstance(result, Decision):
        print(result.choice, dict(zip(result.option_ids, result.probabilities)))
    else:
        print("error:", result.kind, result.error)
Enter fullscreen mode Exit fullscreen mode

As a service: POST /v1/decide for one decision and POST /v1/classify for batches of up to 32. An MCP server exposes two tools, semantic_decision and semantic_decisions, which agents such as Claude Code or OpenCode call directly or through the LiteLLM MCP gateway.

The service holds no API key of its own. Every caller sends their own LiteLLM key, so LiteLLM applies that caller's limits and budgets. The key never reaches responses, logs or metrics: logs only contain an 8-character fingerprint.

Numbers

Setup: Qwen3.8-27B in FP8 (Qwen/Qwen3.8-27B-FP8, served as qwen-3-8-27b) on vLLM, one NVIDIA RTX PRO 6000 Blackwell. This was not a clean benchmark box: the server carried other live traffic at the same time.

Quality is balanced accuracy averaged over task families, the same metric SemIf uses:

Set semantic-if, Qwen3.8-27B FP8 SemIf, Qwen3.5-4B BF16
SemIf authored144 0.955 0.813
SemIf perturbations108 0.981 0.766
ru_custom (90 decisions, Russian) 0.931–0.942 —

The SemIf column comes from its README and uses a smaller model on different hardware, so this is not an apples-to-apples benchmark. It shows what a larger model does with essentially the same method and datasets. It does not show that my implementation is smarter than SemIf.

Besides SemIf's English sets, I tested semantic-if on ru_custom, our own set of 90 decisions in Russian, to see whether the approach holds up outside English.

Speed

  • short prompts (~150 tokens): 8.6 decisions/s at concurrency 1, 30/s at concurrency 8;
  • a cold prompt of ~1,840 tokens: 0.46 s median;
  • through the service in Kubernetes: 0.15–0.3 s per decision; 12 concurrent requests all succeeded, the slowest in 0.74 s.

Are the probabilities useful?

This is the part I cared about most. The values semantic-if returns are normalized next-token probabilities over the options you supplied. That does not automatically make them calibrated estimates of whether the answer is correct, so I checked them on the same 342 labeled decisions:

  • The expected calibration error (ECE) is 0.026.
  • 291 decisions had confidence of 0.9 or higher, and not one of them was wrong.
  • The default low_confidence threshold of 0.8 flags 35 of the 342 decisions and catches 14 of the 16 errors. The two it misses both require arithmetic.
  • Temperature scaling didn't help: it barely changed the ranking of the confidences, and the fitted temperature didn't transfer between sets.
  • Reversing the option order for 36 decisions changed two answers. In both cases the wrong answer had confidence below 0.8, so it was already flagged.

For comparison, AnyJev from Nokia Applied Research adds calibration layers on top of this kind of readout and reports a raw ECE of 0.24–0.33 for Qwen3-8B. On my 27B and these three datasets, I haven't seen enough evidence to justify adding those layers. The obvious limitation: all three datasets are small and hand-written, not real production agent traffic.

In practice, low_confidence means don't act automatically: ask a human, gather more facts, or take the conservative path.

Three lessons

1. One token leaves no room for arithmetic

Every error on the Russian set required a computation: counting days, checking share thresholds, summing invoice lines. The labels were checked by two independent annotators, so these were model errors, not label noise.

Don't ask the model to calculate the facts and make the semantic decision in a single next-token step. Compute deterministic facts in code:

{
  "invoice_date": "2026-08-01",
  "today": "2026-08-22",
  "overdue_days": 21
}
Enter fullscreen mode Exit fullscreen mode

Then let the model decide what those facts mean. Code does arithmetic, the LLM judges semantics.

2. In my vLLM setup, logprobs changed with batching

At concurrency 1–2, repeated runs matched bit for bit. At concurrency 4 and above, on a server shared with other traffic, probabilities moved by as much as 0.17, and in some runs 1–2 answers out of 144 flipped. The semantic-if client can't fix that.

For reproducible decisions, vLLM has a batch-invariant mode (VLLM_BATCH_INVARIANT=1), which I haven't verified yet with this FP8 Qwen3.8 setup. The alternatives are a dedicated instance, or low concurrency plus an explicit tolerance for numerical noise.

For ordinary generation, tiny logit changes rarely matter. For a system whose output is literally derived from those logits, they do.

3. Prefix caching on a hybrid model works in blocks

On my Qwen3.8 + vLLM configuration, the prefix cache works in 800-token blocks, and hits start only from the third request with the same prefix: the first two get nothing from the cache. For a ~1,850-token prompt, that means just 800 tokens are reused.

So shared mode (several questions about the same state) doesn't buy much when the state is a short support ticket. It pays off when the shared state is thousands of tokens long. The exact behavior depends on the model and the vLLM version, so treat these numbers as an observation from this setup, not a general rule.

Try it

With an NVIDIA GPU (24 GB, or 16 GB with FP8 quantization):

git clone https://github.com/rinat-amanbekov/semantic-if.git && cd semantic-if
cp .env.example .env   # set LITELLM_MASTER_KEY, and LLM_API_KEY to the same value

# semantic-if + LiteLLM + vLLM with Qwen3.5-9B; the first start downloads ~19 GB
docker compose up -d --wait
curl -s http://127.0.0.1:8000/semantic-if/health
Enter fullscreen mode Exit fullscreen mode

The Compose stack is the newest and least tested part of the project. It serves Qwen3.5-9B so the default setup fits on a single consumer GPU, but so far it has only run against a fake vLLM in automated tests, not on a real GPU, and I haven't benchmarked the 9B yet.

If you already run vLLM behind LiteLLM, skip Compose and point .env at it:

LITELLM_BASE_URL=http://localhost:4000/v1
LLM_API_KEY=sk-...
LLM_MODEL=qwen-3-8-27b
Enter fullscreen mode Exit fullscreen mode

Then run the CLI on the sample decisions:

uv sync --extra test
uv run semantic-if -i tests/data/decisions.jsonl -o runs/decisions.jsonl --force
Enter fullscreen mode Exit fullscreen mode

How it was built

I built semantic-if spec-first. The first thing I wrote wasn't Python, it was the contract:

  • the prompt must be byte-identical to SemIf's reference prompt;
  • nothing may be silently truncated, and no input row may silently disappear;
  • tokenizer mismatches must fail loudly;
  • a caller's API key must never appear in logs or responses;
  • every stage has explicit acceptance criteria.

A coding agent implemented the code and tests against that spec. I reviewed the behavior, ran the benchmarks on our server, investigated the failures and decided what was good enough to ship.

I think this distinction matters. Typing the implementation wasn't the hard part. Deciding exactly what the system is allowed to do, what it must reject and how to know when it's wrong took much more work.

One acceptance criterion still fails: the spec asked for repeated runs to agree within 1e-3, and a shared vLLM instance doesn't give me that. I didn't quietly weaken the criterion after seeing the result. It's marked as not met in the README.

What's next

Two things: benchmark the Qwen3.5-9B configuration, and choose a better default low_confidence threshold. At 0.9, all 16 errors in the current evaluation are caught, but 35 correct decisions get flagged instead of 21. That trade-off matters to me more than squeezing another point out of raw accuracy: how many uncertain decisions can I catch before the system starts bothering a human too often?

The coding agent may have missed something. If you try semantic-if and find it, open an issue or a pull request: https://github.com/rinat-amanbekov/semantic-if

Top comments (0)