GPUStack v2.3.0rc2 introduced a preview of an advanced routing capability worth testing hands-on: selecting a model through the Jev decision service.
When multiple models are available in GPUStack, an application would normally need to specify which one to call. With decision routing enabled, the application can always use the same entry point. The gateway then chooses a generation model based on the task content and the policy defined by the administrator.
This article follows that path end to end. We first introduce decision models, then deploy two private models in GPUStack, connect DeepSeek and Jev, and finally use tasks of different difficulty levels to verify the resulting model selections.
01 What Is a Decision Model?
Generation models answer questions. Decision models provide evidence for choosing a model.
For example, “Translate Good morning into Chinese” and “Design a cross-region payment system” require different capabilities. A smaller model may be sufficient for the first task, while the second may require a stronger model to reason about failures and consistency.
Jev receives the request content, candidate models, and their capability descriptions. It returns structured selection evidence through the TypeSafe Choice interface. The GPUStack gateway uses that evidence to choose a model and then forwards the request to the selected generation service. The final answer still comes from Qwen3 or DeepSeek.
The key to configuring decision routing is therefore to describe clearly which tasks each candidate model is suited for. A decision model does not establish business-quality standards on behalf of the administrator. The closer the capability descriptions are to real workloads, the easier the selections are to validate.
02 What Problem Does Jev Solve in the Gateway?
GPUStack model routing can place multiple targets behind one entry point. Fixed weights answer the question “How much traffic should each model receive?” and are useful for canary releases and traffic distribution. Jev decision routing goes one step further and asks “Which model is appropriate for this request?”
For an assistant that handles both text and code, we want the following division of work:
| Task type | Candidate in this policy |
|---|---|
| Short translation, field extraction, factual questions, and basic arithmetic | Private Qwen3-8B |
| Constrained algorithm implementation, SQL, and localized code debugging | Private Qwen3-32B |
| Complex concurrent code and cross-region system design | External DeepSeek |
The application uses one route name, while the administrator maintains this division of work in GPUStack. Candidates can be added or task boundaries adjusted without changing how the application calls the route.
This is the gateway-level value of a decision model: centralize task classification in the routing policy so that different models handle the requests best suited to them. Whether this improves quality or reduces cost still has to be validated against real workloads. This article first checks whether task routing follows the configured policy.
03 Prepare the Models and External Services
First, prepare the three candidates that will generate answers. Then connect Jev as the service responsible for making the decision.
Deploy Qwen3-8B and Qwen3-32B-W8A8 through GPUStack. Both models should show one available replica out of one. Send an inference request to each model to confirm that both can answer successfully.
Next, add DeepSeek under Providers. Select the DeepSeek type, enter the API key, and test connectivity. The generation model used in this test is deepseek-flash.
Add Jev as another provider. Select TypeSafe Decision Service (Jev), set the endpoint to https://api.typesafe.ai, and enter the TypeSafe API key. This test uses jev-latest as the decision model.
At this point, the responsibilities are separate. Qwen3 and DeepSeek are route targets and generate answers. Jev is the decision service and provides evidence that helps the gateway choose a target.
04 Configure a Decision Route
Create a route named demo-smart and add the two private deployments and the DeepSeek model as regular route targets. The application only needs to request demo-smart.
Under Basic Information, select Policy, enable Decision Service Routing, choose the Jev provider routing-jev, and enter jev-latest as the decision model.
In this test, all three ordinary traffic weights are set to 0, while the decision contribution weight is set to 20 so that the effect of Jev is easy to observe. The decision contribution weight is a relative influence on the score. It is not a traffic percentage. Session stickiness and the least-outstanding-requests strategy are not combined with it in this test.
Next, define the Decision Instructions and Model Evaluation Criteria. Together, they answer two questions: What is the overall selection principle, and what tasks is each candidate suited for?
| Setting | Core content used in this test |
|---|---|
| Decision instructions | Prefer the least expensive candidate that can complete the task. Send simple tasks to 8B, medium-complexity tasks to 32B, and complex tasks to DeepSeek. |
| 8B capability description | Suitable for short translation, extraction, factual questions, short summaries, and basic arithmetic. The service context budget is 8,192 tokens. |
| 32B capability description | Suitable for SQL, algorithm implementation, and localized debugging. The service context budget is 16,384 tokens. |
| DeepSeek capability description | Suitable for complex reasoning, concurrent code with tests, and distributed architecture design. |
The keys in the capability descriptions must match the actual candidate model names: route-local-qwen3-8b, route-local-qwen3-32b, and deepseek-flash. Do not use route aliases or provider names because they cannot be mapped to the candidates.
This capability split is a policy to be validated. The assumption that 8B is the lower-cost tier, 32B is the middle tier, and DeepSeek is the higher tier is also part of the configuration. This test does not calculate hardware amortization or API charges.
05 Validate Three Model Choices Through One Route
Every test in this section uses the same route, demo-smart. The nine test inputs are nine questions, not nine routes. Only the question content changes. The model field remains unchanged while we observe whether the gateway follows the configured division of work and selects 8B, 32B, or DeepSeek.
Step 1: Write Down the Expected Results
Start with three representative inputs. A short translation is expected to go to 8B, constrained interval-merging code to 32B, and cross-region payment-system design to DeepSeek. Recording the expectations first prevents us from explaining the choice only after seeing the answer.
Select demo-smart in the GPUStack Playground to confirm that the route can complete a conversation.
Step 2: Send the Request and Retain the Headers and Answer
Use the API for the formal test so that the selected model can be traced. First, create a GPUStack inference key and save the following content as request.json:
{
"model": "demo-smart",
"messages": [
{
"role": "user",
"content": "Translate ‘Good morning’ into Chinese. Return only the translation."
}
],
"max_tokens": 128,
"temperature": 0,
"stream": false,
"chat_template_kwargs": {
"enable_thinking": false
},
"thinking": {
"type": "disabled"
}
}
Send the request to the gateway, saving the response headers to response.headers and the answer to response.json:
curl -sS http://localhost:9080/v1/chat/completions \
-D response.headers \
-o response.json \
-H "Authorization: Bearer $GPUSTACK_API_KEY" \
-H 'Content-Type: application/json' \
--data-binary @request.json
For the next test, change only the question in messages and increase max_tokens for code or longer answers. Keep model set to demo-smart. The interval-merging test uses a 1,024-token output budget, and the system-design test uses 1,536 tokens. Replace the gateway address with the address of your own deployment.
Step 3: Find the Actual Model by Request ID
Read x-gpustack-request-id from the response headers, then use it to locate the corresponding Higress log entry:
REQUEST_ID=$(awk 'tolower($1)=="x-gpustack-request-id:" {gsub("\r", "", $2); print $2}' response.headers)
test -n "$REQUEST_ID" || { echo 'response header does not contain a request ID'; exit 1; }
kubectl --context gateway-cluster \
-n inference-gateway logs deployment/higress-gateway --tail=500 \
| rg --fixed-strings "$REQUEST_ID"
Replace the context and namespace with your own gateway configuration. In the matching log entry, inspect upstream_cluster to identify the forwarding target and ai_log.model to confirm the model name. In this test, model-2-6.static maps to 8B, model-3-7.static maps to 32B, and provider-1.dns maps to DeepSeek. These identifiers change with the deployment.
The evidence is the request with one request ID, its response, and the matching access log. An HTTP 200 status only confirms that the request succeeded. It does not prove which model was selected.
Step 4: Compare the Measured Results
The next three figures show the request, response headers and body excerpt, and matching gateway log fields from the archived API test. Each figure retains the request ID. The entry point remains the same, while the model that generates the answer changes.
Short translation → 8B. The request asks for a short translation of “Good morning.” The model returned the expected translation, while the log records route-local-qwen3-8b and upstream model-2-6.static. The figure localizes the body excerpt into English for this edition while preserving the request ID and routing evidence.
Constrained code → 32B. The task asks for interval merging without modifying the input and requires invalid intervals to be rejected. The API returned Python code, while the log records route-local-qwen3-32b and upstream model-3-7.static.
Cross-region system design → DeepSeek. The task asks for an analysis of payment callbacks, inventory deductions, network partitions, and recovery reconciliation. The API returned a design explanation, while the log records deepseek-flash and upstream provider-1.dns.
In addition to these three representative inputs, we tested factual questions, extraction, arithmetic, SQL, and other tasks. All nine inputs went through demo-smart:
| Test input | Expected model | Actual model |
|---|---|---|
| Capital of France | 8B | 8B |
| Short translation of Good morning | 8B | 8B |
| Extract a city from text | 8B | 8B |
| Calculate 2+2 | 8B | 8B |
| Constrained interval-merging code | 32B | 32B |
| PostgreSQL aggregation query | 32B | 32B |
| Analyze and fix an asynchronous-lock deadlock | 32B | 32B |
| Cross-region order-payment system design | DeepSeek | DeepSeek |
| Asynchronous LRU cache with tests | DeepSeek | DeepSeek |
All nine requests returned HTTP 200 with stop as the finish reason, and every actual model matched the expectation. This verifies that the configured task split can be implemented behind one entry point. The translation, interval-merging, and system-design tests were each repeated three times as non-streaming requests, and the model selection remained stable. Streaming versions of all three tasks also selected the same models.
Selecting the Right Model Is Not the Same as Accepting Its Answer
To determine whether this division of work had a concrete basis, we sent the same translation, extraction, arithmetic, and interval-merging tasks to all three candidates with identical input and output budgets.
The code task required correct interval merging, no mutation of the input, and a ValueError for an invalid interval. The generated code was executed in an isolated environment.
| Acceptance check | 8B | 32B | DeepSeek |
|---|---|---|---|
| Short translation, city extraction, and 2+2 | All correct | All correct | All correct |
| Merge results, empty input, and touching endpoints | Passed | Passed | Passed |
| Does not modify the original input | Failed | Passed | Passed |
Raises ValueError for an invalid interval |
Failed | Passed | Passed |
The 8B answer modified the original input and omitted the invalid-interval check. The 32B and DeepSeek answers met both constraints. This provides specific evidence for routing constrained algorithm implementations to 32B, but it does not establish a general ranking of model capability.
The asynchronous-cache code generated by DeepSeek passed its three included tests, but the logs still contained an unhandled Future exception. Outputs from a stronger model still require independent validation.
What Happens When the Jev Decision Service Fails?
Model selection depends on Jev returning valid decision evidence. If Jev is unreachable, times out, or returns an invalid result, the decision plugin does not publish a ranking opinion. The gateway continues with the remaining selection and forwarding process. This is fail-open behavior.
In an isolated failure-test route, we configured an unreachable Jev endpoint while keeping all three generation candidates healthy. The log showed publishing no rank. DeepSeek happened to answer, and the final response was HTTP 200. This demonstrates that processing can continue after a decision failure. It does not mean that failures always route to DeepSeek.
Another input of approximately 79 KB did not succeed. Jev returned 400 and published no ranking opinion. The request then entered the 32B target, which also returned 400 because its context budget was insufficient. When no decision ranking is available, the gateway cannot guarantee selection of a model that meets the task requirements.
A separate translation prompt that instructed the system to ignore the policy and choose DeepSeek still went to 8B, but its answer did not satisfy the request. This again shows that model selection and answer quality must be evaluated separately.
06 Return to the Gateway and Follow the Complete Call Path
After the configuration and validation steps, the division of responsibilities between GPUStack and the gateway becomes easier to understand.
GPUStack manages candidate models, providers, and routing policies, then synchronizes the configuration to Higress. When a request reaches the gateway, plugins prepare the candidate information, the decision plugin calls Jev for selection evidence, and the gateway chooses an upstream target before forwarding the request to a private model or DeepSeek.
The Jev call and the generation call are two different paths. Jev participates in model selection. Qwen3 or DeepSeek generates the answer. If the decision call does not produce valid evidence, the request may continue, but selection of the expected model is no longer guaranteed.
A Deeper Look at How the Plugin Uses Jev Results
To understand how the gateway processes Jev’s evidence internally, read the next figure from top to bottom. The plugin prepares the candidates and task descriptions, calls Jev, converts the returned model probabilities into candidate scores, and lets the load-balancing finisher aggregate the scores and select an upstream target.
The diagram is based on commit 790256e of the decision-plugin source code. The probabilities and confidence value in the diagram illustrate the calculation and are not measured responses from the preceding tests. Confidence adjusts the influence of the decision opinion. It is not the probability that an answer is correct.
For the application, the entry point always remains demo-smart. For the administrator, the work is to maintain the capability boundaries of the candidate models and continuously validate selections against representative tasks. This is the value of decision routing: put the “which model should handle this request” decision into the gateway so that multi-model specialization becomes a configurable and testable policy.
This validation covered 36 requests: 9 distinct inputs through the same demo-smart route, 12 same-task baselines, 6 additional repetitions, 3 streaming requests, and 6 failure or policy-control cases. Routing and transport met their respective criteria, while answer quality was evaluated separately. The article focuses on the capability introduced in v2.3.0rc2 and uses a development environment with Server baseline f0c237f and UI baseline 5a4fc16. It does not include sustained-concurrency testing or a cost-benefit analysis.
Top comments (0)