I wanted to use a decision model on internal project material without sending that material to a hosted inference service. That sounds simple until you get to the practical questions: which checkpoint, which inference code, which Docker image, and how much memory does it actually need?
If you prefer video format:
This is the setup I ended up using: Cloudflare Clef Flash on an NVIDIA DGX Spark, behind a local /v1/systemone endpoint.
I'll walk through that setup, then show what happened when I tested ten local checkpoints and a hosted Jev reference on the same frozen retrieval questions. The recipe and launcher are on GitHub.
The full methods, tables and limitations are in the published technical report, Comparing Open-weight Jev-like checkpoints and TypeSafe's Jev, DOI 10.5281/zenodo.23411130.
A quick bit of context: what goes into a decision API?
The useful shape is simple: supply a state containing evidence, a question, and explicit answer choices. The response selects an answer and can include probabilities. For example: given a retry policy and two operations, which operation is allowed to retry?
Here, “Jev-style” describes that interface. The local checkpoints come from other publishers; this tutorial does not run the hosted Jev model locally. Our hosted comparison used typesafe-ai/jev through Vercel on October 13, 2026, with no immutable model revision exposed in the logs.
For the local choice questions, trained decision heads score the supplied choices. They do not need to generate an answer as a sequence of chat-completion tokens. A response reporting zero output tokens can still contain a real decision.
1. Setting up the runtime
This recipe has been tested on an NVIDIA DGX Spark (Linux ARM64).
The key is choosing a container that matches your hardware and the checkpoint's inference code. For another machine, work through these four checks:
-
Match the CPU architecture. Check the candidate image manifest with
docker buildx imagetools inspect IMAGE:TAG. Look for your platform: typicallylinux/amd64on an x86 Linux workstation orlinux/arm64on Spark. Docker's platform documentation explains why an image's architecture matters. CPU emulation does not establish GPU compatibility. - Match the GPU and driver. Find a candidate in NVIDIA's PyTorch container release notes and framework support matrix. Check its GPU architecture, CUDA and driver requirements. NVIDIA's CUDA compatibility guide also documents feature and PTX limitations; a matching CUDA major version alone is insufficient evidence.
- Match the publisher's loader. Read the pinned checkpoint's model card, dependency files, and the runtime adapters. A compatible PyTorch container supplies the tensor runtime; it still needs the right Transformers version and the publisher's decision-head implementation.
- Pin the combination that you validate. Record the image digest, checkpoint commit, API commit, direct dependency versions and a full environment lock. Test GPU execution, loading, a real decision, overflow handling and peak memory on that machine.
On an x86 NVIDIA system, use an appropriate linux/amd64 base and validate the model there. The supplied launcher targets Linux ARM64; the Spark results do not establish support for other hardware. PyTorch's installation selector is a starting point for a separate platform port.
2. Get the recipe and inspect the actual inference stack
Run these commands on the Spark:
git clone https://github.com/krisitown/local-jev-api.git
cd local-jev-api
# The recipe revision inspected for this article.
git checkout b0787e9425d8abaadf0023bf377f4258d14e9564
The stack is worth understanding before launching it:
Docker packages this stack. The tested recipe does not use vLLM.
Uvicorn handles HTTP, and FastAPI defines the routes. Our wrapper loads the checkpoint once at startup, validates incoming requests, checks encoded context length and calls the publisher's Python inference implementation. That code uses Transformers for model components and PyTorch + CUDA for tensor execution. The model remains loaded between requests.
Clef Flash needs the complete Hugging Face snapshot: four backbone weight shards, the shard index, joint_head.safetensors, joint_head_config.json, tokenizer/processor assets and joint_schema_model.py. The backbone weights alone are not the complete decision model.
Other families have different layouts. PPLX includes its own source/src/autojev implementation; Jeff uses its matching runtime; NeoHorse has a backbone, pointer head and bundled runtime; Laya uses its agent loader and tokenizer layout. The API shape is shared, but those loading paths are not interchangeable.
3. Pull the runtime, then download the pinned checkpoint
The two prebuilt images for the tested Spark setup are:
Image tag under ghcr.io/krisitown/local-jev-api
|
Use |
|---|---|
spark-20261013 |
Clef Flash and eight other tested checkpoints |
spark-omni-20261013 |
Clef Omni's separate compatibility runtime |
Both were anonymously accessible when checked on October 16. Docker needs GPU access; NVIDIA's Container Toolkit guide covers that prerequisite.
Leave room for the downloads and the running model. The common runtime is roughly 20 GB unpacked, and Flash's weights are about 18 GiB. In this workload Flash used 17.75 GiB of loaded CUDA tensors but caused an approximate 36.06 GiB peak drop in host available memory. Those are different measurements. The launcher does not pause other services for you.
For the manual walkthrough, keep using the Spark terminal inside the cloned repository:
docker pull ghcr.io/krisitown/local-jev-api:spark-20261013
docker image inspect ghcr.io/krisitown/local-jev-api:spark-20261013 \
--format '{{.Os}}/{{.Architecture}}'
python3 ops/download_model.py --key clef-flash
The image check should report linux/arm64. The downloader uses the container's installed Hugging Face client, so it does not require installing that client into your host Python. It downloads the runtime snapshot into checkpoints/clef-flash, explicitly requesting:
Repository: Cloudflare/clef-flash
Revision: 8b2e5fd17c09fd49bc2880b805d5323ec7ff4ff3
Underneath, the important operation is snapshot_download(repo_id=..., revision=..., local_dir=...). Hugging Face's download guide documents repository snapshots and revision pinning. Use the full commit, not a moving main branch, when reproducing this setup.
Check what arrived:
ls -lh checkpoints/clef-flash
cat Dockerfile
The Dockerfile pins the NVIDIA PyTorch base by digest and keeps its CUDA-enabled PyTorch. Its direct packages are:
transformers==5.17.0 accelerate==1.14.0
huggingface-hub==1.32.0 safetensors==0.8.0
fastapi==0.136.3 uvicorn==0.52.4
pydantic==2.13.5 pillow==12.1.1
These are the recipe's tested direct pins, not a claim that they are the best versions for every checkpoint. Omni uses Transformers 5.10.2 and CPU-first loading in its separate image. Installing the newest Transformers or replacing NVIDIA's PyTorch with an arbitrary wheel changes the environment being reproduced.
If exact image identity matters, the common published image's registry digest is:
ghcr.io/krisitown/local-jev-api@sha256:b4a417bb7585405cd03688849328ee2b8d316f5f083fca0032ad24be2a04b1ea
You can use that digest reference instead of the tag in docker pull, docker run, the downloader's --image option, or the launcher's IMAGE override. The provenance file records both images. Registry digests and local Docker image IDs are different identifiers. Rebuilding the Dockerfile may resolve different transitive packages; a versioned tag plus direct pins is not a complete environment lock.
Before loading the weights, verify that this container can execute a small GPU operation:
docker run --rm --gpus all --entrypoint python3 \
ghcr.io/krisitown/local-jev-api:spark-20261013 \
-c 'import torch; print(torch.__version__, torch.version.cuda); assert torch.cuda.is_available(); print(torch.cuda.get_device_name(0)); print((torch.ones(4, device="cuda") + 1).cpu())'
That confirms basic CUDA execution in the selected image. Loading and exercising the actual model is the next check.
4. Start the local API
The checkpoint stays outside the image and is mounted read-only:
docker run -d --name local-jev-flash --gpus all --ipc=host \
-p 127.0.0.1:18081:18081 \
-v "$PWD/checkpoints/clef-flash:/model:ro" \
-e MODEL_PATH=/model \
-e MODEL_ID=Cloudflare/clef-flash \
-e MODEL_REVISION=8b2e5fd17c09fd49bc2880b805d5323ec7ff4ff3 \
-e BACKEND=clef \
ghcr.io/krisitown/local-jev-api:spark-20261013
docker logs -f local-jev-flash
Flash took about 157 seconds to load in the benchmark; downloads are additional time. Wait for startup to complete. Ctrl-C closes the log viewer and leaves the container running.
Uvicorn listens on 0.0.0.0 inside the container, while -p 127.0.0.1:18081:18081 publishes the port on the Spark's loopback interface. This experimental wrapper has no authentication layer. Keep this binding for the walkthrough; Docker documents the behavior in its port-publishing guide.
In the Spark terminal, check readiness and inspect the returned identity:
curl --fail --silent --show-error http://127.0.0.1:18081/health
Look for ready: true, model Cloudflare/clef-flash, the pinned revision and backend clef. The runtime uses the mounted files with Hugging Face offline settings. Initial downloads still contact external registries.
The shorter path for your next run
The manual steps above make each piece visible. On a fresh setup, the launcher can pull the image, download the pinned checkpoint, start the container, wait for readiness and send a typed smoke request:
./run.sh clef-flash
Do not run that on top of the manual container: the launcher deliberately refuses to replace an existing container with the same name. At the end of this tutorial, the cleanup commands remove that demo container. You can also supply an existing complete snapshot:
./run.sh clef-flash ./checkpoints/clef-flash
5. Test the API
With the model running, send a decision request from a terminal on the Spark:
curl --fail --silent --show-error \
http://127.0.0.1:18081/v1/systemone \
-H 'Content-Type: application/json' \
--data-binary @- <<'JSON'
{
"state": "Policy: automatic retry is allowed for idempotent operations and prohibited for non-idempotent operations. GET /status is idempotent; POST /charge is not.",
"questions": {
"q1": {
"type": "choice",
"instructions": "Which operation may the handler retry after a timeout?",
"criteria": {
"A": "GET /status",
"B": "POST /charge",
"C": "Both",
"D": "Neither"
}
}
},
"model": "Cloudflare/clef-flash"
}
JSON
Read answers.q1.choice; A is the expected policy answer. Inspect the actual response rather than assuming a successful HTTP request means a correct answer. You can send the same JSON from Insomnia with POST, JSON content type and no bearer key.
The wrapper also provides /inspect and /metrics, and adds _benchmark telemetry. It rejects oversized inputs with HTTP 413 instead of silently dropping the supplied state. Flash's 16,384-token limit includes encoded questions and options, not just the state string.
6. Try another checkpoint deliberately
The launcher supports all ten tested checkpoints:
./run.sh --list
| Key | Checkpoint family |
|---|---|
clef-flash, clef, clef-omni
|
Cloudflare Clef |
pplx-decider-v1.1-27b |
Perplexity PPLX |
neohorse-jev-4b |
TokenRhythm NeoHorse |
jeff-0.8b, jeff-2b
|
Jeff |
laya, laya-multilingual, laya-typed-decisions
|
Convai Innovations Laya |
Each key selects a pinned snapshot and a family-specific backend. Omni automatically selects its separate image. Jeff uses the manifest's download alias at the same resolved commit. All revisions are in the checkpoint manifest.
After stopping the Flash demo, for example, ./run.sh jeff-2b starts Jeff 2B on the default port. If you intentionally have enough headroom to run another model alongside it, PORT=18082 ./run.sh jeff-2b changes the host port. Use the corresponding checkpoint ID in your JSON request; the launcher handles that for its own smoke request.
What the tests actually said
The technical report on Zenodo covers all eleven systems, including hosted Jev, and explains the measurement rules behind the charts below.
I ran 336 frozen cases per local checkpoint on one GB10 DGX Spark: 288 one-hop lookups and 48 two-hop lookups, including missing evidence and near-matching distractors. The ten selected complete runs produced 3,360 outcomes: 2,065 answers and 1,295 context rejections. Rejected inputs are not completed inferences or wrong answers.
The fairest small-input comparison is the 112 cases every model accepted: 96 one-hop cases with 256–1,024 state characters and 16 two-hop cases at 1,024 characters.
Identical short inputs across all eleven systems. Hosted Jev is the deployment observed on October 13, not an evaluation of a later named release.
| Local checkpoint | Correct / 112 | Loaded CUDA GiB | Short median ms |
|---|---|---|---|
| Laya | 46 | 1.57 | 22.5 |
| Laya Typed | 52 | 1.57 | 22.5 |
| Laya Multilingual | 25 | 1.22 | 11.1 |
| Jeff 0.8B | 77 | 1.62 | 40.1 |
| Jeff 2B | 94 | 4.12 | 54.1 |
| NeoHorse 4B | 104 | 8.55 | 120.6 |
| Clef Flash | 102 | 17.75 | 237.2 |
| PPLX 27B | 112 | 48.60 | 606.5 |
| Clef | 104 | 51.19 | 674.4 |
| Clef Omni | 102 | 59.30 | 345.9 |
The latency column comes from a separate repeated 512-character microbenchmark, not the average time for the 112 quality cases. A two-answer difference is too little to establish a general model ranking; the technical report includes descriptive uncertainty intervals and the full local/hosted comparison. The repository also contains the local findings and machine-readable local measurements.
Small checkpoint does not equal total memory requirement
Loaded allocation describes live model tensors. It is not the amount of RAM you can safely provision.
For Flash, the progression was 17.75 GiB loaded → 21.68 GiB peak allocated during inference, with 29.33 GiB peak reserved and roughly 36.06 GiB peak host available-memory drop. Reserved memory includes reusable allocator blocks. Host memory changes include runtime overhead and other activity.
Spark shares CPU and GPU memory, so do not add those CUDA and host numbers together. Clef and Omni each showed over 100 GiB of host available-memory drop in this workload. Startup and long inputs need headroom too.
Context capacity and useful recall are separate results
Here are the configured token limits in the tested local serving recipes. The scope matters: a state-only budget and a full encoded question budget are different constraints.
| Checkpoint | Configured cap, tokens | What the cap covers |
|---|---|---|
| Laya | 512 | Full encoded input per question |
| Laya Typed | 1,024 | Full encoded input per question |
| Laya Multilingual | 1,024 | Full encoded input per question |
| Jeff 0.8B | 8,192 | Full encoded input per question |
| Jeff 2B | 8,192 | Full encoded input per question |
| NeoHorse 4B | 2,048 state; 8,192 branch | Separate state and branch budgets |
| Clef Flash | 16,384 | Total encoded input |
| PPLX 27B | 8,192 | Full encoded input per question |
| Clef | 16,384 | Total encoded input |
| Clef Omni | 64,000 | Total encoded input |
These are serving limits, not theoretical backbone windows. Full encoded input includes the question and choices. The wrapper returned HTTP 413 for overflow; it did not silently truncate scored requests. NeoHorse's successful 2,088-token full probe still respected its 2,048-token state-only budget. The boundary search reached 1,023 tokens for Laya Multilingual and 63,999 for Omni; their configured caps were 1,024 and 64,000 respectively.
Omni processed 63,999 encoded tokens in a boundary probe. On the longest scored one-hop tier, 131,072 state characters, it got 13/32 correct. The probe used repetitive filler; the scored corpus had a different token density. Characters and tokens are not interchangeable units.
Each cell has 32 planned cases. Values are correct/completed; a dash means every case in that cell was rejected. These are four selected systems, not the full ten-model matrix.
Flash's 24/32 at the 32,768-character tier is particularly useful to unpack: 24/24 present needles were correct, while 0/8 missing-evidence controls were correct. Finding supplied evidence did not imply recognizing its absence.
PPLX was correct on all 224/224 accepted cases. Hosted Jev matched it on those exact 224 jointly accepted cases and handled additional longer cases. Across the hosted run, Jev scored 301/304 accepted, with 32 context rejections out of 336 submitted. All three wrong answers involved two-hop lookups at 65,536 characters. That is a strong result on this corpus, not proof of perfect decision-making.
Speed depends on the request shape
These are decision heads, so I report decisions per second. A four-question request helped some runtimes and barely changed others:
This test repeats the same question four times under different IDs. Hosted measurements include network and gateway overhead; local measurements are from Spark localhost.
Flash went from 4.21 decisions/s for one question per request to 9.84 for four questions. Four simultaneous HTTP clients did not deliver the same improvement: our wrapper serializes GPU inference, so concurrency mostly exercised queuing. It is not a dynamic GPU batching engine.
The hosted short sequential median was about 384.5 ms. That includes the client → Vercel → provider path; local latency excludes that network path. Neither comparison isolates model compute. Local timing omitted a separate cached length-inspection preflight but included native inference encoding.
How I'd use these results
For a small local starting point, Jeff 2B is worth testing: 94/112 short cases, 4.12 GiB loaded and about 54 ms on the short microbenchmark, with weaker two-hop results. NeoHorse 4B had strong short results at 8.55 GiB, but only a 2,048-token state budget. PPLX 27B gave the strongest observed local retrieval result, at a much larger memory cost and an 8,192-token serving cap.
Clef Flash is the practical walkthrough choice here: a verified integration path, a 16,384-token budget and a smaller footprint than Clef or Omni. Its missing-evidence failures still matter. Laya variants were the lightest and fastest in these runs, but less accurate on this English corpus; this test says nothing about their multilingual strengths.
The limits of the experiment are real: one Spark, one synthetic English corpus, family-specific loaders and precision, no randomized model order or thermal control, and no independent repeated sessions. Hosted GPU hardware and immutable model revision are unavailable. An interrupted Omni attempt is preserved separately and excluded from the main local comparison. This is not a coding, multilingual or multimodal leaderboard.
Reproducibility note: the runtime and manual inference path were exercised in the experiment and recording. Launcher orchestration was checked with simulated Docker/HTTP responses; the whole fresh-install launcher was not rerun against a live GPU during publication preparation.
Put the endpoint into a workflow
The reason I wanted this interface in the first place was Jev Quality Gate: take bounded project evidence, ask explicit policy questions and retain the responses so the decision can be inspected and replayed.
The local adapter branch supports a configurable endpoint and model, with optional credentials. For this local setup, no bearer key is needed. The pinned Flask PR integration record shows six typed responses retained and replayed against a real public pull request.
That integration proves transport and replay. Its final policy result was uncertain, because global reconciliation was not performed. It was not a passed whole-change compliance check or a code-review accuracy evaluation.
When you finish the demo, run this on the Spark:
docker stop local-jev-flash
docker rm local-jev-flash
Restore any workload you paused.
The repository has the launcher, pinned checkpoint manifest, Dockerfiles and local benchmark evidence. Try a checkpoint that fits your machine, then test it on the decisions your project actually needs—including the cases where the right evidence is missing. If you try a different GPU or runtime combination, include those versions with your results; that is what makes the next person's setup easier.
Report citation: Stoyanov, K. (2026). Comparing Open-weight Jev-like checkpoints and TypeSafe's Jev. Zenodo. https://doi.org/10.5281/zenodo.23411130. The record currently contains the report PDF; the local corpus and outcome package is separate.
Top comments (0)