DEV Community: aivrar

Running vLLM natively on Windows

aivrar — Fri, 09 Oct 2026 06:48:29 +0000

vLLM officially supports Linux only. On Windows the usual answer is WSL or Docker. I wanted it running natively, with real CUDA kernels and an OpenAI-compatible server, and no Linux layer. So I maintain vllm-windows-build: patches, prebuilt wheels, and portable installer scripts for running vLLM directly on Windows 10/11.

Disclosure: I built this repo, and I built it with heavy AI-assisted coding (the README credits Claude). The engine is upstream vLLM. My part is the Windows port, the packaging, and the testing. Everything below comes from the repo's README, docs, and release notes.

What's actually published

There are three current options. The repo's release chooser puts it like this:

Channel	vLLM	Python	Torch / CUDA
Stable / Latest (`v0.27.1-win-cu130`)	0.27.1	3.13.14	2.13.0+cu130 / 13.0
Regular release, not Latest (`v0.29.0-win-cu132-py314`)	0.29.0	3.14.2	2.13.0+cu132 / 13.2
Prerelease (`v0.29.0-win-cu130-rc1`)	0.29.0	3.13.14	2.13.0+cu130 / 13.0

All three use Triton for Windows 3.7.1. The 0.27.1 wheel has kernels for SM 7.5, 8.6, 8.9, and 12.0, so RTX 20/30/40/50 series. It also includes FlashAttention 2, the Rust frontend and tool parser, and opt-in CPU/filesystem prompt-KV offload.

Why is 0.27.1 still the default? The Python 3.14 build uses the CPU TorchAudio wheel, because no matching cu132 wheel was available. GPU audio processing and audio-model serving weren't validated there. Model inference itself still runs on CUDA.

If you just want something that works, use 0.27.1.

Requirements

From the README:

Windows 10 21H2 x64 minimum (22H2 or 11 recommended)
NVIDIA GPU, SM 7.5 or newer, 12 GB VRAM minimum (24 GB recommended)
16 GB RAM minimum (32 GB+ recommended)
NVIDIA driver R580+ for CUDA 13.0
Single GPU per process. NCCL doesn't ship with PyTorch on Windows, so the patch uses a single-rank FakeProcessGroup.

You don't need Visual Studio or the CUDA Toolkit unless you're building from source.

Install option 1: the portable installer (what I recommend)

This route downloads its own Python, so nothing needs to be preinstalled.

Open the release you picked and download Source code (zip). For 0.27.1 that's v0.27.1-win-cu130.
Extract it into a new, separate directory. Don't mix 0.27.1, Python 3.13, and Python 3.14 installs in one folder.
Run:

install.bat

It downloads Python, PyTorch, and the matching prebuilt vLLM and Multi-TurboQuant wheels. Downloads are pinned by exact size and SHA-256. Nothing gets compiled. If you already have a verified 0.27.1 wheel, put it in dist-v0.27.1\ next to install.bat and the script will use it. Running install.bat again repairs an incomplete install.

Start the server:

launch.bat

With no arguments it shows a model picker that scans models\ next to the script. Or pass a model directly:

launch.bat --model E:\models\Qwen3-14B-AWQ-4bit --port 8000

launch.bat checks the install first and reruns install.bat if something is missing. It also sets CUDA_DEVICE_ORDER=PCI_BUS_ID, so --gpu-id matches nvidia-smi ordering.

If you prefer git, a default clone of master gives you the 0.27.1 installer. For the Python 3.14 build:

git clone --branch v0.29.0-win-cu132-py314 https://github.com/aivrar/vllm-windows-build.git vllm-py314-cu132

Install option 2: the wheel in your own venv

For 0.27.1, the README's manual steps are:

py -3.13 -m venv venv
venv\Scripts\activate

pip install torch==2.13.0 torchaudio==2.11.0 torchvision==0.28.0 ^
    --index-url https://download.pytorch.org/whl/cu130
pip install triton-windows==3.7.1.post27
pip install vllm-0.27.1-cp313-cp313-win_amd64.whl
pip install multi_turboquant-0.1.0-py3-none-any.whl

Both wheels are on the release page. Their published SHA-256 hashes are:

vllm-0.27.1-cp313-cp313-win_amd64.whl: 7c13ed44e94694478bdd4f5fcca23e2d66ba1e8fa9bccad9fddb8651d1b2447b
multi_turboquant-0.1.0-py3-none-any.whl: 5b310e05904b588539d9a8e3374dfa6c160f025f9c2099ba5c7877c79b2fa149

If your environment was built from older wheel metadata, there's an optional repair step: pip install "llguidance>=1.7.0,<1.8.0" "xgrammar>=0.2.0,<1.0.0". Windows reports its machine type as AMD64, and older metadata skipped these packages because of that. The 0.27.1 wheel already has the correct markers.

Using it

OpenAI-compatible HTTP. The launcher (vllm_launcher.py) serves /v1/models, /v1/chat/completions (streaming and non-streaming), /v1/completions, /health, and /shutdown. Its default port is 8100 and it binds to 127.0.0.1. A quick test:

import requests
r = requests.post("http://127.0.0.1:8000/v1/chat/completions", json={
    "model": "qwen3-14b",
    "messages": [{"role": "user", "content": "Explain CUDA streams in 3 sentences."}],
    "max_tokens": 200,
})
print(r.json()["choices"][0]["message"]["content"])

It also parses tool calls from <tool_call> tags (Qwen3 format) and from bare JSON, and returns them in tool_calls.

Upstream vllm serve works too. The docs still call the launcher the more reliable path on Windows.

Python embedding. The usage doc sets VLLM_HOST_IP=127.0.0.1, adds the CUDA bin and torch\lib directories with os.add_dll_directory, and then uses LLM(...) / generate() as usual.

Models. Use a Hugging Face-format directory or repo ID with config.json and Safetensors, AWQ, or GPTQ weights. The launcher rejects direct .gguf files.

Memory tuning. If the weights load but you get "No available memory for the cache blocks", the docs say to raise --gpu-memory-utilization if VRAM is free (launcher default 0.6), or lower --max-model-len, --max-num-seqs, and --max-num-batched-tokens.

RTX 20xx (Turing). Add --turing-compat. It selects TRITON_ATTN, float16 KV cache, 32-token blocks, eager mode, 0.89 GPU utilization, one sequence, and 2,048 batched tokens. Any value you pass explicitly overrides the profile.

KV cache options

--kv-cache-dtype accepts auto, the fp8 variants, four upstream TurboQuant variants (turboquant_k8v4, turboquant_4bit_nc, turboquant_k3v4_nc, turboquant_3bit_nc), and six methods from my Multi-TurboQuant library (isoquant3/4, planarquant3/4, turboquant25/35).

Be honest with yourself about the trade-off. The six Multi-TurboQuant methods doubled KV cache capacity in the README's RTX 3090 test (16,336 → 32,672 tokens). But their encode/decode still runs in PyTorch, and throughput drops roughly 30–300×. The upstream turboquant_* variants use fused Triton kernels and don't pay that cost. For normal interactive use, stick with auto.

There's also experimental prompt-KV offload, which is off by default: --kv-offload cpu-lru|cpu-arc|fs-lru|fs-arc. The filesystem modes need --kv-offload-fs-root, and that directory has no automatic quota or cleanup, so keep an eye on its size. Offload only helps when long prompt prefixes repeat. It doesn't speed up uncached generation.

Limitations

Single GPU per process. For multiple GPUs, run separate instances on different ports and load-balance them yourself.
No FlashInfer, FlashAttention 3/4, fastsafetensors, DeepGEMM, or NIXL. None of them have Windows support here. FlashAttention 2 works.
Blackwell FP8/NVFP4: native Blackwell FP4 acceleration isn't included. Some mixed FP8/NVFP4 models fail at startup on unmodified 0.27.1. The troubleshooting doc has a Marlin-based workaround (confirmed for one model), and the 0.29.0 builds include the Marlin fallback fixes.
Cold start: the first inference that uses Triton kernels can take about 1–2 minutes to compile.
Limited testing. Validation is targeted. Most of it ran on one RTX 3090, with an earlier release also run on an RTX 3060, plus single-model confirmations from issue reporters on an RTX 2080 Ti, an RTX 5090, and an RTX PRO 5000 Blackwell. The README's timings are focused correctness checks, not broad benchmarks, and the 0.29.0 Python 3.14 build doesn't claim any speed advantage.
Building from source is possible (VS 2022 + CUDA Toolkit + a pagefile, no sccache), but it's heavy. The build records for each release are in docs/.

Links

Repo: https://github.com/aivrar/vllm-windows-build

More of my tools: https://github.com/aivrar

Running a local AI agent on Windows with LM Studio, no Docker or admin rights

aivrar — Fri, 09 Oct 2026 06:47:45 +0000

I wanted an AI agent on Windows that could use tools (files, a shell, web search, a browser) and talk to a model running on my own GPU. Most agent frameworks expect Linux, Docker, or a system Python you're allowed to install into. A lot of Windows machines don't have any of those, and on plenty of them you can't get admin rights either.

So I built Portable Hermes Agent. It's a portable Windows build of NousResearch/hermes-agent (MIT) with a desktop GUI, an LM Studio panel, a permissions panel, and some extra tools added on top.

Disclosure: I'm the author. I built most of the custom tools, GUI, and integrations with heavy AI-assisted coding (Claude Code, as the README credits). The core agent comes from Nous Research. This post sticks to what the repo and its release notes document.

What you get

From the README, the portable build adds:

A dark-themed Tkinter GUI with chat, a sidebar, and session management. The interface can switch at runtime between English, Traditional Chinese, and Simplified Chinese.
100 tools in 20+ toolsets. That includes 10 LM Studio tools (load/unload models, search Hugging Face, tokenize, embed, direct chat, API key management), workflows, a tool maker, an NVIDIA GPU status tool, a model switcher, and Serper search.
Everything hermes-agent already ships: web search, file operations, browser automation, code execution, delegation, memory, skills, messaging, Home Assistant, and more.
A permissions panel for file, network, and system access.
Optional extension modules for TTS, music, and image generation (ComfyUI).
A "guided mode" that works with no model connected and answers from a built-in user guide.

The LLM provider can be OpenRouter (cloud), LM Studio (local), or any OpenAI-compatible endpoint.

Requirements

Windows 10 or 11
An internet connection for cloud models, or an NVIDIA GPU with 8 GB+ VRAM for local models
No admin rights, no system Python, no Docker

Step 1: Download the right zip

Go to the Releases page and download the file named portable-hermes-agent-v*.zip. At the time of writing that's v1.4.10. Release notes include it, along with a PDF manual.

The release notes say this more than once: use the named portable zip, not GitHub's automatic "Source code" archives. Those aren't the verified release package.

Extract it to a normal user folder, for example:

C:\Users\YourName\Portable-Hermes-Agent

Don't put it in protected locations like C:\Program Files.

Step 2: First launch

Double-click:

START.bat

On first launch, START.bat runs the portable setup. It downloads embedded Python, the dependencies, the LM Studio SDK, and Node.js tools into that folder only. If you'd rather run setup yourself:

install.bat

or from PowerShell:

.\scripts\install.ps1

After that, these are the launchers:

START.bat           :: easiest GUI launch
hermes_gui.bat      :: GUI mode
hermes.bat          :: CLI mode
UPDATE.bat          :: safest one-click update

Step 3: Set up LM Studio as the local backend

This part runs on your own GPU.

Install LM Studio.
Download a model in LM Studio's Discover tab. The manual uses "Qwen 2.5 7B" at Q4 or Q6 as its starter example.
In LM Studio's Developer tab, click Start Server. The manual gives the default port as 1234.
In Hermes, open the LM Studio panel. The README calls it Tools > LM Studio. The manual also lists it as View > LM Studio (Local Models).

The manual describes the panel like this:

an endpoint field (default http://localhost:1234) with a Connect button
a status indicator (green dot when connected)
a model browser, a GPU selector, and a context length setting
Load Model / Cancel Load, Unload, and Use for Chat

Pick your model and GPU, set the context length, click Load Model, then click Use for Chat. Your messages now go to the local model.

Context length matters here. The agent sends a lot of tool definitions, so small contexts don't work. As of v1.4.8, the panel's default context matches the core's minimum of 64K, and the panel warns you before loading a model whose maximum context is too small. (The bundled manual still says 32,768 in places. Trust the newer release note.) More context uses more VRAM, so if a model doesn't fit, try a smaller quantization or a smaller model.

If your LM Studio server needs an API key, you can enter it in the same panel. The panel saves the endpoint and key to .lmstudio_config in the active HERMES_HOME directory, outside the application source. Both update paths keep that file, and release archives leave it out. To remove a saved key, clear the field and click Save. Keep that file private.

On multi-GPU machines, the manual says that choosing a GPU in the panel disables the other GPUs for that load, which forces the model onto one card.

No GPU? Use File > API Key Setup > OpenRouter, paste a key, and start chatting. You can switch between cloud and local later, from the sidebar or by asking the agent (for example "Switch to my local model").

Step 4: Set permissions before you give it real work

An agent that can run commands and edit files needs limits, so open Tools > Permissions before doing anything real. The manual lists categories for file reading, file writing, file deletion, package installation, command execution, and network access. Each one goes from Level 0 (disabled) to Level 4 (system/admin).

Defaults per the manual:

File reading: App + Home (Hermes folder plus your user folder)
File writing / deletion: App Only (inside the Hermes folder)
Package installation: App Only (into the bundled portable Python)
Command execution: App + Safe (runs commands, no admin/system changes)
Network: Web + APIs

If you want to keep everything on your machine, Network Level 1 ("Local Only") limits it to localhost services like LM Studio and the extensions. I'd start with the defaults and raise one category at a time when a task actually needs it.

Step 5 (optional): Extensions

Three extensions are separate portable servers I also maintain:

Extension	Port	GPU
TTS Server	8200	4 GB+
Music Server	9150	4 GB+
ComfyUI	5000	6 GB+

Each one installs itself on first use into the extensions/ folder. You manage them from Tools > Extensions (install, start/stop, status). The manual notes that the downloads are several GB. Remember they share the GPU with your LM Studio model, so you may need to unload one to run the other.

Workflows and the tool maker

Two features beyond plain chat:

Workflow engine: chain tool calls into pipelines with data flow, conditions, loops, parallel execution, error handling, and cron scheduling.
Tool maker: create new tools at runtime, either by wrapping a REST API or by writing a Python handler. They persist across sessions and reload automatically.

Updating

There are two separate update channels, and updating one doesn't update the other:

The portable distribution (launchers, GUI, integrations, portable tools): close Hermes and run UPDATE.bat, or hermes.bat update --backup --yes. This preserves .hermes/, custom tools, extensions, and python_embedded/.
Upstream hermes-agent core: start Hermes and ask it, e.g. "Check whether upstream Hermes Agent has updates. Do not install anything yet." Restart after a successful upstream update.

The README suggests running the portable updater when a release or security notice comes out (or every week or two). Recent releases (v1.4.9, v1.4.10) were mostly dependency security updates.

Limitations

Windows only, and local models need an NVIDIA GPU (8 GB+ suggested).
Context is the main constraint locally. The tool definitions are large, so you need a model and GPU that can handle about 64K context. That rules out a lot of small-VRAM setups.
Authenticated LM Studio model loading through the SDK needs an LM Studio Python SDK version that supports api_token. The v1.4.8 notes say the SDK installed at the time didn't, and that live auth enforcement wasn't tested. REST model discovery and chat still work without the SDK.
The GUI is Tkinter. It works, but it's plain.
Local model quality varies a lot. The manual itself notes quality differs a lot between models. Expect smaller local models to be less dependable at multi-step tool use.

Links

Repo: https://github.com/aivrar/portable-hermes-agent

More of my tools: https://github.com/aivrar