ExLlamaSharp v1.2.1-beta is out.
Local LLM server for Windows with NVIDIA GPUs — OpenAI-compatible /v1, Blazor admin, and EXL3 inference.
Release notes
Pre-release: zero-gaps on top of continuous batching. This is not the GitHub Latest download; stable remains 1.1.1. ## Install Download ExLlamaSharp-Setup-win-x64.exe from this pre-release (not the one-liner, which still installs 1.1.1). Run it (UAC), then open http://127.0.0.1:14563 - Admin UI: admin / changeme (change this in Setup) - API key: sk-exllamasharp-dev ## Changes - A/B, tenants, and LoRA adapter HTTP APIs are real (CRUD + OpenAI X-Ab-Test-Id / X-Adapter-Id) - EXL3 worker: LoRA load/unload, tools/JSON-schema prompt hints, multi-GPU CUDA_VISIBLE_DEVICES, speculative draft settings - Embeddings via ONNX when model.onnx is present (local fallback otherwise) - Quantize/import jobs call real Python convert / folder registration - SignalR metrics + Charts / A/B UI; tenant filtering when multi-tenancy is enabled - Worker-only production generate; OpenAI images/audio stay 501 (Media version later) ## Notes Treat this as a beta. Keep a 1.1.1 install if you need the last stable build.
Full notes: https://github.com/Kortexio/ExLlamaSharp/releases/tag/v1.2.1-beta
Top comments (0)