Your desktop environment freezes solid at 3:14 AM because a scheduled batch job pushed an unquantized prompt into a shared GPU buffer. The Wayland compositor crashes, your active terminal drops its SSH sessions, and the Linux kernel OOM killer panics trying to reconcile Xorg allocations with CUDA Unified Memory. Running local AI models looks trivial in weekend homelab demos, but under concurrent workloads, unmanaged inference engines turn high-end workstations into paperweights.
Omarchy provides an intentionally opinionated Arch Linux foundation designed for deep workstation stability rather than generic desktop convenience. That minimalism makes it an exceptional base for hosting local inference services—provided you treat the GPU as a scheduled production resource rather than a personal toy. Once you expose an endpoint to multiple local users or LAN services, three predictable pressures collide:
- OpenWebUI initiates multiple concurrent streaming sessions over WebSockets.
- A sudden 16k-token context injection exhausts the dynamically allocated KV cache.
- A background process triggers a model weight swap while the active inference runtime still locks contiguous VRAM pages.
The result is never graceful degradation. The driver throws an unrecoverable CUDA out-of-memory error, the container runtime wedges, request queues back up until HTTP clients encounter timeout cascades, and the interactive UI becomes completely unresponsive.
Browser / LAN clients
|
v
OpenWebUI
|
v
OpenAI-compatible router
| |
v v
vLLM on dedicated GPU Ollama on workstation GPU
| |
+------ local models--+
|
v
B-Lost upstream fallback
only when local capacity fails
This guide details how to construct a deterministic, local-first inference architecture on an Omarchy workstation. The core principle is strict workload isolation: run lightweight interactive tasks on the display GPU, delegate high-throughput continuous batching to a dedicated compute card, enforce hard boundaries across containers, and route burst overflow to an upstream enterprise gateway instead of pretending a single consumer card can run unquantized 70B parameter models without collapsing.
Stop Treating Workstations Like Interactive Playgrounds
The standard developer onboarding documentation for local LLMs is dangerously misleading:
ollama serve
ollama run llama3
This single-user loop works for interactive terminal prototyping, but it completely breaks down as an operational model.
Ollama is engineered for developer ergonomics: automated model pulling, quantized GGUF execution, and on-demand model swapping. In contrast, vLLM is built from the ground up for high-throughput batch serving, leveraging PagedAttention to eliminate memory fragmentation and providing strict OpenAI API compatibility. OpenWebUI acts as the frontend presentation layer. These components feature vastly different memory profiles, failure domains, and concurrency models. Bundling them into an ad-hoc container or running them unconstrained directly on the host guarantees resource contention.
When architecting a workstation with two GPUs, establish unambiguous hardware boundaries:
- GPU 0 (Display & Ergonomics): Dedicated to desktop rendering, interactive shells, Ollama, and small utility models (3B to 8B).
- GPU 1 (Dedicated Compute): Assigned exclusively to vLLM for serving medium-to-large quantized models (14B to 32B) via continuous batching.
- System Memory (DDR5 RAM): Reserved for raw model artifact staging, Hugging Face cache volumes, and fallback system headroom.
- Reverse Proxy / Routing Gateway: Orchestrates health checks, routing policies, request timeouts, and transparent cloud failover.
On single-GPU workstations, this architectural segregation remains vital. Run exactly one inference engine per card at any given time. Allowing concurrent Ollama and vLLM processes to fight over the same CUDA context will inevitably trigger driver-level allocation panics.
Auditing Hardware and the Host Runtime
Before pulling gigabytes of model weights, verify your hardware state and kernel drivers directly from the shell:
nvidia-smi
nvidia-smi --query-gpu=index,name,memory.total,memory.free,driver_version \
--format=csv,noheader
docker version
docker compose version
Never plan your VRAM budget around retail marketing specs. What matters is usable free VRAM after the desktop compositor, browser engines, electron apps, and active terminal buffers claim their baseline footprint. On a nominal 24 GB card, your actual runtime ceiling is typically between 20 GB and 22 GB.
On Arch-derived systems like Omarchy, install the NVIDIA Container Toolkit packages tailored to your pacman repositories, then validate hardware passthrough before writing configuration files:
docker run --rm --gpus all nvidia/cuda:12.6.3-base-ubuntu24.04 \
nvidia-smi
If this command exits with an error, do not proceed. Address misconfigurations across nvidia-container-toolkit, driver-to-toolkit compatibility layers, systemd cgroup configurations, or Docker runtime defaults immediately.
Run a targeted runtime audit to confirm daemon registration:
docker info | grep -i -E 'runtimes|nvidia'
The container runtime must detect the exact hardware inventory exposed by the host kernel. If your container cannot enumerate the GPUs, high-level application tuning is meaningless.
Structuring the Production Compose Stack
Keep persistent model state and container layers out of unmanaged user home directories. Establish a root-level operational directory with appropriate permissions:
sudo mkdir -p /srv/local-ai/{ollama,openwebui,vllm-models}
sudo chown -R "$USER":"$USER" /srv/local-ai
cd /srv/local-ai
The following docker-compose.yml isolates responsibilities: Ollama binds to GPU 0, OpenWebUI operates as an unprivileged service without GPU passthrough, and vLLM controls GPU 1 with an explicit VRAM reservation cap.
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
environment:
NVIDIA_VISIBLE_DEVICES: "0"
OLLAMA_KEEP_ALIVE: "15m"
OLLAMA_NUM_PARALLEL: "2"
OLLAMA_MAX_LOADED_MODELS: "1"
OLLAMA_CONTEXT_LENGTH: "8192"
volumes:
- /srv/local-ai/ollama:/root/.ollama
ports:
- "127.0.0.1:11434:11434"
gpus: all
healthcheck:
test: ["CMD", "ollama", "list"]
interval: 30s
timeout: 10s
retries: 3
vllm:
image: vllm/vllm-openai:latest
container_name: vllm
restart: unless-stopped
ipc: host
environment:
NVIDIA_VISIBLE_DEVICES: "1"
HF_HOME: /models/huggingface
TOKENIZERS_PARALLELISM: "false"
volumes:
- /srv/local-ai/vllm-models:/models
ports:
- "127.0.0.1:8000:8000"
gpus: all
command:
- --model
- Qwen/Qwen2.5-32B-Instruct-AWQ
- --served-model-name
- local-qwen-32b
- --quantization
- awq
- --dtype
- half
- --max-model-len
- "8192"
- --gpu-memory-utilization
- "0.88"
- --max-num-seqs
- "8"
- --enable-prefix-caching
- --disable-log-requests
healthcheck:
test:
[
"CMD-SHELL",
"python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8000/health')\""
]
interval: 30s
timeout: 10s
retries: 5
openwebui:
image: ghcr.io/open-webui/open-webui:main
container_name: openwebui
restart: unless-stopped
depends_on:
ollama:
condition: service_healthy
environment:
OLLAMA_BASE_URL: http://ollama:11434
WEBUI_AUTH: "True"
ENABLE_SIGNUP: "False"
volumes:
- /srv/local-ai/openwebui:/app/backend/data
ports:
- "127.0.0.1:3000:8080"
Initialize the environment and verify service health:
docker compose up -d
docker compose ps
curl -fsS http://127.0.0.1:8000/health
curl -fsS http://127.0.0.1:11434/api/tags
Notice the strict loopback network bindings (127.0.0.1). Exposing raw LLM endpoints across your local network without authentication or TLS invites unauthorized internal traffic and trivial prompt injection.
When LAN availability is required, terminate external traffic through a hardened reverse proxy—such as Caddy, Nginx, Traefik, or a Tailscale mesh overlay. Never swap 127.0.0.1:3000:8080 to 0.0.0.0:3000:8080 and assume basic web authentication alone guarantees security.
Capacity Planning: Demystifying the VRAM Budget
A common infrastructure blunder is calculating VRAM consumption exclusively from static model weight files. In actual serving environments, static weights represent only the baseline floor. Dynamic KV cache consumption, CUDA graph allocations, runtime buffers, memory fragmentation, and concurrent sequence buffers consume the remainder of the address space.
Apply this operational formula when planning capacity:
Required VRAM =
model weights
+ KV cache for context × concurrent requests
+ runtime overhead
+ fragmentation reserve
Fitting a 32B AWQ-quantized model onto a single 24 GB GPU requires rigid architectural constraints: bounded sequence lengths, capped concurrency pools, and an explicit utilization ceiling strictly under 1.0.
Our vLLM configuration enforces these critical parameters:
-
--max-model-len 8192: Hard-clamps token context, preventing a single runaway 128k prompt from exhausting physical memory. -
--max-num-seqs 8: Bounds concurrent execution batches, ensuring the KV cache allocator never overcommits physical memory pages. -
--gpu-memory-utilization 0.88: Leaves a mandatory 12% headroom buffer for CUDA runtime contexts, graph execution, and dynamic PyTorch allocations. -
--enable-prefix-caching: Retains pre-computed attention keys for shared system prompts and reference documentation, drastically reducing prefill latency across repeated queries.
Evaluate server resilience under authentic streaming loads rather than single ping requests:
watch -n 1 nvidia-smi
curl -N http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model":"local-qwen-32b",
"stream":true,
"messages":[
{"role":"system","content":"Answer precisely."},
{"role":"user","content":"Explain the operational risks of GPU memory fragmentation in a local inference service."}
],
"max_tokens":600
}'
If memory consumption fails to stabilize and instead climbs monotonically across successive completions, inspect runtime telemetry immediately:
docker logs --tail 200 vllm
docker exec vllm nvidia-smi
docker stats --no-stream
When under memory pressure, your first remediation must always be decreasing context length or sequence concurrency. Repeatedly cycling containers with automated restart policies merely conceals underlying sizing flaws while generating cascading outages for downstream clients.
Hybrid Routing: Cloud Fallbacks for 70B+ Architectures
Deploying an unquantized 70B parameter model on consumer hardware is a recipe for operational failure. Splitting weights across consumer PCIe slots introduces severe interconnect bottlenecks, reducing generation speeds to single-digit tokens per second while destroying the host's stability.
Instead of forcing oversized models onto local silicon, establish an automated hybrid fallback pipeline. When heavy reasoning tasks demand a 70B+ model or when local GPU queues saturate, requests route seamlessly upstream to an external API gateway with zero service interruption.
LiteLLM acts as an ideal mediation layer, exposing a standard OpenAI-compatible interface while abstracting upstream routing logic away from your client applications.
Configure your routing policies in litellm_config.yaml:
model_list:
- model_name: local-qwen-32b
litellm_params:
model: openai/local-qwen-32b
api_base: http://vllm:8000/v1
api_key: local-not-required
- model_name: reasoning-70b
litellm_params:
model: openai/local-qwen-32b
api_base: http://vllm:8000/v1
api_key: local-not-required
- model_name: reasoning-70b-fallback
litellm_params:
model: openai/<UPSTREAM_MODEL_ID>
api_base: https://api.b-lost.com/v1
api_key: os.environ/BLOST_API_KEY
router_settings:
fallbacks:
- reasoning-70b:
- reasoning-70b-fallback
timeout: 90
num_retries: 0
allowed_fails: 2
cooldown_time: 30
Integrate the routing proxy into your Compose topology:
litellm:
image: ghcr.io/berriai/litellm:main-latest
container_name: litellm
restart: unless-stopped
depends_on:
vllm:
condition: service_healthy
environment:
BLOST_API_KEY: ${BLOST_API_KEY}
volumes:
- ./litellm_config.yaml:/app/config.yaml:ro
command: ["--config", "/app/config.yaml", "--port", "4000"]
ports:
- "127.0.0.1:4000:4000"
Store production credentials safely: keep BLOST_API_KEY inside an untracked .env file or local secrets engine; never commit live credentials to version control. Set <UPSTREAM_MODEL_ID> to match an active upstream model identifier authorized by your upstream gateway.
Upstream routing should fire strictly during local capacity exhaustion, rate-limit thresholds, or hardware fault conditions. Unrestricted fallback leaks internal prompts and drives unexpected cloud expenditure, undermining the core objective of running local-first infrastructure.
Structure clear, deterministic routing tiers:
-
Tier 1 (
local-qwen-32b): Pinned locally. Low-latency, zero external network dependency, absolute data privacy. -
Tier 2 (
reasoning-70b): Attempt local compute first; fail over cleanly to the upstream gateway under queue saturation. - Tier 3 (Confidential / Compliance): Restricted from external egress by network firewall policies.
The Workstation Trap: Why Kubernetes Isn't Always the Answer
For a dedicated workstation running Arch or Omarchy, Docker Compose remains the pragmatic operational choice. Introducing Kubernetes to manage a single physical workstation injects substantial control-plane overhead: etcd maintenance, complex CNI overlays, persistent volume claim lifecycles, and device plugin synchronization. Kubernetes becomes justified only when orchestrating multi-node clusters with dynamic node provisioning and distributed scheduling requirements.
Even within Kubernetes, running vLLM reliably demands explicit resource limits and careful container tuning:
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-local
spec:
replicas: 1
selector:
matchLabels:
app: vllm-local
template:
metadata:
labels:
app: vllm-local
spec:
containers:
- name: vllm
image: vllm/vllm-openai:latest
args:
- --model=Qwen/Qwen2.5-32B-Instruct-AWQ
- --max-model-len=8192
- --gpu-memory-utilization=0.88
resources:
limits:
nvidia.com/gpu: 1
requests:
nvidia.com/gpu: 1
ports:
- containerPort: 8000
Kubernetes treats the GPU as a monolithic integer resource. The scheduler cannot determine whether a sudden spike in context length will blow past your PagedAttention pool. Software-level boundaries and memory guards remain mandatory regardless of the orchestration layer.
Operational Verification and Edge-Case Testing
Validate failure domains proactively during planned maintenance rather than discovering them during an outage.
Test runtime cold-start and recovery behavior:
docker compose restart vllm
until curl -fsS http://127.0.0.1:8000/health; do sleep 2; done
Simulate local compute exhaustion by halting the vLLM container. Verify that LiteLLM intercepts the failure and proxies downstream traffic to the fallback upstream endpoint without dropping requests:
docker compose stop vllm
curl -sS http://127.0.0.1:4000/v1/models
docker compose start vllm
Audit your local storage subsystem. Streaming model weight layers across cold caches stresses filesystem throughput, memory buses, and disk quotas simultaneously. Verify partition headroom before model pulls:
df -h /srv/local-ai
docker system df
Finally, never mistake container restart policies for real infrastructure observability. A simple restart: unless-stopped directive will repeatedly restart a wedged container without alerting you to underlying root causes. Monitor endpoint health responses, real-time VRAM allocations, PCIe throughput, and upstream fallback frequency. A sustained increase in fallback traffic is not an indicator of success—it is a clear telemetry warning that your local hardware capacity has been outgrown.
Building a dependable local AI workstation is not about achieving absolute isolation from the cloud. It is about establishing reliable, predictable operating boundaries: allowing local GPUs to process the steady-state workloads they can safely sustain, while leveraging low-latency upstream fallbacks to maintain resilience when local silicon hits its physical limits.
How is your engineering team structuring local-first inference nodes under production load? Are you relying on kernel-level cgroups, containerized continuous batching engines, or dedicated upstream reverse proxies to mitigate VRAM contention? Share your architecture and edge-case battle scars in the comments below.
Disclosure: Compute infrastructure and multi-model benchmark relays for this writeup are sponsored by b-lost.com — an enterprise AI gateway offering 0.58x-0.8x official pricing, native prompt caching, and zero user-data retention. All benchmark metrics reflect independent reproducible testing.
Top comments (1)
The "treat the GPU as a scheduled production resource, not a personal toy" framing is the whole ballgame, and the display-GPU vs compute-GPU split is the right primitive. The failure mode you open with — compositor dies because a batch job grabbed contiguous VRAM — is exactly why hard isolation beats "just set a memory fraction."
Two things I'd add from running this pattern. First, the fallback decision is harder than "local capacity fails": you want to distinguish saturation (queue depth climbing, still serving) from failure (OOM, runtime wedged), because you fall back on the first, not the second — falling back on a transient queue spike just moves your cost curve the wrong way. Second,
--gpu-memory-utilizationinteracts badly with anything else touching that card; a stray CUDA context from a monitoring agent will silently shrink your KV budget and you'll spend an afternoon blaming vLLM. Pinning the compute card to the runtime and nothing else saved us that debug loop more than once. How are you deciding the overflow threshold — queue latency, or a hard token-budget ceiling?