The 3 AM Page: GPU Out of Memory
Your voice cloning demo was flawless yesterday. This morning, transcription stalls mid-sentence, synthesis requests queue for thirty seconds, and nvidia-smi shows 23.8 GiB used on a 24 GiB card. You added a local LLM for transcript cleanup overnight. Now neither the audio pipeline nor the text model are usable.
The failure signature is predictable: KV cache and synthesis buffers compete for the same memory pool, queueing latency climbs before the allocation fails, and you discover that quantization only compressed the weights—context still needs full-precision cache.
Put a Boundary Around the Text Model
VoiceStudio is an open-source, fully local voice workstation covering transcription, cloning, dubbing, and synthesis across 646 languages. My concern as an external integrator is narrower: how do we add local text generation without starving the audio pipeline?
The architecture I propose keeps VoiceStudio responsible for its supported audio workflows. An application-side adapter sends optional transcript cleanup or narration rewriting to a separate, OpenAI-compatible vLLM process. Accepted text returns to the audio workflow.
That adapter must enforce bounded input, deadlines, and admission control. Audio jobs and text requests need separate queues; otherwise, a long rewrite blocks an interactive voice operation before either reaches the GPU.
For initial deployment, I allow two simultaneous text requests and reject excess work upstream rather than accumulate an unbounded queue. Whether VoiceStudio's selected audio backends remain resident is backend-dependent. Measure that behavior before assigning the remaining memory to vLLM.
The Failure Starts Before the Crash
A misleading tutorial pattern is to load a quantized model, accept its advertised context length, and increase concurrency until throughput looks impressive.
Quantization primarily reduces weight storage. It does not make KV-cache growth disappear.
A 70B deployment makes this especially painful. Partial GPU offloading may get weights loaded, but additional context still needs cache storage whose placement depends on the runtime. A successful load therefore proves very little about generation-time headroom.
The production failure signature is usually rising queue latency followed by allocation failure when synthesis and long-context generation overlap.
Likewise, memory that stays high after requests finish is not automatically a KV-cache leak. Allocator retention and deliberately preallocated cache are normal. Suspect a leak when repeated, identical workloads produce an increasing memory floor or declining throughput after accounting for queue depth and context length.
A Bounded Starting Configuration
This example assumes an NVIDIA GPU with 24 GiB, an installed CUDA-compatible vLLM environment, and no other text server. It reserves roughly 45% of device memory for the vLLM instance, including weights, activations, and KV cache—not 45% for cache alone.
The remaining capacity is headroom for audio, not a guarantee that every voice backend fits. Run this on a trusted workstation; the API binds only to loopback.
The single block launches the server and writes an aiohttp benchmark. Wait for vLLM readiness before running the final command.
python -m pip install aiohttp
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
--host 127.0.0.1 --port 8000 \
--quantization awq --dtype half \
--gpu-memory-utilization 0.45 \
--kv-cache-dtype auto \
--max-model-len 4096 \
--max-num-seqs 2 \
--max-num-batched-tokens 2048 \
--enable-chunked-prefill \
>vllm.log 2>&1 &
echo $! >vllm.pid
cat >bench.py <<'PY'
import asyncio
import json
import os
import time
import aiohttp
BASE = os.getenv("BASE_URL", "http://127.0.0.1:8000/v1")
MODEL = os.getenv("MODEL", "Qwen/Qwen2.5-7B-Instruct-AWQ")
HEADERS = {"Authorization": f"Bearer {os.getenv('API_KEY', 'local')}"}
async def gpu_mib():
process = await asyncio.create_subprocess_exec(
"nvidia-smi", "--id=0",
"--query-gpu=memory.used", "--format=csv,noheader,nounits",
stdout=asyncio.subprocess.PIPE,
)
output, _ = await process.communicate()
if process.returncode:
raise RuntimeError("GPU memory sampling failed")
return float(output.decode().strip())
async def request(session):
payload = {
"model": MODEL,
"messages": [{
"role": "user",
"content": "Rewrite this narration clearly: Local audio stays private."
}],
"max_tokens": 128,
"temperature": 0,
"stream": True,
}
started = time.perf_counter()
first = None
async with session.post(
f"{BASE}/chat/completions", json=payload
) as response:
response.raise_for_status()
async for raw in response.content:
line = raw.decode().strip()
if not line.startswith("data:"):
continue
data = line[5:].strip()
if data == "[DONE]":
break
event = json.loads(data)
for choice in event.get("choices", []):
if choice.get("delta", {}).get("content") and first is None:
first = time.perf_counter() - started
if first is None:
raise RuntimeError("No generated text received")
return {"ttft_s": first, "latency_s": time.perf_counter() - started}
async def main():
baseline = await gpu_mib()
samples = [baseline]
async def sample():
while True:
samples.append(await gpu_mib())
await asyncio.sleep(0.2)
timeout = aiohttp.ClientTimeout(total=180)
async with aiohttp.ClientSession(
headers=HEADERS, timeout=timeout
) as session:
monitor = asyncio.create_task(sample())
try:
for batch in range(3):
results = await asyncio.gather(
request(session), request(session)
)
print(json.dumps({"batch": batch, "requests": results}))
finally:
monitor.cancel()
await asyncio.gather(monitor, return_exceptions=True)
print(json.dumps({
"gpu_baseline_mib": baseline,
"gpu_peak_mib": max(samples),
"gpu_peak_delta_mib": max(samples) - baseline,
}))
asyncio.run(main())
PY
# After the server reports readiness:
python bench.py
Read the Measurements Honestly
TTFT here means time to the first nonempty text delta, including HTTP and queueing overhead. Latency measures the complete streamed response. Neither measures speech-generation latency.
GPU memory is device-wide, sampled every 200 milliseconds. Short spikes may escape sampling, and vLLM's preallocated cache can make the incremental delta deceptively small. Record the absolute baseline as well as the delta.
Run one warm-up pass, then repeat while a representative VoiceStudio audio job is active. The six requests above are a smoke test; diagnosing long-run degradation requires sustained repetition with fixed inputs.
For a remote comparison, the same harness can target B-Lost's relay endpoint, https://b-lost.com/v1, using an authorized key and an available model identifier. That changes the privacy boundary: send synthetic text only. Local GPU measurements then describe the workstation, not relay-server memory.
The trade-off is deliberate. A 4,096-token ceiling and two active sequences sacrifice long-document convenience and peak text throughput. In exchange, they create a measurable resource boundary around the text service. For an interactive voice workstation, predictable audio latency matters more than winning an isolated tokens-per-second benchmark.
The Hard Question
The bounded configuration above assumes you are willing to reject text requests when the queue is full. That decision forces a choice: should the adapter retry silently, return an error to the user, or fall back to a simpler heuristic that does not require GPU inference?
If you retry, you risk masking persistent overload until an interactive operation times out. If you return an error, users see intermittent failures during normal operation. If you fall back to rule-based cleanup, you lose the quality improvement that justified adding the LLM.
I picked explicit rejection because it makes the capacity boundary visible during development. Production deployments may need a different answer.
What failure mode have you chosen when local inference cannot keep up, and what operational signal told you that choice was wrong?
Top comments (0)