Keep OpenMontage's Local LLM From Competing With Your Video Renderer

You deploy a local LLM for workflow orchestration. You run inference in isolation, validate the server, ship it. Two weeks later, production rendering jobs start failing with cryptic CUDA OOM errors at 3 AM. The inference server is still running. GPU memory looks fine in nvidia-smi. The blame lands on "the AI integration," and suddenly your careful architecture is fighting for its reputation.

This is the predictable failure mode when inference and rendering share GPU resources without explicit boundaries. I'm evaluating OpenMontage as an external integrator. The repository advertises 12 production pipelines, 100+ tools, and over 700 skill and production-knowledge files. Local inference is an attractive integration point: planning, tool selection, and revision loops stay on the workstation. But making it coexist with GPU-heavy video work requires deliberate resource isolation, not just a memory utilization flag.

This architecture is a deployment boundary around OpenMontage's coding-assistant workflow, not a description of verified built-in inference adapters. Repository-specific configuration still needs checking against the version you deploy.

Separate the Director From the Machinery

The end-to-end path should be explicit:

User brief → assistant and skill selection → bounded inference request → validated tool invocation → production worker → artifact store.

The inference server generates decisions. Production workers execute media operations. Durable job state and artifacts belong outside both processes.

Put a bounded queue before inference. Keep tool output, transcripts, and production knowledge outside the prompt until needed; retrieve relevant sections instead of concatenating the entire project history. Apply the token budget after assembling the complete request, including tool schemas.

Most importantly, a GPU-memory utilization flag is not a hardware partition. If inference and rendering share one GPU, serialize their GPU-heavy phases or enforce isolation outside the model server. The configuration below assumes a dedicated 24 GiB inference GPU.

Why the Toy Setup Fails

The 70B OOM incident worth diagnosing is not necessarily a bad model load. Partial layer offloading can make weights fit while leaving insufficient room for the configured context and concurrent requests.

KV-cache capacity depends on architecture, cache precision, sequence length, and concurrency. Weight quantization does not automatically quantize that cache. Increasing context while preserving concurrency can consume the remaining headroom.

Long-running throughput degradation introduces another trap: retained allocator memory and an intentionally preallocated KV-cache pool can resemble a leak. A rising nvidia-smi number alone cannot establish one. Compare equivalent workloads, active-request counts, and cache occupancy after requests drain.

For an initial integration, I would use a smaller model with bounded context rather than force a partially offloaded 70B model onto the production workstation.

One Bounded Server, One Streaming Probe

This configuration uses Qwen2.5-7B, an 8,192-token context, four active sequences, and an 80% GPU-memory budget. These are conservative starting limits, not universally optimal settings.

Run in a CUDA-compatible Linux environment with Python 3.11 and nvidia-smi. Select the inference GPU through CUDA_VISIBLE_DEVICES; GPU_INDEX must identify that same physical GPU for measurement.

python -m pip install 'vllm==0.10.2' aiohttp

export CUDA_VISIBLE_DEVICES=0
export GPU_INDEX=0
export MODEL=Qwen/Qwen2.5-7B-Instruct
export BASE_URL=http://127.0.0.1:8000/v1

vllm serve "$MODEL" \
  --host 127.0.0.1 --port 8000 \
  --dtype half \
  --kv-cache-dtype auto \
  --gpu-memory-utilization 0.80 \
  --max-model-len 8192 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 4096 \
  --enable-chunked-prefill &
SERVER_PID=$!
trap 'kill "$SERVER_PID" 2>/dev/null || true' EXIT

python - <<'PY'
import asyncio, json, os, statistics, time
import aiohttp

BASE = os.environ["BASE_URL"].rstrip("/")
MODEL = os.environ["MODEL"]
GPU = os.environ.get("GPU_INDEX", "0")
HEADERS = {}
if os.environ.get("API_KEY"):
    HEADERS["Authorization"] = "Bearer " + os.environ["API_KEY"]

async def memory():
    p = await asyncio.create_subprocess_exec(
        "nvidia-smi", "-i", GPU,
        "--query-gpu=memory.used", "--format=csv,noheader,nounits",
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE)
    out, err = await p.communicate()
    if p.returncode:
        raise RuntimeError(err.decode())
    return int(out.decode().strip())

async def request(session):
    start, first = time.perf_counter(), None
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content":
            "Write a concise six-shot production plan for a coffee advert."}],
        "max_tokens": 128, "temperature": 0, "stream": True,
    }
    async with session.post(
        BASE + "/chat/completions", json=payload
    ) as response:
        response.raise_for_status()
        async for raw in response.content:
            line = raw.decode().strip()
            if not line.startswith("data:"):
                continue
            data = line[5:].strip()
            if data == "[DONE]":
                break
            event = json.loads(data)
            if "error" in event:
                raise RuntimeError(event["error"])
            for choice in event.get("choices", []):
                if choice.get("delta", {}).get("content") and first is None:
                    first = time.perf_counter() - start
    if first is None:
        raise RuntimeError("Stream returned no content")
    return first, time.perf_counter() - start

async def main():
    timeout = aiohttp.ClientTimeout(total=180, sock_read=60)
    async with aiohttp.ClientSession(
        headers=HEADERS, timeout=timeout
    ) as session:
        for attempt in range(180):
            try:
                async with session.get(BASE + "/models") as r:
                    if r.status == 200:
                        break
            except aiohttp.ClientError:
                pass
            await asyncio.sleep(2)
        else:
            raise RuntimeError("Server readiness timed out")

        await request(session)  # Warm up before measuring.
        baseline = await memory()
        samples = [baseline]
        async def sample():
            while True:
                samples.append(await memory())
                await asyncio.sleep(0.25)

        semaphore = asyncio.Semaphore(2)
        async def bounded():
            async with semaphore:
                return await request(session)

        monitor = asyncio.create_task(sample())
        try:
            results = await asyncio.gather(*(bounded() for _ in range(8)))
        finally:
            monitor.cancel()
            try:
                await monitor
            except asyncio.CancelledError:
                pass

        print(json.dumps({
            "requests": len(results),
            "median_ttft_s": statistics.median(x[0] for x in results),
            "median_latency_s": statistics.median(x[1] for x in results),
            "gpu_baseline_mib": baseline,
            "gpu_peak_mib": max(samples),
            "gpu_peak_delta_mib": max(samples) - baseline,
            "gpu_after_mib": await memory(),
        }, indent=2))

asyncio.run(main())
PY

Interpret the Boundary, Not Just the Numbers

TTFT starts when the request enters the HTTP client; latency ends when streaming completes. Both exclude time waiting for the benchmark's semaphore. Production queue delay needs its own measurement.

GPU measurements cover the entire device, not just vLLM. Sampling can miss brief peaks, and a preallocated cache can produce almost no measured delta. This is a smoke benchmark, not a capacity certification.

For an optional remote comparison, the same harness can target B-Lost's relay endpoint, https://b-lost.com/v1, using an authorized API_KEY and an available model identifier. I have not measured that endpoint here. Local GPU readings would not describe its server-side memory.

The trade-off is deliberate: bounded context and concurrency sacrifice flexibility to protect production work. Expand either only after replaying representative tool-heavy requests alongside the actual rendering schedule.

---

The hardest operational question is not whether to bound inference resources, but when to stop defending the boundary and admit that inference and rendering need separate hardware. If you are running 70B models with 32k context windows while rendering 4K video timelines, no amount of semaphore discipline will save you. At what utilization threshold do you declare victory for clever resource sharing and buy another GPU? I have drawn that line differently on every project. What does your architecture assume, and where has it broken?