Adrian Makridenko

Posted on Sep 2 Originally published at makridenko.ru

Building an Internal LLM: vLLM, OpenWebUI, and a Few Hacks

#ai #webdev #tutorial

It so happened that the company I currently work for wasn't just unfamiliar with AI agents before I joined — most developers had never even seen what all this AI hype was about. Classic enterprise environment.
As a result, I ended up becoming both the initiator and the driving force behind introducing AI into the development process.

So I decided to share my experience and show exactly what I did, so you can reproduce it yourself if needed.

Since this was still a heavily regulated enterprise environment, the question of running an LLM locally, inside the company perimeter, came up almost immediately. Claude, Codex, and other cloud-based solutions were completely off-limits.

What I Had

After several months of negotiations, I was finally given a machine with an NVIDIA H100, and I started figuring out how all of this worked. My previous experience was limited to running models on a home PC, so at first it seemed like everything would be more or less the same.
It didn't take long to discover that enterprise infrastructure always has a few surprises waiting for you.
My GPU only had 80 GB of VRAM. By modern model standards, that's not a lot, so model selection had to be done carefully.

I considered:

DeepSeek
Qwen3
Qwen3.6
Qwen3-Coder

After a few tests, I settled on Qwen/Qwen3-Coder-30B-A3B-Instruct because deepseek-ai/deepseek-coder-33b-base (the only reasonably usable DeepSeek model that fit into memory) turned out to be fairly weak, while Qwen3.6 wouldn't start at all because of driver issues.

First vLLM Launch

Like any reasonable person, I started by deploying the model with vLLM and calling it directly through the API.

mkdir vllm && cd vllm

pip install --upgrade pip
pip install uv

uv venv --python 3.12 --seed --managed-python
source ~/vllm/.venv/bin/activate

uv pip install -U \
  "transformers<5" \
  "vllm==0.10.2"

uv run vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
  --host 0.0.0.0 \
  --port 8000

That's where the adventure began.

When Enterprise Infrastructure Meets LLMs

Some of you may have noticed that the versions of vllm and transformers are far from the latest. That wasn't accidental.
The GPU was provisioned through MIG, and along with it came a whole collection of limitations. The biggest issue was the drivers. I couldn't update them, and due to a combination of architectural constraints and bureaucracy, the administrators couldn't update them either.
That's exactly why I never managed to get Qwen 3.6 running.

But the fun didn't stop there.
Every night at around 2:00 AM, the MIG instance would disappear from the system for a few seconds and then come back with a new UID. I had no interest in figuring out who was responsible, so I chose the most engineering-oriented solution possible:
Every morning at 5:00 AM, the server simply reboots.
As a result:

a fresh UID appears;
accumulated issues disappear;
the system gets a preventive reboot before the workday starts.

Yes, it's a hack, but it works.

Why vLLM Alone Isn't Enough

For several days I used the model on my own without any issues.
It quickly became obvious, however, that this approach only works for a single person. If other employees started using the model, I would need:

a web interface;
user management;
integration with corporate authentication;
the ability to revoke access quickly;
an open-source solution that could be modified internally.

After a bit of research, I settled on OpenWebUI. It had everything I needed and then some.

OpenWebUI as the Entry Point

Initially, OpenWebUI lived on the same machine as vLLM.
Very quickly, though, it became obvious that it was consuming resources I'd rather leave available for the model itself. So I deployed a separate virtual machine.

The resulting architecture looked like this:
The user interacts with OpenWebUI, and OpenWebUI sends requests to vLLM.

This turned out to be convenient for several reasons:

access can be restricted to specific users or groups;
usage statistics can be collected;
response ratings can be gathered;
system prompts can be managed centrally.

Squeezing the Most Out of 80 GB of Memory

At this point I started experimenting with model settings.
My first idea was to increase the context window.
I tried this:

uv run vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
  --max-model-len 120000

It didn't work.

After a series of experiments, I arrived at the following configuration:

uv run vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype bfloat16 \
  --max-model-len 70000 \
  --gpu-memory-utilization 0.82 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --served-model-name Qwen-Coder \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --kv-cache-dtype fp8

This gave me roughly 70,000 tokens of context.
Unfortunately, it didn't take long to discover that even that wasn't enough.

Hack #1: Rebuilding the Cache After Reboot

The most obvious solution was to use prefix caching. The problem was that every morning the server rebooted, and the entire cache disappeared along with it.

So another hack was born.
After startup, the system automatically sends a predefined set of popular requests to the model.

# -*- coding: utf-8 -*-
# /opt/openwebui/scripts/token_recache_service.py

RECACHE_PROMPT: str = load_recache_data()

client.chat.completions.create(
    model="qwen",
    messages=[
        {"role": "system", "content": RECACHE_PROMPT},
        {"role": "user", "content": "warmup"}
    ]
)

In practice, I'm simply forcing the model to recompute the tokens I want cached.
The list of popular requests was assembled together with an AI agent and continues to grow over time.

So far, this solution has been sufficient.

Wrapping Everything Into a Service

From there, it was mostly standard infrastructure work.

I deployed PostgreSQL instead of SQLite, configured nginx, issued certificates, and locked down access to the vLLM server as much as possible.

Only OpenWebUI is allowed to communicate with the model.

services:
  postgres:
    image: postgres:17
    container_name: open-webui-postgres
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped

This immediately solved several problems.

First, nobody can access the model directly anymore.
Second, if I need to perform maintenance or testing, I can simply disable the model in OpenWebUI.
Third, the system is no longer limited to developers using VSCode, Zed, or Opencode. Any employee can use it through a browser.

Hack #2: The Model Wants Coffee Too

After some time, I noticed an interesting pattern.
If nobody used the model for a long period, the first few requests in the morning performed noticeably worse. Responses took longer, hallucinations became more frequent, and overall behavior felt strange.

After several requests, everything returned to normal.
At first I assumed the GPU was simply sitting idle and "cooling down."
I did some searching, found that similar observations weren't unique to me, and wrote a simple script that generated a small amount of continuous GPU load.

import time
import torch

DEVICE = "cuda"

a = torch.randn((2048, 2048), device=DEVICE, dtype=torch.float16)
b = torch.randn((2048, 2048), device=DEVICE, dtype=torch.float16)

while True:
    c = torch.matmul(a, b)
    torch.cuda.synchronize()
    time.sleep(10)

It worked. But it felt like I was wasting resources.
Later I replaced it with a different approach.
Now, every few minutes, the system sends a meaningless request to the model:

import os
import time
import requests

URL = os.getenv("LLM_API")
TIMEOUT = 300

while True:
    try:
        requests.post(
            URL,
            json={
                "model": "qwen",
                "messages": [
                    {"role": "user", "content": "ping"}
                ],
                "max_tokens": 1,
            },
            timeout=30,
        )
    except Exception:
        pass

    time.sleep(TIMEOUT)

As a result, the model never stays idle for too long, and the first real user requests tend to perform much more consistently.

Final Architecture

As a finishing touch, our DevOps engineer and I configured authentication through FreeIPA, issued an internal certificate, and added the service to the corporate DNS.

The final architecture ended up looking like this:

What's Next?

The system is now being actively used within the company and is gradually being enhanced with additional features. As a next step, I want to move the project knowledge into a separate RAG service and focus separately on the long-term storage and restoration of caches. And very soon I’ll have a machine running on an H200, where I’ll be testing more sophisticated models.

But that’s a story for the next article.

DEV Community