DEV Community: Adrian Makridenko

Building an Internal LLM: vLLM, OpenWebUI, and a Few Hacks

Adrian Makridenko — Wed, 02 Sep 2026 15:41:47 +0000

It so happened that the company I currently work for wasn't just unfamiliar with AI agents before I joined — most developers had never even seen what all this AI hype was about. Classic enterprise environment.
As a result, I ended up becoming both the initiator and the driving force behind introducing AI into the development process.

So I decided to share my experience and show exactly what I did, so you can reproduce it yourself if needed.

Since this was still a heavily regulated enterprise environment, the question of running an LLM locally, inside the company perimeter, came up almost immediately. Claude, Codex, and other cloud-based solutions were completely off-limits.

What I Had

After several months of negotiations, I was finally given a machine with an NVIDIA H100, and I started figuring out how all of this worked. My previous experience was limited to running models on a home PC, so at first it seemed like everything would be more or less the same.
It didn't take long to discover that enterprise infrastructure always has a few surprises waiting for you.
My GPU only had 80 GB of VRAM. By modern model standards, that's not a lot, so model selection had to be done carefully.

I considered:

DeepSeek
Qwen3
Qwen3.6
Qwen3-Coder

After a few tests, I settled on Qwen/Qwen3-Coder-30B-A3B-Instruct because deepseek-ai/deepseek-coder-33b-base (the only reasonably usable DeepSeek model that fit into memory) turned out to be fairly weak, while Qwen3.6 wouldn't start at all because of driver issues.

First vLLM Launch

Like any reasonable person, I started by deploying the model with vLLM and calling it directly through the API.

mkdir vllm && cd vllm

pip install --upgrade pip
pip install uv

uv venv --python 3.12 --seed --managed-python
source ~/vllm/.venv/bin/activate

uv pip install -U \
  "transformers<5" \
  "vllm==0.10.2"

uv run vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
  --host 0.0.0.0 \
  --port 8000

That's where the adventure began.

When Enterprise Infrastructure Meets LLMs

Some of you may have noticed that the versions of vllm and transformers are far from the latest. That wasn't accidental.
The GPU was provisioned through MIG, and along with it came a whole collection of limitations. The biggest issue was the drivers. I couldn't update them, and due to a combination of architectural constraints and bureaucracy, the administrators couldn't update them either.
That's exactly why I never managed to get Qwen 3.6 running.

But the fun didn't stop there.
Every night at around 2:00 AM, the MIG instance would disappear from the system for a few seconds and then come back with a new UID. I had no interest in figuring out who was responsible, so I chose the most engineering-oriented solution possible:
Every morning at 5:00 AM, the server simply reboots.
As a result:

a fresh UID appears;
accumulated issues disappear;
the system gets a preventive reboot before the workday starts.

Yes, it's a hack, but it works.

Why vLLM Alone Isn't Enough

For several days I used the model on my own without any issues.
It quickly became obvious, however, that this approach only works for a single person. If other employees started using the model, I would need:

a web interface;
user management;
integration with corporate authentication;
the ability to revoke access quickly;
an open-source solution that could be modified internally.

After a bit of research, I settled on OpenWebUI. It had everything I needed and then some.

OpenWebUI as the Entry Point

Initially, OpenWebUI lived on the same machine as vLLM.
Very quickly, though, it became obvious that it was consuming resources I'd rather leave available for the model itself. So I deployed a separate virtual machine.

The resulting architecture looked like this:
The user interacts with OpenWebUI, and OpenWebUI sends requests to vLLM.

This turned out to be convenient for several reasons:

access can be restricted to specific users or groups;
usage statistics can be collected;
response ratings can be gathered;
system prompts can be managed centrally.

Squeezing the Most Out of 80 GB of Memory

At this point I started experimenting with model settings.
My first idea was to increase the context window.
I tried this:

uv run vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
  --max-model-len 120000

It didn't work.

After a series of experiments, I arrived at the following configuration:

uv run vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype bfloat16 \
  --max-model-len 70000 \
  --gpu-memory-utilization 0.82 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --served-model-name Qwen-Coder \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --kv-cache-dtype fp8

This gave me roughly 70,000 tokens of context.
Unfortunately, it didn't take long to discover that even that wasn't enough.

Hack #1: Rebuilding the Cache After Reboot

The most obvious solution was to use prefix caching. The problem was that every morning the server rebooted, and the entire cache disappeared along with it.

So another hack was born.
After startup, the system automatically sends a predefined set of popular requests to the model.

# -*- coding: utf-8 -*-
# /opt/openwebui/scripts/token_recache_service.py

RECACHE_PROMPT: str = load_recache_data()

client.chat.completions.create(
    model="qwen",
    messages=[
        {"role": "system", "content": RECACHE_PROMPT},
        {"role": "user", "content": "warmup"}
    ]
)

In practice, I'm simply forcing the model to recompute the tokens I want cached.
The list of popular requests was assembled together with an AI agent and continues to grow over time.

So far, this solution has been sufficient.

Wrapping Everything Into a Service

From there, it was mostly standard infrastructure work.

I deployed PostgreSQL instead of SQLite, configured nginx, issued certificates, and locked down access to the vLLM server as much as possible.

Only OpenWebUI is allowed to communicate with the model.

services:
  postgres:
    image: postgres:17
    container_name: open-webui-postgres
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped

This immediately solved several problems.

First, nobody can access the model directly anymore.
Second, if I need to perform maintenance or testing, I can simply disable the model in OpenWebUI.
Third, the system is no longer limited to developers using VSCode, Zed, or Opencode. Any employee can use it through a browser.

Hack #2: The Model Wants Coffee Too

After some time, I noticed an interesting pattern.
If nobody used the model for a long period, the first few requests in the morning performed noticeably worse. Responses took longer, hallucinations became more frequent, and overall behavior felt strange.

After several requests, everything returned to normal.
At first I assumed the GPU was simply sitting idle and "cooling down."
I did some searching, found that similar observations weren't unique to me, and wrote a simple script that generated a small amount of continuous GPU load.

import time
import torch

DEVICE = "cuda"

a = torch.randn((2048, 2048), device=DEVICE, dtype=torch.float16)
b = torch.randn((2048, 2048), device=DEVICE, dtype=torch.float16)

while True:
    c = torch.matmul(a, b)
    torch.cuda.synchronize()
    time.sleep(10)

It worked. But it felt like I was wasting resources.
Later I replaced it with a different approach.
Now, every few minutes, the system sends a meaningless request to the model:

import os
import time
import requests

URL = os.getenv("LLM_API")
TIMEOUT = 300

while True:
    try:
        requests.post(
            URL,
            json={
                "model": "qwen",
                "messages": [
                    {"role": "user", "content": "ping"}
                ],
                "max_tokens": 1,
            },
            timeout=30,
        )
    except Exception:
        pass

    time.sleep(TIMEOUT)

As a result, the model never stays idle for too long, and the first real user requests tend to perform much more consistently.

Final Architecture

As a finishing touch, our DevOps engineer and I configured authentication through FreeIPA, issued an internal certificate, and added the service to the corporate DNS.

The final architecture ended up looking like this:

What's Next?

The system is now being actively used within the company and is gradually being enhanced with additional features. As a next step, I want to move the project knowledge into a separate RAG service and focus separately on the long-term storage and restoration of caches. And very soon I’ll have a machine running on an H200, where I’ll be testing more sophisticated models.

But that’s a story for the next article.

Delta by Zed

Adrian Makridenko — Mon, 31 Aug 2026 09:24:33 +0000

I got into the Delta whitelist and spent some time using it for work and not-so-work-related stuff. And I really liked it — this is exactly what I was missing. It’s not some ugly terminal thing written in JS and React, nor a janky stripped-down extension for Zed or VS Code. It’s a native, fast app that does a lot right out of the box.

What does it have that Claude Code, Codex, and OpenCode don’t?

First of all, really damn good code review

Before this, I used revdiff by Umputun, but objectively speaking, it’s a pretty hacky workaround. Opening some TUI program through tmux and terminal overlays, only for it to write something like line_number: comment when you exit — eh, not exactly great.

Here, you just walk through the code, leave comments, and then send them to the agent in one action. It’s very similar to how we do reviews in GitLab or GitHub. Really damn nice.

Second, worktree-based workflow

The folks at Atom — the same people building Zed and Delta — have been pushing Git worktrees for a long time, going all the way back to the Atom days.

I like this approach too. Especially in the age of agentic coding, I don’t really want to let an agent loose directly in my working tree. It’s much more convenient to create a worktree from the current state of the project and let the agent have fun there. After the review, the changes can be moved into the main branch.

That’s exactly what Delta does: each thread gets its own worktree, and all the work happens there. It’s both safer and more convenient, especially when multiple agents are working in parallel.

Third, a rather niche use case, but...

Collaborative work.

I can share a session and work on a task together with a colleague in real time. I really like this feature in Zed itself. At work, I occasionally use it to look at code together with colleagues, and it’s much more convenient than sharing your screen during a call and trying to show someone what you’re doing.

And once coding started turning into chasing AI agents around, it became logical to share not the code itself, but the session: the plans, the reasoning, the same code, and now also the diffs produced by the agent.

Not many people need this, but personally, I consider it one of the best features in both Zed and Delta.

Fourth, NATIVENESS

YES. GOD. YES.

I am so damn tired of this entire zoo of Electron, JS, and TS. Why? What for?

(Ironically, the people who created Zed also created Electron for Atom back in the day, and now they’re trying to do everything natively. Making up for their sins 100%.)

Why, for example, would you write Bitwarden in TypeScript? Both the desktop app and the CLI. Why would you write Claude Code in TypeScript? And React on top of that?

Electron has generally become a genuine plague of modern desktop software, so the fact that Delta is fast and native is a huge plus.

What is it actually missing?

First and foremost, custom providers.

I really want to connect a work model using our own tokens (mostly so the security people don’t come after me) and work through it. Maybe this isn’t available yet simply because Delta is still in closed beta. But I didn’t see a single word about custom providers on the roadmap.

Then, of course, Git integration.

Right now, they suggest moving changes over using stash or checking out commits from the worktree in your terminal outside of Delta. I’m not really against that, but I’d like to have a slightly more convenient way of doing it. Fortunately, this is already on the roadmap, so they’ll probably improve it.

And finally, switching agents.

I’d like to first build a plan with a read-only agent and then hand it over to another agent to actually do the work. That’s not possible yet, but we’ll see how the product evolves.

At some point while working, I caught myself thinking that I don’t really perceive Delta as yet another tool for working with AI — or even as an “agentic IDE.”

With the arrival of full-fledged agents in our workflow, the development process itself has changed. Previously, a code editor was primarily where you wrote code yourself. Now, you open it mostly to read the code the agent wrote — and even that happens pretty rarely, because the diffs in Claude Code / Codex are often enough.

In Delta, all of this is there. You can read the entire project, manually edit the code if you need to, and leave comments for the robot even outside of diffs.

That’s exactly why Delta feels less like yet another little chat window bolted onto an editor and more like an attempt to build the next generation of code editor.

Here, the agent isn’t an add-on to the familiar workflow anymore. It becomes one of the core entities of the system, alongside the code (or perhaps even above it), Git, and the developer.