I use Claude and ChatGPT every day. At some point I started wondering: how does the text actually appear word by word like that? Why does it feel instant even though the model is generating a whole response? Why do some responses feel faster than others? And what does it really cost to run one of these things – in hardware?
So I built my own, from scratch. Not to replace Claude or GPT — I don't have a datacenter 😅 — but to understand the stack. The result is InferenceX: a self-hosted AI inference server that runs on a single laptop GPU, with a terminal chat interface, a side-by-side model comparison tool, and a benchmark suite that tells you which model actually fits your hardware.
This is what I learned.
First: how does the text appear word by word?
When you type a message to Claude or ChatGPT, it doesn't generate the entire response and then send it all at once. It generates one token at a time — a token being roughly one word, or part of a word — and streams each one to you as soon as it's ready.
This technique is called server-sent events (SSE). The server keeps the HTTP connection open and pushes small chunks of data as they're generated, instead of waiting to send everything at the end.
Client sends: POST /v1/chat/completions {"stream": true, ...}
Server replies (over time, same connection):
data: {"choices":[{"delta":{"content":"The"}}]}
data: {"choices":[{"delta":{"content":" quick"}}]}
data: {"choices":[{"delta":{"content":" brown"}}]}
data: {"choices":[{"delta":{"content":" fox"}}]}
...
data: [DONE]
That's the entire mechanism. "Streaming" in AI chat is just HTTP kept alive, with the model sending tokens as fast as it generates them.
The reason it feels fast isn't that the model is fast — it's that you see something almost immediately. The time from sending your message to seeing the first word appear is called time-to-first-token (TTFT). On a well-tuned server this is under 200ms. The total generation might take 10 seconds, but you're reading while it generates, so it feels much quicker than waiting for the full response.
Here's what that looks like in my terminal chat interface:
The chat CLI streaming a response token by token. Each word appears as the model generates it — the same mechanism Claude and ChatGPT use.
The hardware reality
This is the part that surprised me most.
I have two machines I tested this on: a laptop with an NVIDIA RTX 3060 (6 GB VRAM) and a desktop with an RTX 4060 (8 GB VRAM). That sounds like a lot on paper — modern AAA games run fine on both. But loading even a small language model immediately consumed most of that memory:
On the 3060 (6 GB), loading qwen2.5-0.5b breaks down like this:
| What's using VRAM | How much |
|---|---|
| GPU driver overhead | ~0.5 GB |
Model weights (qwen2.5-0.5b) |
~1.0 GB |
| KV cache (where the model stores context while thinking) | ~0.5 GB |
| Total | ~2 GB out of 6 GB |
InferenceX caps the context window at 8,192 tokens for small models — enough for any realistic local conversation — so the KV cache stays small and leaves room to run a second model alongside.
And qwen2.5-0.5b is a tiny model — 500 million parameters. Neither Anthropic nor OpenAI publicly discloses the parameter counts for their production models — but based on what these models can do, they are widely estimated to be in the hundreds of billions to trillions of parameters.
The models running Claude.ai and ChatGPT require hundreds of gigabytes of VRAM. Anthropic and OpenAI run them on clusters of specialized server GPUs (H100s and newer B200s) with 80–192 GB of memory each — and they run dozens of them in parallel to serve millions of users simultaneously.
My laptop GPU running a 1B parameter model is the absolute floor. It's like comparing a bicycle to a cargo ship.
| Model | Parameters | Throughput (RTX 3060) | Throughput (RTX 4060) | max_model_len | VRAM delta |
|---|---|---|---|---|---|
| opt-125m | 125M | 325 tok/s | 403 tok/s | 2048 | ~5.2 GB |
| tinyllama-chat | 1.1B | 79 tok/s | 97 tok/s | 2048 | ~5.0 GB |
| qwen2.5-0.5b | 500M | 137 tok/s | 179 tok/s | 8192 | ~5.4 GB |
| minicpm5-1b | 1B | 96 tok/s | 127 tok/s | 8192 | ~5.3 GB |
| qwen2.5-1.5b | 1.5B | — | 71 tok/s | 8192 | ~7.1 GB |
Note: qwen2.5-1.5b was only benchmarked on RTX 4060; it exceeds available VRAM on the RTX 3060 Laptop (6 GB) at max_model_len 8192.
minicpm5-1b and qwen2.5-1.5b are newly added models in v0.1.2; both require at least 8 GB VRAM for comfortable operation.
Here's the model advisor built into InferenceX, showing what my hardware can actually support:
The model advisor showing available VRAM and which configured models are viable. Models that would OOM at load time get marked viable=False and score 0.
This is why cloud AI services are priced the way they are. The hardware is genuinely expensive and the memory requirements are enormous.
What I actually built
InferenceX has four main parts:
An OpenAI-compatible inference API. POST /v1/chat/completions works exactly like OpenAI's API — same request shape, same response shape, same streaming protocol. Any client that works with OpenAI works with InferenceX. I used vLLM as the actual inference engine — it handles the hard GPU math; I built the API layer around it.
A model registry and router. Models are configured in a YAML file. The server loads whichever ones you specify at startup and routes requests to the right one. Multiple models can live in one server process if they fit in VRAM together.
An observability layer. Every request is timed and counted — latency, token counts, error rate — without touching any API handler code. This is done with middleware: code that wraps every request/response automatically.
Three interactive surfaces:
-
make chat— the daily-driver terminal chat -
make playground— side-by-side model comparison -
make benchmark/make advise— throughput benchmarks and hardware-aware recommendations
The loading screen
The first thing you see when you run make chat or make playground is a loading screen. The model doesn't load instantly — on first run it may need to download several gigabytes of weights from HuggingFace, then load them into GPU memory, then warm up the CUDA kernels. This takes 90 seconds to several minutes.
Rather than showing a spinner, I wired the loading screen to tail the server's actual log file in real time, filtering the raw log output into human-readable status lines:
The loading screen reading from the server log in real time. "GPU memory for model=opt-125m: utilization=0.184" means the model has claimed 18.4% of available VRAM.
What you're watching is vLLM:
- Loading model weights from disk (or downloading them)
- Allocating KV cache blocks in GPU memory
- Running CUDA graph captures to optimize token generation
- Signaling ready
Once that sequence completes, the chat interface appears.
Comparing two models side by side
One of the more useful things you can do with a self-hosted setup is compare how different models respond to the same prompt — not just the quality of the answer, but the speed.
make playground loads two models simultaneously and streams both responses in parallel:
Two models responding to the same prompt simultaneously on an RTX 4060. opt-125m finished in 1.0s with 13 completion tokens. qwen2.5-0.5b took 6.1s and generated 330 completion tokens — a structured, on-topic answer.
| opt-125m | qwen2.5-0.5b | |
|---|---|---|
| Time | 1.0s | 6.1s |
| Completion tokens | 13 | 330 |
| Total tokens | 16 | 333 |
opt-125m replied in one second: "That's what I'm doing. I will be making an album about it tomorrow." Thirteen tokens, done. It did not answer the question — it hallucinated a continuation as if mid-conversation.
qwen2.5-0.5b took six seconds and produced a full structured response: bullet points covering customer service, chatbots, NLP, virtual assistants, data analysis, and cybersecurity, plus a concluding paragraph on how NLP is a rapidly growing field. Same prompt, same hardware, radically different usefulness.
The status bar read Done — 346 tokens in 6.1s — wall-clock time is dominated by the slower model, since both stream in parallel.
A few observations from running this:
Smaller parameter count does not mean better. opt-125m (125 million parameters) was 6× faster, but the response is incoherent relative to the prompt. qwen2.5-0.5b (500 million parameters) is 4× larger and instruction-tuned. The quality gap is not subtle — it's the difference between a random sentence fragment and a usable answer.
Token count explains the latency gap. opt-125m generated 13 tokens; qwen2.5-0.5b generated 330. At ~55 tok/s effective throughput for the longer response, 330 tokens in 6.1 seconds is consistent. The speed difference is mostly how much text each model chose to produce, not a 6× throughput gap on the GPU.
Throughput is what matters for feel. qwen2.5-0.5b on my RTX 4060 produces ~179 tok/s in isolated benchmarks. Human reading speed is roughly 250 words per minute — around 4 words per second. At that generation rate, text appears faster than you can read it, which is why streaming chat feels instant even when a 330-token answer takes six seconds end to end.
What the code actually looks like
The core of the streaming path is about 30 lines:
# ChatService — the part that drives the model and yields tokens
async def stream_response(self, request: ChatRequest) -> AsyncIterator[str]:
engine = self._pool.get(request.model)
prompt = self._build_prompt(request.messages)
async for token in engine.generate_stream(prompt, request.sampling_params):
# Format each token as an OpenAI-compatible SSE chunk
chunk = {
"object": "chat.completion.chunk",
"choices": [{"delta": {"content": token}, "index": 0}]
}
yield f"data: {json.dumps(chunk)}\n\n"
yield "data: [DONE]\n\n"
# FastAPI route handler — just calls the service and returns a streaming response
@router.post("/v1/chat/completions")
async def chat_completions(request: ChatRequest, deps = Depends(get_deps)):
if request.stream:
return StreamingResponse(
deps.chat_service.stream_response(request),
media_type="text/event-stream",
)
# non-streaming path omitted for brevity
That's it. StreamingResponse keeps the connection open and flushes each yielded chunk to the client immediately. The client — whether it's my terminal chat, curl, or any OpenAI SDK — reads the SSE stream and renders tokens as they arrive.
The tricky part is not the streaming itself. It's everything around it: loading the right model, routing to the right engine, measuring latency without slowing down the response, and handling errors cleanly.
Two things that went wrong (and how I fixed them)
1. vLLM on WSL2 would crash after loading weights
What happened: The model weights loaded successfully, then vLLM died with:
Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist
Why: vLLM defaults to using FlashInfer for token sampling — a library that JIT-compiles custom CUDA kernels at runtime. WSL2 (Windows Subsystem for Linux) typically has GPU drivers but no CUDA compiler installed at the expected path. The compilation failed.
Fix: One environment variable: VLLM_USE_FLASHINFER_SAMPLER=0. This tells vLLM to use PyTorch's built-in sampler instead — no compilation required. I set this automatically in the codebase when a WSL2 environment is detected.
This took several hours to diagnose because the error occurs after the weights load, making it look like a memory or driver issue rather than a compiler issue.
2. Reading the response body for metrics broke the response
The problem: The observability middleware needs to read the response body to extract token counts. But HTTP responses in Python's Starlette framework are streams — once you read the bytes, they're gone. The client receives an empty response.
The fix: Don't patch the stream. Consume it, then construct a fresh Response object with the same bytes:
# Consume the stream
body = b""
async for chunk in response.body_iterator:
body += chunk
# Extract token counts from the JSON
usage = json.loads(body).get("usage", {})
# Return a NEW response — Starlette recalculates content-length automatically
return Response(
content=body,
status_code=response.status_code,
headers={k: v for k, v in response.headers.items() if k != "content-length"},
)
This only applies to non-streaming responses. For SSE streams, buffering is not possible (and not needed — there's no single JSON body to parse).
One thing I got wrong (and had to fix)
When I first configured the VRAM allocation, I hardcoded gpu_memory_utilization: in the model config — which I thought meant "use 90% of available memory."
0.90
It doesn't. vLLM treats it as 90% of total VRAM — regardless of what the GPU
driver, desktop compositor, or other processes already have in use.
On my 8 GB card with ~1 GB of overhead, 0.90 required 7.2 GB. The server
refused to start:
RuntimeError: Insufficient GPU memory to start qwen2.5-0.5b.
Free VRAM 6.93 GiB is less than gpu_memory_utilization=0.9 requires
(7.20 GiB of 8.00 GiB total).
The fix: calculate the right value from what's actually free, not from the total:
gpu_memory_utilization = (vram_free − safety_buffer) / vram_total
= (6.93 − 0.4) / 8.0
= 0.82
InferenceX now computes this at startup using torch.cuda.mem_get_info —
the same allocator view vLLM uses — and logs it:
Loading qwen2.5-0.5b
gpu_memory_utilization: auto → 0.82
(6.93 GB free − 0.40 GB buffer) / 8.00 GB total
max_model_len: 8192
The gpu_memory_utilization: auto config works on any machine without manual tuning, whether you have 6 GB or 24 GB of VRAM.
That fix shipped early. It turned out to be half the story — and the other half took a bad benchmark run to surface.
The bug that used 7 GB of VRAM to run a 125M-parameter model
A week after the fix above, I ran a routine benchmark on the smallest model in the registry — opt-125m, 125 million parameters, the kind of model that should be trivial to fit in almost any amount of VRAM:
Mean throughput : 16.7 tok/s
p50 latency : 11030 ms
Peak VRAM delta : 7.06 GB
Both numbers were wrong in the same direction: an 8 GB card was 88% full, and the server was slower than it had been weeks earlier. Something had regressed, and it took two separate bugs to explain it.
Bug one — "auto" ignored the model entirely. The gpu_memory_utilization: auto logic I described above only ran for the first engine sizing pass. A later refactor added a shortcut: if the value was "auto", grab (free_vram − buffer) / total_vram and stop — the same flat fraction regardless of whether the model was 125M or 7B parameters. A tiny model and a huge model requesting "auto" got sized identically. opt-125m claimed the same slice of the GPU a model 50× its size would need.
The fix was to make "auto" mean what it always should have: "no manual ceiling set," not "ignore the model's actual weight and KV-cache footprint." Both auto and an explicit number now flow through the same footprint calculation — weights, KV cache, and vLLM's own runtime overhead, all estimated from the model's real HuggingFace config before a single byte is allocated.
Bug two — the server was quietly rate-limited. Streaming and non-streaming requests share one thread that owns the model and calls its step() function in a loop — one thread per model, so two concurrent requests to the same model never race for the same GPU call. That thread had a one-twentieth-of-a-second poll-and-wait built into every loop iteration, including the ones where work was already queued up and ready to run. The intent was to avoid spinning the CPU when there was nothing to do — but the same wait fired even mid-generation, capping every model in the registry to roughly 20 steps per second no matter how fast the GPU could actually go. The fix: only wait when the queue is genuinely empty; otherwise go straight to the next step.
Fixing both together:
| Before | After | |
|---|---|---|
| Peak VRAM (opt-125m) | 7.06 GB | 1.65 GB |
| Throughput | 16.7 tok/s | 329.3 tok/s |
| p50 latency | 11,030 ms | 518 ms |
Same model, same GPU, same request. One bug was silently reserving 4× the VRAM the model needed; the other was silently capping every model's speed to a fifth of what the hardware could deliver. Neither showed up as an error — the server started fine and answered requests fine, just wastefully and slowly. That's the uncomfortable part of self-hosting inference: correctness bugs throw exceptions, capacity bugs just quietly cost you.
Testing whether a "real" 2B model fits
With both fixes in place, I went back and re-benchmarked every model in the registry, and specifically tried to answer a question I'd been putting off: can this 8 GB card actually run something in the 2-billion-parameter range, or is that permanently out of reach on a laptop GPU?
| Model | Parameters | Throughput | p50 latency | Peak VRAM |
|---|---|---|---|---|
| opt-125m | 125M | 329.3 tok/s | 518 ms | 1.65 GB |
| qwen2.5-0.5b | 500M | 159.9 tok/s | 1145 ms | 2.73 GB |
| tinyllama-chat | 1.1B | 92.0 tok/s | 1076 ms | 3.66 GB |
| minicpm5-1b | 1B | 119.7 tok/s | 1947 ms | 6.55 GB |
| qwen2.5-1.5b | 1.5B | 66.7 tok/s | 1397 ms | 5.35 GB |
| qwen1.5-1.8b | 1.8B | 73.6 tok/s | 3060 ms | 5.71 GB |
The answer is yes: qwen1.5-1.8b — a real, ungated, chat-tuned 1.8B model — runs comfortably, sizing itself to 70% GPU utilization automatically and leaving headroom to spare. A year-old midrange gaming laptop can run something in the same rough parameter class as the earliest GPT-3-era chatbots, at a usable
speed, without any manual tuning.
I also tried the natural next step up — a 4-bit quantized 7B model
(qwen2.5-7b-awq) — and it exposed a subtler bug in the VRAM estimator itself. The math said it should fit: about 5.9 GB of an 8 GB budget. In practice, it failed to load, with vLLM reporting negative KV-cache memory. The estimator had assumed 4-bit quantization shrinks every parameter in the model by the same factor. It doesn't — quantization schemes like AWQ compress the matrix multiplications inside each transformer layer, but leave the embedding table and (for models with a separate output head) the un-tied output projection at full precision. For a model with a 150,000-word vocabulary, that's roughly a billion parameters — about 2 GB — that the estimator was quietly assuming were 4-bit when they were actually full-size. Splitting those layers out and pricing them separately fixed the estimate, and the same model that failed to load before now loads and serves normally.
A related, smaller mystery turned up with minicpm5-1b: despite being the
smallest model on the list by parameter count, it consistently measured more VRAM in use than models twice its size — about 2.8 GB more than the generic weights-plus-KV-cache estimate predicted. I never fully root-caused why (it may be a non-standard component in its architecture that sits outside where the inference engine reports memory usage), so rather than pretend the estimate is exact, the registry now carries an explicit, logged correction for this one model — a stopgap with an honest comment, not a real fix.
The compare view that couldn't start, and couldn't be killed
Fixing the VRAM sizing surfaced one more bug, this time in the side-by-side compare tool. Picking two real, comfortably-sized models —
qwen2.5-0.5b and qwen2.5-1.5b, together well under half the GPU — produced this on startup:
The compare tool's failure screen — note the error text cuts off mid-sentence, and (before the fix) neither Escape nor Ctrl+C could close this screen.
Startup failed
Models [qwen2.5-0.5b, qwen2.5-1.5b] cannot load sequentially on a
8 GiB GPU: Model qwen2.5-0.5b cannot fit in the remaining GPU
memory for this pool. Need gpu_memory_utilization >= 0.116 but
capped at
Two things were wrong with that screen, and neither was really about the
error itself.
The math was double-counting reserved memory. When loading models
one after another into the same process, the code sets aside a fixed
chunk of VRAM as overhead for the next model in line, on top of that
model's own already-generous footprint estimate — the two were meant to
be independent safety margins, but stacked, they reserved more memory
than either model actually needed, leaving a negative number for the
first model to work with. Recalibrated after checking, live, exactly how
much VRAM two real models actually use side by side.
The error message cut off mid-sentence. Notice the screenshot text above ends at "capped at" — nothing after it. The failure banner had a flat 200-character limit with no regard for where words ended, and the most useful part of the message (a specific next step: "Load fewer models or use a GPU with more VRAM") landed just past the cutoff, every time. Fixed by truncating at the nearest word boundary with a longer limit, so a long error still ends in a readable sentence instead of hanging mid-word.
And a UI bug: the failure screen couldn't be dismissed. Once the error
appeared, Ctrl+C — the universal "get me out of here" — did nothing. The
terminal UI framework quietly reserves plain Ctrl+C for text-copying on any screen with a modal dialog open (a loading or error overlay counts), which silently overrides the app's own "quit" shortcut for that same key unless you explicitly tell the framework "no, this one always wins." The app wasn't frozen — it just had two different components claiming the same keystroke, and the wrong one was winning.
None of these three were the kind of bug a quick manual test would have
caught by luck — they only showed up by trying an unusual-but-completely-
reasonable combination (two mid-size models, an error message long enough
to matter, a keypress against a modal). Worth remembering: the failure paths of a self-hosted tool need testing as much as the success path does.
What I actually learned
Streaming is simple. Everything around it is not. The SSE mechanism that makes AI chat feel fast is straightforward — keep a connection open, send chunks. The hard parts are model loading, memory management, routing, and error handling.
The hardware gap is real. Running a 500M parameter model on a consumer GPU is technically possible but practically constrained. At 137 tok/s with ~2 GB VRAM consumed on the 3060, I have headroom for a second model but nowhere near the throughput needed for a real service. The models powering Claude and ChatGPT are 1000× larger and run on hardware that costs more than a car, per card.
Inference is a systems problem. vLLM is fast because of continuous batching, KV cache management, and CUDA kernel optimization — not because of anything in my code. My contribution is the API layer, routing, and tooling around it. Understanding where the actual compute happens (inside vLLM, on the GPU) versus where the plumbing happens (my code, on the CPU) was clarifying.
The OpenAI API format won. Every major inference library — vLLM, Ollama, LM Studio, llama.cpp — speaks the OpenAI API format. Building to that format meant my client code worked against any of them without modification. Knowing the spec directly, rather than through a client library, is worth it.
A working demo is not the same as a correct one. The VRAM over-allocation and throughput-capping bugs described above shipped, passed their tests, and ran a working chat session for weeks before a routine benchmark caught them — because both bugs made the server slower and hungrier, not broken. Nothing threw an exception; nothing looked wrong until I measured it against a number I could check. Self-hosting inference means you're on the hook for noticing regressions that a hosted API would hide behind someone else's ops team.
Estimates need a reality check, not just a formula. The AWQ VRAM estimator was principled — quantized weights use fewer bytes per parameter, so multiply and done — and it was still wrong, because "every parameter" quietly included layers that quantization doesn't touch. The fix wasn't a smarter formula, it was going back to what the model actually contains. A benchmark that runs the real thing on real hardware catches classes of bugs a spreadsheet never will.
Try it
git clone https://github.com/coeusyk/inference-x
cd inference-x
uv sync
cp .env.example .env # add HF_TOKEN if needed for gated models
make chat # requires a CUDA GPU
You'll need Python 3.13+, uv, and a CUDA-capable GPU. WSL2 works. The first run will download model weights (~1 GB for qwen2.5-0.5b).
The source is MIT-licensed. If you're curious about any specific part of the implementation — the engine pool, the observability middleware, the streaming chat TUI — the code is straightforward Python and the architecture docs in docs/ explain the design decisions.
Built on WSL2 with an RTX 3060 Laptop GPU (6 GB) and an RTX 4060 Desktop GPU (8 GB). 430 unit tests. v0.1.2 released June 2026, VRAM/throughput fixes and 2B-class validation added July 2026.
Top comments (0)