DEV Community

Cover image for Hetzner Inference: First Look
Jonas Scholz
Jonas Scholz Subscriber

Posted on • Originally published at sliplane.io

Hetzner Inference: First Look

#ai #cloud #infrastructure #llm

Hetzner is experimenting with LLM inference.

That is not a sentence I expected to write, but I think it is pretty interesting :)

Before anyone moves their production AI workloads to Hetzner: this is very much an experiment. There is no billing, no SLA, no production guarantee, and currently only one model. Hetzner says it wants to learn whether people actually want this, how the system scales, which features matter, and what kind of load it can handle.

So this is not a finished product launch. It is Hetzner putting something early in front of users and seeing what happens. I really like that approach.

What Is Hetzner Inference?

Hetzner Inference is an OpenAI-compatible API running on Hetzner's own infrastructure. You create an API token in the Experiments dashboard, point an OpenAI client at Hetzner's base URL, and use it like most other inference APIs.

Right now, the only available model is Qwen/Qwen3.6-35B-A3B-FP8. It is a 35-billion-parameter Mixture-of-Experts model with 3 billion active parameters. It accepts text and images, has a 262K context window, and uses FP8-quantized weights.

That is a perfectly reasonable model for an experiment. It is small enough to serve without a ridiculous GPU cluster, but still useful enough to test the API with real workloads.

Dashboard

Hetzner also published a short tutorial for connecting OpenCode to the API, if you want to try it without writing any code.

I Tried It

Because the API is OpenAI-compatible, there is almost nothing special about the integration:

pip install openai
Enter fullscreen mode Exit fullscreen mode 
from openai import OpenAI

client = OpenAI(
    base_url="https://inference.hetzner.com/api/v1",
    api_key="YOUR_TOKEN",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B-FP8",
    messages=[
        {"role": "user", "content": "Explain why the sky is blue in one sentence."}
    ],
    extra_body={
        "chat_template_kwargs": {
            "enable_thinking": False,
        }
    },
)

print(response.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

The enable_thinking option is worth mentioning. Without it, the model can spend a surprising amount of the completion budget reasoning before it returns a visible answer. The option worked in my tests, but it is not documented by Hetzner, so I would not build anything important around that exact request shape yet.

I ran a few small tests on July 23, 2026. I do not want to turn this post into a giant benchmark report, because the product is experimental and a benchmark against it will probably age badly. But the rough numbers were:

  • 153 ms median time to first token across seven short requests on an already open connection
  • 224 output tokens per second across five longer generations capped at 512 tokens

That is fast! It is also just one test from one client at one point in time. It is not an SLA, and it says almost nothing about what happens when many people use the service at once.

The model itself was roughly what I expected. It followed most formatting and retrieval instructions, handled an image correctly, and failed two very simple arithmetic questions. So: a small, slightly shitty LLM :D

The Product Is More Interesting Than the Model

The Qwen endpoint is fun, but I do not think the current model is the interesting part.

The interesting part is why Hetzner is testing inference in the first place.

Everything from here on is purely my speculation. I have no insider information, and nobody at Hetzner told me what they are planning. I am just looking at the product and trying to connect a few dots.

Open-weight inference is a commodity market. Everyone can download the same weights, run more or less the same serving software, and expose an OpenAI-compatible API. Switching providers is also easy, especially with products like OpenRouter or LiteLLM for those you self-host.

That makes it hard to build huge margins unless you have some kind of advantage. Usually that means:

  • you can buy and operate GPU hardware very cheaply;
  • you are exceptionally good at keeping that hardware busy;
  • or you already own GPUs that would otherwise sit around waiting for customers.

Hetzner is very good at buying hardware, putting it into its own data centers, and operating it with a brutally efficient cost structure. That is basically the whole company. If anyone can turn inference into another low-margin infrastructure product, Hetzner is at least a believable candidate.

There is also a nice utilization story here. A rented bare-metal GPU belongs to one customer, whether that customer uses it or not. An inference API can share GPU capacity across many users and keep the hardware busy. If Hetzner has spare GPU capacity — or plans to build a much larger GPU fleet — an inference product could help turn that capacity into revenue.

Again, I have no idea whether this is actually what they are doing. It would just make economic sense to me.

The Big Question Is Hardware

This is where I am not yet convinced.

Hetzner's current public dedicated GPU server lineup uses two GPU types:

  • NVIDIA RTX 4000 SFF Ada Generation with 20 GB of VRAM
  • NVIDIA RTX PRO 6000 Blackwell Max-Q with 96 GB of VRAM

Those are capable GPUs, and the 96 GB RTX PRO 6000 is a pretty nice inference machine for small and medium-sized models. The FP8 files for Hetzner's current Qwen model are around 38 GB, with actual VRAM use landing somewhere above that depending on context length, cache size, and serving setup.

But these are workstation GPUs, not the dense multi-GPU systems you need for the really large open models.

Take GLM-5 as an extreme example. It has 754 billion parameters, and the official serving recipe splits it across eight GPUs. Even with aggressive quantization, you are talking about hundreds of gigabytes of VRAM. Realistically, that is B200/B300-class hardware, or something similar, with very fast links between multiple GPUs.

Hetzner does not currently offer that kind of hardware in its public bare-metal lineup.

Of course, that does not tell us what sits behind the experimental API. Hetzner may use completely different internal hardware, and a public inference product does not have to mirror its dedicated-server catalogue.

Still, this is the part I am watching.

If Hetzner keeps serving one or two smaller models, I do not really see it becoming an important inference provider. That would be a cool experiment, but not much more.

If this experiment is the first step toward larger GPU clusters, a proper model catalogue, and B200/B300-class hardware, then it gets much more interesting. Hetzner already has the data centers, network, hardware experience, European positioning, and reputation for aggressive pricing. That combination could make it a serious competitor.

For now, the API is fast, free, and fun to try. The next hardware announcement will tell us much more than another small model would.

Cheers,

Jonas

Top comments (0)