Companies keep asking me for an AI model on their own hardware. Usually the honest answer is "not yet": for short questions, a cloud API stays cheaper than a GPU workstation for about a year and a half. The picture changes when the model has to read a lot of internal documents on every request, or when the data simply can't leave the building. Then a single-GPU machine can pay for itself in roughly four months.
Below is how I size the hardware, what it costs (I'm based in Russia, so prices are from the Russian market in October 2026), and how to do the payback math for your own workload.
Why a company wants its own server at all
Three reasons come up again and again:
- Data must stay inside. Contracts, medical records, customer emails, source code. Even when a cloud provider promises zero retention, legal or security teams often say no. A box in the office ends the discussion.
- API bills scale with volume. Short questions cost almost nothing. When every request carries ten pages of documents, the bill multiplies. Your own hardware costs the same at any load.
- Predictability. A hosted model can be updated, repriced or rate-limited. Your own model behaves the same until you change it.
If none of these apply, start with a cloud API. It's faster and cheaper, and you can move to your own server once you know your real request volume.
How much memory a model needs
The key spec is GPU memory, not CPU power. The model has to fit entirely in VRAM, otherwise it runs many times slower or doesn't start.
Local models are almost always quantized to 4 bits per parameter, with barely noticeable quality loss. That gives a simple rule of thumb: about 0.5 GB per billion parameters, plus headroom for context.
- 7–8B: about 5 GB
- 14B: 8–9 GB
- 32B: 18–20 GB
- 70B: about 40 GB
The headroom matters. Long documents and several concurrent users add gigabytes on top of the weights.
My measurement: what running out of memory looks like
I ran the same task on my own laptop, a MacBook with Apple M5 and 16 GB of memory, using Ollama: write a 150-word business email about a three-day delivery delay.
| Model | Generation | Reading a 45-page document | Whole email |
|---|---|---|---|
| Qwen 3.5 4B (~4 GB) | 35 tokens/s | 312 tokens/s, 67 s | 9 s |
| Gemma 4 12B (~8 GB) | 20 tokens/s | 72 tokens/s, almost 5 min | 23 s |
The small model writes faster than a person reads. The model three times larger generates almost half as fast, and reads a long document four times slower: on 16 GB it's competing with the OS and open apps. For a short email you barely notice. Hand it a 45-page document and an employee waits almost five minutes. (Updated Oct 28: the first version quoted prompt speed measured on a 36-token request, which isn't meaningful; the numbers above are from a real document.)
The lesson for a server: buy 1.5–2x more memory than the model weighs. A 20 GB model on a 24 GB card is a squeeze; on 32 GB it's comfortable.
What to buy
| Task | Models | Hardware | Price (Russia, 2026) |
|---|---|---|---|
| Writing assistant, summaries, pilot | 7–14B | PC with RTX 5090 (32 GB) or a Mac with 32–64 GB | card 393–490k RUB, ready PC from 639k RUB |
| Department knowledge base, document search | up to 32B | Same RTX 5090 PC: 18–20 GB for the model, the rest for context | from 639k RUB |
| 70B-class models or gpt-oss-120b | 70–120B | Workstation with RTX PRO 6000 Blackwell (96 GB) | card 1.08–1.8M RUB |
| The largest open models, 1–2 users | hundreds of B | Mac Studio M3 Ultra, 512 GB unified memory | 2.79–2.99M RUB |
| Rent instead of buying | any | RTX 5090 / H100 80 GB by the hour | 66 RUB/h / 353–359 RUB/h |
| Cloud API | YandexGPT, gpt-oss, Qwen, DeepSeek | no hardware, pay per token | 0.1–0.8 RUB per 1,000 tokens |
Sources: Digital Razor for RTX 5090 and PCs, the RTX PRO 6000 launch at DNS at 1,079,999 RUB in February 2026, Megabait for Mac Studio, zeroquant.ru for GPU rental, Yandex AI Studio pricing.
Two notes. NVIDIA cards have faster memory, so they answer quicker and handle several users at once better. A Mac gives you more memory per ruble and is nearly silent, but serves fewer people concurrently. For a team of 10–30 I'd pick a GPU; for one analyst or lawyer with huge documents, a Mac. And the RTX 5090 draws up to 575 W, the RTX PRO 6000 up to 600 W: this is a hot, loud machine that belongs in a server room, not under a meeting-room desk.
When your own server beats the cloud
Example: a team of 20, each making 30 requests a day, 22 working days a month.
| Scenario | Tokens per month | YandexGPT Pro 5.1 | Payback of a 639k RUB PC |
|---|---|---|---|
| Short questions (~3,000 tokens per request) | ~40M | ~32k RUB/month | ~20 months |
| Answers over a knowledge base (~15,000 tokens: question + document chunks) | ~198M | ~158k RUB/month | ~4 months |
YandexGPT Pro 5.1 costs 0.8 RUB per 1,000 tokens including VAT. Open models in the same cloud are cheaper (gpt-oss-120b at 0.3 RUB, gpt-oss-20b at 0.1 RUB), which pushes payback out another 2.5–8x.
Renting is the middle ground. An RTX 5090 at 66 RUB an hour, kept only during working hours (about 200 hours a month), costs around 13k RUB a month; around the clock, about 48k. An H100 around the clock is about 255k RUB a month, which only makes sense under heavy load.
Not included: electricity, setup and maintenance, and model quality. A 32B model that fits on an RTX 5090 trails YandexGPT Pro on some tasks and matches it on others. Test that on your own data before buying anything.
How I'd roll it out
- Pilot in the cloud. Two or three weeks of real tasks through an API, using open models of the same size you'd later host. You learn your volume and whether the model copes.
- Do the math with real numbers.
- Buy for the model, not the other way round. Pick the model that solves the task, then hardware with 1.5–2x memory headroom.
- Integrate. A model without access to documents, CRM and email can't do much. Most of the work is here, not in installing the server.
My default for a company starting out: don't buy a server right away. A month on a cloud API shows the real volume. If data can't leave the office or the model works over a document base, start with one RTX 5090 machine: it handles models up to 32B for a team of a few dozen people. The RTX PRO 6000 and a 512 GB Mac Studio are for tasks where a smaller model has proven not to be enough.
Is your team running models on its own hardware yet? What tipped the decision, cost or security?
Originally published at klukyanov.ru.
Shorter weekly write-ups (in Russian) — on Telegram.
Top comments (0)