If you are the engineer asked to put one self-hosted model behind a whole company, the model choice is the short meeting. The long tickets are about keys, queues, alerts and the invoice. We recorded a demo of a private ChatGPT shared by fifty teams with exactly that setup: fifty keys, one endpoint that speaks the OpenAI API format, ten GPU servers, and a cost screen that splits an hourly bill per key.
One disclosure first, because it shapes everything below: the ten "H100" machines (an H100 is a high-end Nvidia GPU) were test-mode processes on one ordinary host, returning filler text instead of running a model. The panel, keys, queue and accounting were the real product. So this run says a lot about the management side and nothing about how fast a real model answers.
Scope every key to one endpoint
The demo issues a key per team, often per use case: roadmap, competitive intel, security policy Q&A, board reporting. Fifty in total, and each one is allowed on a single endpoint, nexora-private-gpt/enterprise-chat.
Two reasons for that. A leaked key can only reach the company chat, not anything else the project runs. And the key is the unit of accounting, so a key per use case is what lets you tell finance that board reporting, not "finance", is the heavy user. Offboarding a team is unticking its key. How keys, scopes and environment variables work is in the Tokens & API keys guide.
Because the endpoint accepts the same request format as OpenAI, existing clients only change a base URL and a key.
Let workers pull, do not route
In the panel, the served model is a workload and each machine serving it is a worker. Ten workers serve one workload, and each takes up to 8 requests at the same time (its concurrency), so the fleet has 80 slots. All fifty keys feed one queue and whichever worker has a free slot claims the next job. You decide which machines serve the model, and the queue decides which one answers a given request.
That pays off when you scale: you pick a registered machine or a new cloud GPU, set its concurrency, and no team changes its key or URL. Nobody clicked "Add worker" while the traffic was running, and the write-up of the fifty-team demo lists that, and a worker dying mid-load, among the scenes it has not filmed yet.
Read the backlog trend, not the backlog
The backlog is the number of requests waiting for a free slot, and throughput is how many get finished per minute. A backlog of 20 says little alone. One that keeps climbing is the signal to add a machine. The drawer turns that into a label. It stays on "Healthy flow" as long as throughput is above zero and the backlog, compared over the last complete minutes (five at most), has grown by no more than 25% or 2 jobs, whichever is larger. A falling backlog with throughput going up shows "Recovering". A rising one shows "At risk", and "Critical" when throughput falls at the same time.
A trap on the same screen: the jobs-per-minute counter in the header only counts completions your open tab has seen in the past minute, so it restarts at zero on every page load. In the take it climbed from 0 to about 3,700. The real load is the drawer's five-minute average, 4,252 finished per minute, from a script pushing about 70 requests a second.
The bill is two lines of arithmetic
Nothing is priced per word. The panel prices reserved worker time and only uses tokens (the chunks of text a model reads and writes, roughly three quarters of a word each) to split the total:
project_cost = reserved_hours * hourly_price # 134.53 h * $2.4306/h = $326.97
key_cost = key_tokens / project_tokens * project_cost
# 17.37M / 93.07M = 18.7% -> $61.03 for finance-board-reporting
The hourly price is $1,750 a month over 720 hours. Ten workers cost $24.31 an hour, about $17,500 a month if they never stop. A worker accrues hours for the whole time it stays assigned to the project, busy or not. The total moves by about a cent every 1.5 seconds, so screenshots taken seconds apart differ by a few cents. Do not write a test that expects them to match.
Do not quote the per-token number
The panel shows $3.51 per million tokens. Do not put that next to a public API's price. Each worker was reserved for about 13.5 hours, and at its steady rate the test traffic would have filled only around 72 minutes of that, so the figure mostly measures machines waiting. When you pay by the hour, that unit cost drops the busier you keep the fleet, and on real GPUs it comes down to how many tokens per second each machine can produce.
What you would still have to measure on real GPUs:
- tokens per second per machine for your model and your answer lengths
- how close to the 80 slots you can run before latency hurts (the script never sent more than 32 at once)
- error behavior, since the 0 errors and the 250 ms p95 latency (95% of requests answered at least that fast) come from the test mode
- each team's real share, since the script sent most traffic from a few keys and tokens were counted on filler text
Like a soundcheck with the PA on and no band, it tells you the cables and the desk work. It does not tell you how loud the room gets.
The full article on one private assistant for fifty teams has the video, every panel screen and the full cost table per key.
Top comments (0)