A model on your own graphics card can answer your team or paying customers through an API, with a key for each of them, a price per token and a stop when the credit runs out, while the card itself never gets an address on the internet. A handful of open-source tools and a payment provider make that possible, and each of them has a setting that is easy to get wrong.
This post goes through them in the order to build them, as they run behind Blixild AI, our self-hosted LLM API for open models on RTX 3090s and RTX 5090s in our own servers in Poland. At each stop of a request I say what the tool does, why I chose it and what to take care of.
Along the way the post shows where a card had a quarter less room for conversations than our own calculator estimated, and why a model server's own key check does not protect it. It explains why a long prompt can stall where short requests pass and why every payment arrives twice. It ends with what a setup for one person or one company can leave out, and with the six steps of the build.
I wrote it for engineers who want to put an open model behind an API, for their own company or for customers. An earlier post explains why we serve Arete 27B this way.
What a model API needs beyond the model
Serving software loads a model onto a graphics card (GPU) and answers requests. A service needs more than that. Somebody has to know who is asking, count the tokens, put a price on them and stop when the credit runs out. Customers need a way to sign in and to pay, and I need to know that all of it works.
I followed one rule: a ready-made tool for every job, and my own code only where the tools meet. The figure shows the result along the path of one request. A small server in Helsinki is the only machine the internet can reach, and the cards sit behind it. The sections below take the stops in the order I built them, from the cards outwards: the card and its model server, the gate in front of it, the gateway that charges for tokens, the portal where customers sign in, and the checks that hold them together.

The figure shows the path of one request and the tool at each stop. It is an illustration, not a measurement.
The calculator sizes the card and the endpoint checks it
The first stop is the card, and the first question is whether the model fits on it and how fast it will answer. Our LLM GPU Token Speed Calculator estimates both before anything is loaded. It takes the card, the model, the size of its weights and the number of requests at once. It returns the memory left for conversations, the time to the first token and the tokens per second each user gets.
For this post I entered the RTX 3090's setup with Arete 27B and wrote the results down before I compared them with the measurements. The table shows both.
| Calculator, practical range | Endpoint, measured | |
|---|---|---|
| Room for conversations | 147,000 to 174,000 tokens | 109,000 tokens |
| Speed of one request | 43.0 to 47.1 tokens/s | 46.9 tokens/s |
| Speed of each of two long requests at once | 39.3 to 43.6 tokens/s | 40.8 tokens/s |
| First token after a 1,000-token prompt | 0.71 to 0.77 s | 0.79 s |
The speeds fell inside the calculator's range, and the first token came 2% later than the range's upper end. The room for conversations did not match: the endpoint had about a quarter less than the lowest estimate. The serving software's start-up log shows where the difference lies. The software reserved about three times the working memory that the calculator assumed for a text-only model.
The calculator's practical estimate now counts that larger working memory. For the same setup it gives about 96,000 to 105,000 tokens, just below the endpoint's 109,000.

The figure is a screenshot of the calculator's Memory step for Arete 27B in 4 bits on an RTX 3090, with an 8-bit conversation memory, text only and a 98.9% memory share. It shows the estimate as the calculator gives it since the comparison above.
An estimate chooses the card. It does not set the service's limits. Those come from a run with every slot busy and the longest prompts the model accepts, because that run shows the memory left at the worst moment. Neither the estimate nor these measurements say how well the model answers.
vLLM serves one model on each card
With the card chosen, the next choice is the software that loads the model onto it and answers requests. I use vLLM, because it answers several requests at once from one card and speaks the same format as OpenAI's API. One vLLM process serves one model, so a card holds one model: Arete 27B runs on an RTX 3090 and Bielik 11B v3.1 on an RTX 5090.
Each model has to fit its card. Bielik fits the 5090's 32 GB with its weights stored in 8 bits each. Arete 27B at full precision needs more than twice the 3090's 24 GB, so that card runs a 4-bit build. A 4-bit build can cost some accuracy, and I have not yet measured how much for Arete.
What the weights leave free holds the conversations in progress. On the 3090, storing that conversation memory in 8 bits and leaving out the model's image input about doubled the room. vLLM can also draft several tokens per step, which on that card was about one and a half times faster for a single request and cost 60% of that room. I chose the room, because long documents and agents need it.
Two settings deserve care:
- Set the conversation memory yourself. By default vLLM measures its working memory at each start and gives conversations the rest. The measurement can differ between starts, and a start that gives conversations too much runs out of memory. A fixed size, taken from a run that held with every slot busy, starts the same way every time.
- Switch tool calling on and test it. It works only when vLLM starts with the parser for the model's format, and the two models here need different ones. Without it, plain chat works and every request that offers tools fails.
With these settings the model answers anyone who can reach its port. The next stop decides who can.
The cards stay off the internet
vLLM can check an API key itself, but that option guards only some of its addresses. Others on the same port answer without a key, among them one that generates text. vLLM's own documentation says not to rely on the option alone and to put a reverse proxy in front.
I put nginx there as a key gate, the same on every GPU server. It passes requests under /v1 that carry a known key and refuses everything else. vLLM listens only on the machine itself, so the gate is the only way in.
Only one machine needs to reach the gate: the gateway server in Helsinki. It connects over Tailscale, a private encrypted network between our machines. The network's access policy lets the gateway reach that one port on each GPU server and nothing else, and the policy has tests that fail if a change opens more. Check the plan before a service depends on it: Tailscale's free plan is for non-commercial use.
The gate only decides who gets in. It holds no budgets and counts no tokens, and that is the gateway's job.
LiteLLM turns tokens into charges
The gateway is LiteLLM, an open-source proxy for model APIs. Each model is one entry in its config, with the address and the key of the server that runs it, so a request that names Bielik goes to a 5090 and one that names Arete to a 3090.
LiteLLM issues the keys customers use, counts each request's tokens, charges them against the customer's budget and refuses requests when the budget is spent, and it limits how many requests a key runs at once. Caddy, a web server, stands in front of it, obtains the TLS certificates and forwards only the /v1 addresses. Both run in containers on the gateway server, next to a PostgreSQL database. The gateway and the portal have a server to themselves.
Three things need care here:
- Price every custom model before you give credit. LiteLLM knows the prices of public models. A custom model has no price until both of its rates are in the config, and a request without a price draws nothing from the credit. My check sends a real request to every model and compares the charge with its published price.
- Publish /v1 only. LiteLLM's admin API manages keys and budgets, so it stays on the private side, where only the portal calls it.
- Test the longest prompt through every hop. A long prompt is a large upload, and it can stall in a layer that short requests never stress. With rootless Docker that layer was pasta, the user-mode network between a container and the host. Timing each hop separately found it, and switching the network back end to slirp4netns fixed it.
LiteLLM does not load or swap models, and it does not sign anyone in or take a payment. Something has to create each customer and raise the budget when they pay, and that is the portal.
The portal handles sign-in, credit and payments
The portal at ai.blixild.com is the part I wrote: a Next.js application with Better Auth for sign-in. Customers sign in with Google or GitHub only, so the portal stores no passwords and every email address arrives verified. On the first visit it creates the customer in LiteLLM with a key and $1 of credit.
Credit is a ledger of grants in the portal's own database, and each grant sets the customer's budget in LiteLLM to the sum of the ledger. Customers add credit by paying through Stripe, which acts as merchant of record: it sells the credit in its own name, works out and pays the buyer's VAT or sales tax and sends the receipt.
Three things need care here too:
- Expect every payment twice. A payment is reported when the customer returns from the payment page and again through Stripe's webhook, which Stripe repeats when it gets no answer. Each grant therefore carries the payment's reference and counts once.
- Say whether tax is included. Unless the call states it, the Stripe account's default decides whether tax is inside the price or added to it, and a test account and a live one can differ.
- Audit the dependencies before launch. A project that starts from a template or an example inherits the versions of the day that was written, and some of them will have known security holes by now.
The portal sells prepaid credit only. It has no subscriptions and issues no invoices of its own.
One check proves the whole path
That completes the path, from the portal's sign-in to the card. Each tool on it passes its own tests, and the service can still fail between them. So one check runs the whole path against the real models, through the public address. It signs up a throwaway customer and makes a paid request. It verifies that the charge matches the published price, that requests stop at $0 and that a deleted key no longer works. Last, it confirms that the logs do not hold the prompt. The privacy promise rests on settings, and this check proves them after every change.
In operation, the gateway keeps sending a short real request to each model, along the path customers use. A model that stops answering triggers an email, sent from the gateway's side, so the alert does not depend on the machine it reports on.
One detail deserves a deliberate test. A restart rule that waits for an error code misses a model server that shuts down cleanly after its engine has died. I found this by stopping the engine on purpose, and the containers now restart whenever they stop.
None of these checks measures how good the answers are. That takes tests of the model itself, on the work it will be used for.
A smaller setup needs fewer stops
Blixild AI sells access to people I have never met, so it needs every stop above. A setup with fewer strangers needs fewer stops, and the same order tells which ones.
- One person, one machine. The card, its sizing and vLLM are the whole stack while the model is used on the machine it runs on. Once a laptop or a phone should reach it, add the key gate and the private network, and nothing else. Blixild AI ran this way before it had customers.
- A company serving its own staff and applications. Keep the gate and the private network, and add the gateway. Its keys then stand for a team or an application instead of a customer. Its budgets and its limit on requests at once keep one script from taking the whole card, and its usage records show who needs how much, which is what a decision on a second card rests on. Leave out the portal and the payments, because the staff already have accounts and nobody pays by card.
Six steps from a card to a service
The path of this post is also the path of a build, from the card outwards, with each step proved before the next. How far you follow it depends on who will call the model. One person needs steps 1 to 3, a company serving its own staff needs steps 1 to 4, and a public service needs all of them. Step 6 belongs to every size.
- Size the card. Estimate whether the model fits and how fast it answers, then measure it with every slot busy.
- Serve the model. Run one model per card, set its conversation memory and test tool calling.
- Protect the model server. Put a key gate in front of it and reach it over a private network.
- Meter the use. Add a gateway that issues keys, prices tokens and enforces budgets, and decide what its logs may hold. Prove it with one test customer, down to the refusal at $0.
- Publish the service. Give it a public address that forwards only the API, then add sign-in, credit and payments.
- Check the whole path. Run one check from end to end after every change, and keep a real request going to each model.
Each further card repeats steps 1 to 3 and adds one entry to the gateway's config.
To see the result from the customer's side, sign in at ai.blixild.com with Google or GitHub. The key comes with $1 of credit and no card is needed, and any OpenAI client library works with api.blixild.com/v1.
If you want this stack on your own premises or in your own cloud account, with the model of your choice, our AI workload hosting sets it up and runs it for you.
Sources
- vLLM (2026). Security, read on 19 November 2026. https://docs.vllm.ai/en/latest/usage/security.html
- LiteLLM (2026). Custom LLM Pricing. https://docs.litellm.ai/docs/proxy/custom_pricing
- LiteLLM (2026). Spend Tracking. https://docs.litellm.ai/docs/proxy/cost_tracking
- Tailscale (2026). Tailscale pricing, read on 19 November 2026. https://tailscale.com/pricing
- Stripe (2026). How Managed Payments works. https://docs.stripe.com/payments/managed-payments/how-it-works
Top comments (0)