DEV Community

Cover image for Run your own model endpoint on a rented GPU with vLLM: your own API key, an SSH tunnel, and an optional LoRA adapter
CloudGPU
CloudGPU

Posted on

Run your own model endpoint on a rented GPU with vLLM: your own API key, an SSH tunnel, and an optional LoRA adapter

#ai #python #devops #tutorial

Short answer: rent a GPU with SSH access, install vLLM in a fresh Python environment, start it so it listens only on the instance's own localhost, require an API key of your own, and reach it from your machine through an SSH tunnel. If you have a LoRA adapter for the same base model, vLLM can serve it next to the base model under its own name. When you are done, copy out what you need and destroy the instance.

Disclosure: I work on CloudGPU, and the GPU rental in this guide is ours. Each step ends with a "What to check" line, because versions, flags and defaults change.

Which kind of endpoint is this?

CloudGPU's terms (section 6) say that each running instance with a service URL gets a public HTTPS address; these URLs are "hard to guess but they are not authenticated by default", and if you need authentication you configure it inside your instance, for example an API key on your Ollama deployment. The docs say that templates that serve an API, such as Ollama and ComfyUI, get such a URL.

This guide takes a different route on purpose. You rent a raw GPU, run vLLM on it, and do not publish any URL: the server listens only on the instance's own localhost, requires your own API key, and you reach it through an SSH tunnel from your machine. That gives you an endpoint only you can use, as long as you keep the SSH access and the key to yourself.

vLLM's own security guide is clear about the limits of the key: --api-key protects only the endpoints under the /v1, /v2, /inference and /cohere prefixes, other endpoints on the same server are not covered, and it says not to rely on --api-key alone. It also notes that some underlying components may listen on all network interfaces. So treat the key as one layer, keep the tunnel as the way in, and do not send anything to the instance that you would not put on a rented machine.

Read these things first

  1. The license of your base model. An adapter sits on top of a base model, so the base model's license applies to what you serve. The walkthrough uses Mistral 7B Instruct v0.3, whose model card lists the Apache-2.0 license (read 2026-10-27). If you use a small Llama or Gemma model instead, read the Llama 3.2 Community License or the Gemma Terms of Use first, including what they say about derivative models. This guide gives no legal conclusions.
  2. What your adapter was trained on. Use only data you have the right to use; CloudGPU's acceptable use policy (section 1.8) prohibits training on datasets you do not have rights to use. If your adapter was trained on text generated through DeepSeek's API and you publish its outputs, DeepSeek's Terms of Use (Last Update: March 27, 2026) require you to label published outputs as AI-generated (section 3.1).
  3. CloudGPU's terms. Read the terms before you rent. Section 1 says compute comes from third-party data centres and supplier networks, which can change, so this guide does not claim your data stays on any particular machine.

What you need

  • A CloudGPU account with balance. GPU instances are billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance (terms). Rates are on the pricing page. Payment is by card or USDT (payments).
  • A terminal with ssh, scp and Python 3 on your own machine.
  • Optional: a LoRA adapter trained on Mistral 7B Instruct v0.3, stored in the usual Hugging Face PEFT layout (a folder with adapter_config.json and adapter_model.safetensors), for example the one from the self-critique guide in this series. Without an adapter, do everything below and skip the adapter parts; you will serve the base model only.

What to check: ssh -V prints a version, and python3 --version prints a version. If you have an adapter, its folder contains the two files above.

Step 1: make your endpoint key and look at your adapter (on your own machine)

The key is a long random string that only you and the server know. Generate one:

python3 -c "import secrets; print(secrets.token_urlsafe(32))"
Enter fullscreen mode Exit fullscreen mode

Keep it in a password manager. Make a new one for every instance; never reuse a key from another service.

If you have an adapter, open adapter_config.json in its folder and note two fields: base_model_name_or_path and r (the LoRA rank).

What to check: the command printed a long random string that you have stored somewhere other than a shell history you share. For an adapter, base_model_name_or_path is mistralai/Mistral-7B-Instruct-v0.3 (if not, the adapter belongs to another base model and this guide's command will not match it), and r is a number you can compare with the rank limit in Step 4.

Step 2: rent a raw GPU

On Deploy, choose Raw GPU and a card. The CloudGPU docs say that every template is a ready-made image with root SSH and that for anything a template does not cover you rent a raw GPU (Deploy → Raw GPU) and install what you need over SSH. The docs' GPU table lists 16 GB as the minimum VRAM for models from 0.5B to 8B parameters; vLLM also needs Linux and an NVIDIA GPU with compute capability 7.5 or higher, according to its installation page. A 7B model in 16-bit needs memory for the weights plus a cache for requests, so a card with more than the minimum is the easy fix if the server reports it cannot fit.

Connect with the SSH details of your running instance. If you have an adapter, upload its folder from your own machine (scp takes the port with a capital -P):

scp -r -P <ssh-port> ./adapter-selfloop <ssh-user>@<ssh-host>:/workspace/
Enter fullscreen mode Exit fullscreen mode

What to check: ssh -p <ssh-port> <ssh-user>@<ssh-host> connects, nvidia-smi lists a GPU, and (with an adapter) ls /workspace/adapter-selfloop shows adapter_config.json and adapter_model.safetensors.

Step 3: install vLLM in a fresh environment

The vLLM installation page recommends uv for the Python environment. uv can be installed from PyPI (see the uv installation page). On the instance:

pip install uv
cd /workspace
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
Enter fullscreen mode Exit fullscreen mode

These lines follow the vLLM page as read on 2026-10-27; if the page now shows different commands, use its commands.

What to check: with the environment active, vllm --version prints a version, and python3 -c "import torch; print(torch.cuda.is_available())" prints True. If the install stops with an error, read the error text and the troubleshooting parts of the vLLM page.

Step 4: start the server on localhost, with your key

Use your own key in place of <your-endpoint-key>. With an adapter:

cd /workspace && source .venv/bin/activate
nohup vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
  --host 127.0.0.1 --port 8000 \
  --api-key <your-endpoint-key> \
  --tokenizer-mode hf \
  --enable-lora --max-lora-rank 16 \
  --lora-modules selfloop=/workspace/adapter-selfloop \
  > /workspace/vllm.log 2>&1 &
Enter fullscreen mode Exit fullscreen mode

Without an adapter, remove the --enable-lora, --max-lora-rank and --lora-modules lines.

What the flags do, from the vLLM serve reference and LoRA page: --host 127.0.0.1 keeps the HTTP server on the instance's localhost; --api-key makes requests under /v1 carry that key; --tokenizer-mode hf uses the Hugging Face tokenizer instead of letting auto pick the mistral_common tokenizer for Mistral models; pinning it is meant to match how an adapter was trained with the Hugging Face tokenizer, so drop the flag if your adapter was trained another way; --lora-modules name=path registers the adapter under the name selfloop; --max-lora-rank must be at least the adapter's r and the docs list the allowed values (the default is 16), so if your r from Step 1 is larger than 16, use the next allowed value. The key is visible in the instance's process list while the server runs, which is one more reason to use a fresh key per instance.

Wait until the log says the server is ready before you send a request. If the process exits, read the end of the log with tail -n 50 /workspace/vllm.log. A message about memory for the cache is fixed by adding --max-model-len 8192 or by choosing a card with more memory.

What to check: on the instance, curl -s -i http://127.0.0.1:8000/v1/models | head -n 1 prints an HTTP status line that is not a success (an authentication error is expected, because you sent no key), and curl -s -H "Authorization: Bearer <your-endpoint-key>" http://127.0.0.1:8000/v1/models prints JSON that lists the base model and, with an adapter, selfloop.

Step 5: open the tunnel and call your endpoint (on your own machine)

In one terminal, open the tunnel. Nothing is printed while it runs; leave it open:

ssh -N -L 8000:127.0.0.1:8000 -p <ssh-port> <ssh-user>@<ssh-host>
Enter fullscreen mode Exit fullscreen mode

If local port 8000 is already used, pick another local port (for example -L 8001:127.0.0.1:8000) and use it in the URLs below. In a second terminal:

curl http://127.0.0.1:8000/v1/models -H "Authorization: Bearer <your-endpoint-key>"
Enter fullscreen mode Exit fullscreen mode

Now the same request through the OpenAI SDK (pip install openai on your own machine). Put this in a file named ask.py:

import os
import sys

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key=os.environ["ENDPOINT_KEY"])

BASE = "mistralai/Mistral-7B-Instruct-v0.3"
ADAPTER = "selfloop"  # delete this name from the loop below if you started without an adapter

question = sys.argv[1] if len(sys.argv) > 1 else "Explain what a LoRA adapter is in two sentences."

for name in (BASE, ADAPTER):
    reply = client.chat.completions.create(
        model=name,
        messages=[{"role": "user", "content": question}],
        temperature=0,
        max_tokens=300,
    )
    print("==", name)
    print(reply.choices[0].message.content)
    print()
Enter fullscreen mode Exit fullscreen mode

Run it with the key in an environment variable, not in the file:

export ENDPOINT_KEY="<your-endpoint-key>"
python3 ask.py "Explain what a LoRA adapter is in two sentences."
Enter fullscreen mode Exit fullscreen mode

Any tool that lets you set an OpenAI-compatible base URL and a key can use the same two values (http://127.0.0.1:8000/v1 and your key) while the tunnel is open.

What to check: the curl call lists the model names you expect; ask.py prints one answer per model name; and a call with a wrong or missing key is refused rather than answered. If the tunnel command prints "address already in use", use another local port. Reading two answers is not a test of your adapter; for that, use questions the adapter never saw and compare them yourself, as in the self-critique guide.

Step 6: stop, copy out what you need, destroy

Close the tunnel with Ctrl+C in its terminal. Before you destroy anything, copy what you want to keep. The docs have a section called "What to save before you destroy an instance"; read it in the CloudGPU docs. For example, copy the server log from your own machine:

scp -P <ssh-port> <ssh-user>@<ssh-host>:/workspace/vllm.log ./
Enter fullscreen mode Exit fullscreen mode

Then destroy the instance from your Dashboard (the docs: "You can destroy an instance at any time"). Minutes spent copying are billed like any other minute, and billing stops once the instance is destroyed. The endpoint key was meant for this instance only; do not reuse it.

What to check: the instance no longer appears as running in your Dashboard, and the curl call from Step 5 now fails to connect.

Privacy and access, in plain terms

  • This is your own endpoint, with your own API key, reached through your own SSH tunnel. It is not a public URL.
  • Per CloudGPU's terms (section 5), you keep ownership of any data, models, code or output you create on or upload to an instance, and CloudGPU says it does not access, inspect, copy or back up the contents of your instance except in three listed cases: a credible report of a violation of section 3, valid legal process, or to protect the safety of users or third parties.
  • Per the same terms, destroying an instance permanently deletes its local storage, and you are responsible for backing up what you want to keep.
  • Whatever you send to the endpoint is processed on that rented instance. Do not send secrets or personal data you would not put on a rented machine.
  • If you call a model through CloudGPU's AI API instead of your own instance, the request is forwarded to the model provider and its terms apply; see the privacy policy.

Sources

All read on 2026-10-27 (UTC+8).

Top comments (0)