DEV Community: CloudGPU

Use DeepSeek V4.1 Flash and Kimi in opencode via a custom OpenAI-compatible provider

CloudGPU — Sun, 25 Oct 2026 22:22:44 +0000

opencode ships with a long list of built-in providers, but you are not limited to them. Its docs describe a custom provider path for any OpenAI-compatible endpoint: you declare the provider in opencode.json, point it at a base URL, list the model IDs, and they show up in the model picker.

This post walks through that setup with two models, DeepSeek V4.1 Flash (deepseek-v4.1-flash) and Kimi K3 (kimi-k3), plus two extras in the same config: kimi-k2.6 and glm-5.3-flash. The endpoint in the examples is CloudGPU's OpenAI-compatible API at https://cloudgpu.app/v1, but the same steps apply to any provider that speaks /v1/chat/completions.

What's tested and what isn't: on 2026-10-26 we ran this setup end to end in opencode 1.19.4. The opencode.json below loaded as written with no schema errors, opencode models cloudgpu listed all four models, and opencode run -m ... returned a reply from both deepseek-v4.1-flash and kimi-k3. All four model IDs were also listed by GET /v1/models, and a direct chat request to each one replied normally. Not yet verified: the interactive /models screen in the TUI, the /connect key flow, and how limit.output maps onto requests. Those parts follow opencode's official docs, so read "according to the docs" literally.

What you need

opencode installed (see the opencode docs for install options).
An API key for your OpenAI-compatible endpoint. For CloudGPU, keys are created on the API page and look like cgw-sk-....
The exact model IDs your endpoint serves. Here: deepseek-v4.1-flash, kimi-k3, kimi-k2.6, glm-5.3-flash.

Step 1: Export your key

opencode's config supports {env:VARIABLE_NAME} substitution (Config → Env vars), so the key never has to live in the JSON file:

export CGW_API_KEY="cgw-sk-..."   # add to ~/.bashrc / ~/.zshrc to keep it across shells

Step 2: Add the provider to `opencode.json`

Create (or edit) opencode.json in your project root. Per the docs, project config takes precedence over the global file at ~/.config/opencode/opencode.json, and config files are merged, so you can also put the provider block in the global file if you want it in every project.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "cloudgpu": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "CloudGPU",
      "options": {
        "baseURL": "https://cloudgpu.app/v1",
        "apiKey": "{env:CGW_API_KEY}"
      },
      "models": {
        "deepseek-v4.1-flash": {
          "name": "DeepSeek V4.1 Flash"
        },
        "kimi-k3": {
          "name": "Kimi K3"
        },
        "kimi-k2.6": {
          "name": "Kimi K2.6"
        },
        "glm-5.3-flash": {
          "name": "GLM-5.3 Flash"
        }
      }
    }
  }
}

What each field does, per opencode's Providers docs:

Field	Meaning
`cloudgpu`	The provider ID. You pick it; it becomes the first half of the model reference (`cloudgpu/<model>`).
`npm`	The AI SDK package. `@ai-sdk/openai-compatible` is the one the docs name for OpenAI-compatible providers that use `/v1/chat/completions`.
`name`	Display name in the UI.
`options.baseURL`	The API endpoint. Note the `/v1` at the end.
`options.apiKey`	Optional; set via `{env:...}` syntax. If you would rather store the key with `/connect` (see below), you can drop this line.
`models`	The model IDs to expose. Each key must match the ID the endpoint expects; `name` is only the label.

Only need the two headline models? Delete the kimi-k2.6 and glm-5.3-flash entries; nothing else changes.

Alternative: store the key with `/connect`

The docs also describe a credential flow: run /connect inside opencode, scroll to Other, enter the provider ID (cloudgpu, matching the key in opencode.json), then paste your key. That stores the credential only; you still need the provider block above. Use one approach or the other.

Step 3: Pick the model

Start opencode in the project and run:

/models

According to the docs, your custom provider and its models appear in this selection list. Choose DeepSeek V4.1 Flash or Kimi K3. (In our test we checked the list from the command line, not this screen.)

To check from the command line that opencode picked up the provider, run:

opencode models cloudgpu

In our test with opencode 1.19.4, this listed:

cloudgpu/deepseek-v4.1-flash
cloudgpu/glm-5.3-flash
cloudgpu/kimi-k2.6
cloudgpu/kimi-k3

To choose a model from the command line instead, use --model / -m in provider/model form. In our test, -m cloudgpu/deepseek-v4.1-flash (the dot in the ID is fine) and -m cloudgpu/kimi-k3 both returned replies:

opencode -m cloudgpu/deepseek-v4.1-flash
opencode run -m cloudgpu/kimi-k3 "Explain what this repo's main entry point does"

You can also set a default in opencode.json with the top-level "model" key, using the same provider_id/model_id form, for example "model": "cloudgpu/deepseek-v4.1-flash".

Which one for what?

I'm not putting benchmark numbers here; try both on your own repo. A practical split:

deepseek-v4.1-flash: a sensible default for everyday edits, explanations, and small refactors.
kimi-k3: a reasoning model; switch to it with /models for multi-step changes or a second opinion on the same task. It spends output tokens on reasoning before it answers (see troubleshooting #4).
kimi-k2.6 and glm-5.3-flash: already in the config, so comparing is one /models switch away.

Troubleshooting

This section is based on a real question we answered: someone added a DeepSeek model as a custom provider in a coding agent, entered the endpoint, key, and two models, and the model never showed up in the picker. The model ID they had entered was a short alias, not the exact ID the endpoint lists. We can't say for sure that was the cause in their case, but a short alias like that is a common cause, and the same checklist applies to opencode.

1. The model doesn't appear in `/models` (or requests fail with "model not found")

The keys under models must match, character for character, the IDs your endpoint returns from GET /v1/models. Check them directly:

curl https://cloudgpu.app/v1/models \
  -H "Authorization: Bearer $CGW_API_KEY"

Copy the id values from the response into opencode.json. Common mismatches: a short alias (deepseek-flash), a missing version segment (deepseek-v4-flash vs deepseek-v4.1-flash), or different capitalization.

Also confirm the provider ID matches everywhere: the key under provider, the ID you typed in /connect (if you used it), and the prefix in -m cloudgpu/.... opencode's own troubleshooting list calls out the /connect ID mismatch, and opencode auth list shows which credentials are stored.

2. The base URL is missing `/v1`

@ai-sdk/openai-compatible appends paths like /chat/completions to baseURL. Use:

https://cloudgpu.app/v1      ✅
https://cloudgpu.app         ❌
https://cloudgpu.app/v1/chat/completions   ❌ (too specific)

3. The environment variable isn't exported

The docs note that if a variable referenced with {env:...} is not set, it is replaced with an empty string, so no key is sent and the request is rejected. In our test (opencode 1.19.4, CGW_API_KEY empty) the error was:

Error: Provide an Authorization: Bearer <api_key> header.

Check the variable in the same shell you launch opencode from:

echo "${CGW_API_KEY:+set}"   # prints "set" if the variable is present

If you set it in ~/.zshrc or ~/.bashrc, open a new terminal (or source the file) before starting opencode. Desktop launchers and IDE terminals may not read your shell profile.

4. Empty replies from reasoning models

kimi-k3, kimi-k2.6, and glm-5.3-flash reason before they answer, and the reasoning counts against the output limit. In our 2026-10-26 test against the API directly:

with max_tokens: 20, all three returned empty or null content, because reasoning used up the tokens first;
with max_tokens: 800, all three replied normally;
kimi-k3 used 1027 completion tokens to answer "ok".

If the reply is empty, raise max_tokens or leave it unset.

In opencode, the per-model knob the docs describe for custom providers is the optional limit block (limit.context, limit.output). If you set limit.output low, or a tool or agent config passes a small output cap, you may hit the same empty-reply behavior. If you add limit, use real values from your provider's model docs:

"kimi-k3": {
  "name": "Kimi K3",
  "limit": { "context": <context tokens>, "output": <max output tokens> }
}

(Replace the placeholders with numbers; the snippet is not valid JSON as written.) How opencode maps limit.output onto the request is per its docs; I haven't verified it against these models inside opencode.

Billing

CloudGPU's API runs on a prepaid balance (card, Alipay, Google Pay, Apple Pay, or USDT). The model list and API key setup are on the CloudGPU API page.

Recap

export CGW_API_KEY=...
Add a provider block with "npm": "@ai-sdk/openai-compatible", "baseURL": "https://cloudgpu.app/v1", and "apiKey": "{env:CGW_API_KEY}".
List exact model IDs under models, verified against GET /v1/models.
Run opencode models cloudgpu to confirm the models are listed, then pick cloudgpu/deepseek-v4.1-flash or cloudgpu/kimi-k3 with /models or -m.
Empty reply from a reasoning model? Raise max_tokens or leave it unset.

Docs used: opencode Providers → Custom provider, opencode Config → Env vars, opencode Models, opencode CLI.

I work on CloudGPU.

Self-Host an LLM on a Rented GPU or Pay per Token? A Break-Even Calculator (2026)

CloudGPU — Sun, 25 Oct 2026 13:08:44 +0000

Self-hosting on a rented GPU is cheaper than a per-token API only when you keep the GPU busy: the break-even is the hourly GPU price divided by the APIâ€™s price per token, expressed as tokens you must actually generate and process every hour. Below that throughput the API wins; above it the GPU wins. This post gives the formula, a worked example with an open 32B model on a single 32 GB card, and a script to run with your own measurements.

The formula

api_cost_per_hour  = (input_tokens_per_hour Ã— input_price + output_tokens_per_hour Ã— output_price) / 1,000,000
gpu_cost_per_hour  = hourly_gpu_price                     # you pay it whether the GPU is busy or idle

break_even_tokens_per_hour = hourly_gpu_price / blended_price_per_token
blended_price_per_token    = (r Ã— input_price + output_price) / ((r + 1) Ã— 1,000,000)    # r = input:output ratio

Then correct for utilisation: if your GPU is busy only 40% of the hours you rent it, the throughput you need while busy is break-even Ã· 0.4.

Worked example: Qwen3 32B (4-bit) on an RTX 5090 32G

Qwen3 32B is an open-weight model (Apache-2.0) that is not offered on the CloudGPU API, so self-hosting is the way to run it here. We use the 4-bit AWQ build because it fits a 32 GB card.

Model size (estimate). Hugging Face metadata for Qwen/Qwen3-32B lists 32,762,123,264 parameters. In Qwen/Qwen3-32B-AWQ, 31.21B of them are stored at 4 bits and 1.56B stay in BF16:

31.21B Ã— 0.5 byte           â‰ˆ 15.6 GB   (4-bit weights)
 1.56B Ã— 2 bytes            â‰ˆ  3.1 GB   (BF16 embeddings, norms, output layer)
AWQ scales/zeros, group 128 â‰ˆ  0.6 GB
total                       â‰ˆ 19.3 GB   (the repository's files add up to 19.33 GB)

The FP8 build (Qwen/Qwen3-32B-FP8, about 34.3 GB of files) does not fit in 32 GB of VRAM; it needs a 48 GB card such as the RTX 4090 48G. Figures read from the Hugging Face API on 2026-10-23 21:12 Beijing time; see also Qwen3 32B VRAM.

Context headroom (estimate). With an FP16 KV cache, Qwen3 32B stores 2 Ã— 64 layers Ã— 8 KV heads Ã— 128 Ã— 2 bytes = 262,144 bytes (about 0.26 MB) per token: about 2.1 GB at 8k tokens and 8.6 GB at 32k. After about 19.3 GB of weights, a 32 GB card has roughly 12.7 GB for the KV cache and runtime overhead, so at most about 48k cached tokens shared by all concurrent requests, fewer once the overhead is counted.

Inputs (2026-10-23 21:13 Beijing time):

Item	Value	Source
RTX 5090 32G list price	$0.59 per hour, billed per minute	https://cloudgpu.app/gpu-prices.json
Per-token reference prices on the same balance	glm-5.3-flash $0.16 in / $0.55 out; gpt-oss-120b $0.135 / $0.54; deepseek-v4-flash $0.27 / $1.08 (per 1M tokens)	/api

What a self-hosted token costs. At a fixed $0.59 per hour, the cost per million tokens depends only on the throughput you sustain. The throughputs below are inputs to the arithmetic, not measurements:

Sustained throughput (all requests, input + output)	Tokens per hour	Cost per 1M tokens
100 tokens/s	0.36M	â‰ˆ $1.64
250 tokens/s	0.90M	â‰ˆ $0.66
500 tokens/s	1.80M	â‰ˆ $0.33
1,000 tokens/s	3.60M	â‰ˆ $0.16

Break-even against per-token prices (3 input : 1 output):

Per-token reference	Blended price	Break-even per hour	Throughput needed at 100% / 50% utilisation
glm-5.3-flash	$0.2575 / 1M	â‰ˆ 2.29M tokens/h	â‰ˆ 636 / 1,273 tokens/s
gpt-oss-120b	$0.236 / 1M	â‰ˆ 2.50M tokens/h	â‰ˆ 694 / 1,387 tokens/s
deepseek-v4-flash	$0.4725 / 1M	â‰ˆ 1.25M tokens/h	â‰ˆ 347 / 694 tokens/s

These are different models from Qwen3 32B, so the table compares cost only: decide on quality first, then use the arithmetic. How to read it: on one card you would need to sustain several hundred tokens per second, aggregated across concurrent requests, for every hour you pay for. That needs batching and steady traffic, and you should verify it with a benchmark on the card. For spiky or low-volume traffic, a per-token API is usually the lower-cost choice.

We do not publish a measured tokens-per-second figure for this card here; measure your own with your prompt lengths and concurrency.

When renting the GPU wins anyway

Price per token is not the only reason to rent:

Situation	Why a GPU makes sense
The model is not offered as an API (e.g. Qwen3 32B on CloudGPU, a community fine-tune, your own LoRA)	There is no per-token option to compare against
Large batch jobs with steady load (offline eval, synthetic data, embeddings, transcription batches)	Utilisation is high and the batch finishes; destroy the instance afterwards
You need custom serving: LoRA adapters, constrained decoding, specific quantisation	You control the stack
You need image pipelines with custom nodes (ComfyUI)	APIs expose fixed models only

When the API wins

Situation	Why
Traffic is spiky or low	You would pay for idle GPU hours
The model is too large for one rented card (e.g. DeepSeek V4, Kimi K3)	Self-hosting would need far more VRAM than one card has
You need the model now, without ops work	No boot, no downloads, no monitoring
You need a closed frontier model	Only available as an API

For that last case, the same CloudGPU balance pays for the API. Our per-1M-token rate for each Claude model is shown next to the official list price on the API page; official list prices were checked 2026-10-24 on the official Claude pricing page and can change after that date. Rates for the other models are on the same page.

What an always-on GPU costs

If you keep a card running 24/7 to serve traffic, the bill is fixed regardless of tokens (list prices, 2026-10-23 21:13 Beijing time):

Card	Per day (24 h)	Per 30 days
RTX 4090 24G ($0.33/h)	$7.92	$237.60
RTX 5090 32G ($0.59/h)	$14.16	$424.80
RTX 4090 48G ($0.65/h)	$15.60	$468.00
A800 80G ($1.09/h)	$26.16	$784.80
A100 80G ($1.49/h)	$35.76	$1,072.80

â€œFixed costâ€ does not mean unlimited output: the cardâ€™s throughput caps how many tokens you can serve. Also budget for restarts â€” destroying an instance deletes its disk, so a redeploy re-downloads weights. Stock changes during the day; check Deploy.

A script for your own numbers

def break_even(gpu_price_h, in_price_m, out_price_m, in_out_ratio=3.0, utilisation=1.0):
    blended = (in_out_ratio * in_price_m + out_price_m) / (in_out_ratio + 1) / 1e6   # $ per token
    tokens_per_hour = gpu_price_h / blended
    return tokens_per_hour, tokens_per_hour / 3600 / utilisation                       # tokens/h, tokens/s while busy

def monthly(gpu_price_h, hours_per_day=24, days=30):
    return gpu_price_h * hours_per_day * days

def cost_per_million(gpu_price_h, tokens_per_second):
    return gpu_price_h / (tokens_per_second * 3600 / 1e6)                              # $ per 1M tokens

# Qwen3 32B AWQ on an RTX 5090 32G at $0.59/h vs a $0.16 in / $0.55 out per-token reference (2026-10-23 21:13)
tph, tps = break_even(0.59, 0.16, 0.55, in_out_ratio=3, utilisation=0.5)
print(f"break-even: {tph/1e6:.2f}M tokens/h, {tps:.0f} tokens/s while busy")
print(f"self-hosted cost at 500 tokens/s: ${cost_per_million(0.59, 500):.2f} per 1M tokens")
print(f"always-on RTX 5090 32G for 30 days: ${monthly(0.59):.2f}")

Replace the prices with the live values from /pricing and /api on the day you decide, and replace utilisation and the input:output ratio with your logs.

FAQ

Is self-hosting an LLM cheaper than using an API?

Only if you keep the GPU busy enough. Divide the GPUâ€™s hourly price by the APIâ€™s blended price per token; if your measured sustained throughput is above that number, self-hosting is cheaper.

How do I calculate the break-even between GPU rental and API pricing?

break-even tokens per hour = hourly GPU price Ã· blended API price per token, where the blended price weights input and output by your traffic ratio. Divide by utilisation to get the throughput you need while busy.

What utilisation do I need for self-hosting to pay off?

It depends on the model and card; compute break-even from the formula, then compare with the throughput you measure under realistic concurrency.

Why rent a GPU if the API is cheaper per token?

For models with no API, fine-tunes and LoRA adapters, custom serving stacks, ComfyUI pipelines, or steady batch jobs that keep the card busy.

Can I use the same balance for GPUs and the API?

Yes. One prepaid balance covers per-minute GPU rental and per-token API calls.

If you want to check the hourly side of the formula, the per-minute GPU rental page explains how a job is billed. Originally published at cloudgpu.app.

Break-even figures are arithmetic on list prices, not performance measurements. Prices were read on 2026-10-23 21:13 Beijing time (UTC+8); model sizes are estimates from Hugging Face metadata read at 21:12; current prices on /deploy and /api take precedence.

KV Cache VRAM Calculator: How Much Memory Does Context Length Need? (2026)

CloudGPU — Sat, 24 Oct 2026 13:07:30 +0000

The KV cache is the memory a transformer uses to remember every token in the context, and its size grows linearly with context length: bytes per token = 2 × layers × KV heads × head dimension × bytes per value. For Qwen3 32B at FP16 that is 256 KiB per token, or about 8 GB for a 32k-token context, on top of the model weights. This is why a model that loads fine can still run out of memory when you raise the context.

This post gives the formula, the per-token numbers for popular open models (read from their config.json), and how much context fits on 24, 32, 48 and 80 GB cards.

The formula

kv_bytes_per_token = 2 × num_hidden_layers × num_key_value_heads × head_dim × bytes_per_value
kv_total_bytes     = kv_bytes_per_token × context_tokens × concurrent_sequences

The 2 is for keys and values.
num_key_value_heads is smaller than the number of attention heads in models with grouped-query attention (GQA), which is why modern models have much smaller caches than older ones.
bytes_per_value is 2 for FP16/BF16, 1 for 8-bit KV cache, about 0.5 for 4-bit.
Concurrent sequences multiply the cache. If your server keeps four parallel slots, budget four times the context.

Per-token KV cache for popular models (FP16)

Architecture values are from each model’s config.json on Hugging Face (read 2026-10-23). GB here means GiB (1024³ bytes).

Model	Layers	KV heads	head_dim	KV per token	8k ctx	32k ctx	128k ctx
Llama 3.1 8B	32	8	128	128 KiB	1.0 GB	4.0 GB	16 GB
Qwen3 8B	36	8	128	144 KiB	1.1 GB	4.5 GB	18 GB*
Qwen3 14B	40	8	128	160 KiB	1.25 GB	5.0 GB	20 GB*
Mistral Small 3 24B	40	8	128	160 KiB	1.25 GB	5.0 GB	— (32k max)
Qwen3 32B	64	8	128	256 KiB	2.0 GB	8.0 GB	32 GB*
Llama 3.3 70B	80	8	128	320 KiB	2.5 GB	10 GB	40 GB
Qwen2.5 72B	80	8	128	320 KiB	2.5 GB	10 GB	40 GB*

* Beyond the model’s native window. Qwen3 dense models are documented with a native 32,768-token context, extendable with YaRN rope scaling; check the model card before going past the native length.

The formula does not apply as-is to models with sliding-window layers (for example Gemma 3 and gpt-oss) or latent attention (DeepSeek V3-style MLA); their caches are smaller than this formula suggests.

How much context fits on each card

Total memory ≈ weights + KV cache + overhead. Weights are the Q4_K_M sizes on our /run pages; overhead is an assumed 1.5 GB for CUDA context, buffers and the runtime. The result is a ceiling for a single sequence; runtimes reserve extra buffers, so plan below it.

Qwen3 32B at Q4_K_M (≈ 19.5 GB weights, 256 KiB/token):

Card class	Left for KV after weights + 1.5 GB	Context ceiling (1 sequence)	Practical guidance
24 GB (RTX 4090 / 3090)	≈ 3 GB	≈ 12k tokens	about 8k; tight
32 GB (RTX 5090)	≈ 11 GB	≈ 45k (native max 32k)	16k–32k comfortable
48 GB (RTX 4090 48G)	≈ 27 GB	≈ 110k	full native 32k, or several parallel slots

Llama 3.3 70B at Q4_K_M (≈ 41.5 GB weights, 320 KiB/token):

Card class	Left for KV	Context ceiling (1 sequence)	Practical guidance
48 GB (RTX 4090 48G / A6000)	≈ 5 GB	≈ 16k	about 8k; tight
80 GB (A100 / A800)	≈ 37 GB	≈ 121k	32k comfortable, with room for parallel requests

The “practical guidance” column matches the per-model notes on our Qwen3 32B page and Llama 3.3 70B page.

A small calculator

def kv_gb(layers, kv_heads, head_dim, ctx, bytes_per_value=2, seqs=1):
    return 2 * layers * kv_heads * head_dim * bytes_per_value * ctx * seqs / 1024**3

def max_ctx(vram_gb, weights_gb, layers, kv_heads, head_dim,
            bytes_per_value=2, overhead_gb=1.5, seqs=1):
    free = vram_gb - weights_gb - overhead_gb
    per_tok = 2 * layers * kv_heads * head_dim * bytes_per_value / 1024**3
    return max(0, int(free / (per_tok * seqs)))

# Qwen3 32B (64 layers, 8 KV heads, head_dim 128), Q4 weights ≈ 19.5 GB
print(kv_gb(64, 8, 128, 32768))          # 8.0
print(max_ctx(32, 19.5, 64, 8, 128))     # ≈ 45k (native limit is 32k)
print(max_ctx(24, 19.5, 64, 8, 128, bytes_per_value=1))  # 8-bit KV: ≈ 24k

Read num_hidden_layers, num_key_value_heads and head_dim (or hidden_size / num_attention_heads) from the model’s config.json.

Three ways to fit more context

1. Quantise the KV cache. Halving bytes per value halves the cache.

# Ollama: 8-bit KV cache (requires flash attention)
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0
ollama serve

# vLLM: FP8 KV cache
vllm serve Qwen/Qwen3-32B-AWQ --kv-cache-dtype fp8 --max-model-len 32768

Quality impact is usually small at 8-bit and more noticeable at 4-bit; test on your own prompts.

2. Set the context you need, not the maximum. In Ollama, pass "options": {"num_ctx": 16384} per request (or set OLLAMA_CONTEXT_LENGTH). In vLLM, --max-model-len caps the context and --gpu-memory-utilization sets how much VRAM the cache may take.

3. Watch parallelism. Ollama’s OLLAMA_NUM_PARALLEL and vLLM’s batching both allocate cache per sequence. If memory runs out under load but not in single tests, this is usually why.

Troubleshooting order when you run out of memory

Say you run Qwen3 32B (Q4_K_M) with Ollama on a 24 GB RTX 4090, set num_ctx to 32768, and get out-of-memory errors or a sudden slowdown (layers spilling to CPU). The formula explains it: about 19.5 GB of weights plus about 8 GB of FP16 KV cache is more than 24 GB. Try these, least effort first:

Lower the context. Back to 8192 and the KV cache is about 2 GB; total about 23 GB — it runs, but tight.
Quantise the KV cache. With OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0, the same memory holds roughly twice the context — about 24k by the formula (a ceiling; leave headroom).
Check parallelism. Set OLLAMA_NUM_PARALLEL=1 to rule out several slots sharing the memory.
Change cards. Still short? Move to a 32 GB RTX 5090 or a 48 GB RTX 4090 48G. Switching means waiting about 2 minutes for the new instance and downloading the model again.

After each change, check real usage with nvidia-smi and note the number; next time you size a card, you will have your own data.

Picking a card from the numbers

Once you have weights + KV + overhead, round up to the next VRAM class and check Deploy for live stock. List prices on 2026-10-23 23:58 Beijing time: RTX 4090 24G $0.33/h, RTX 5090 32G $0.59/h, RTX 4090 48G $0.65/h, A800 80G $1.09/h, A100 80G $1.49/h, billed per minute. Instances start in about 2 minutes. For a side-by-side of those cards, see RTX 4090 vs RTX 5090 vs A100.

FAQ

How do I calculate the KV cache size for a model?

Multiply 2 × layers × KV heads × head dimension × bytes per value (2 for FP16) to get bytes per token, then multiply by context length and the number of parallel sequences. The values are in the model’s config.json.

How much VRAM does a 32k context need for Qwen3 32B?

About 8 GB of FP16 KV cache (256 KiB per token × 32,768 tokens), on top of about 19.5 GB of Q4_K_M weights, so a 32 GB card is the comfortable choice.

Why does my model run out of memory only at long context?

The weights are fixed, but the KV cache grows with every token of context and with each parallel request. Lower num_ctx / max-model-len, reduce parallel slots, or quantise the KV cache.

Does an 8-bit KV cache halve memory?

Yes, the cache itself is halved compared with FP16. In Ollama set OLLAMA_KV_CACHE_TYPE=q8_0 with flash attention on; in vLLM use --kv-cache-dtype fp8.

Does the formula work for every model?

It works for standard and grouped-query attention. Models with sliding-window layers or latent attention use less memory than it predicts.

Once you have your own numbers, the RTX 4090 vs RTX 5090 vs A100 comparison lines up the cards by VRAM class. Originally published at cloudgpu.app.

CloudGPU (cloudgpu.app) has no partnership with, authorization from, or endorsement by the model vendor. Model names are trademarks of their owners. Context ceilings are estimates under the stated assumptions, not measurements. Prices are list prices read on 2026-10-23 23:58 Beijing time (UTC+8); live prices and stock on /deploy take precedence.

DEV Community: CloudGPU

Use DeepSeek V4.1 Flash and Kimi in opencode via a custom OpenAI-compatible provider

What you need

Step 1: Export your key

Step 2: Add the provider to opencode.json

Alternative: store the key with /connect

Step 3: Pick the model

Which one for what?

Troubleshooting

1. The model doesn't appear in /models (or requests fail with "model not found")

2. The base URL is missing /v1

3. The environment variable isn't exported

4. Empty replies from reasoning models

Billing

Recap

Self-Host an LLM on a Rented GPU or Pay per Token? A Break-Even Calculator (2026)

The formula

Worked example: Qwen3 32B (4-bit) on an RTX 5090 32G

When renting the GPU wins anyway

When the API wins

What an always-on GPU costs

A script for your own numbers

FAQ

KV Cache VRAM Calculator: How Much Memory Does Context Length Need? (2026)

The formula

Per-token KV cache for popular models (FP16)

How much context fits on each card

A small calculator

Three ways to fit more context

Troubleshooting order when you run out of memory

Picking a card from the numbers

FAQ

Step 2: Add the provider to `opencode.json`

Alternative: store the key with `/connect`

1. The model doesn't appear in `/models` (or requests fail with "model not found")

2. The base URL is missing `/v1`