<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Viswaretas Kotra</title>
    <description>The latest articles on DEV Community by Viswaretas Kotra (@viswaretas_kotra).</description>
    <link>https://dev.to/viswaretas_kotra</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4172307%2F52f79e2f-4ce1-40f8-bb27-0a62c752e52c.png</url>
      <title>DEV Community: Viswaretas Kotra</title>
      <link>https://dev.to/viswaretas_kotra</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/viswaretas_kotra"/>
    <language>en</language>
    <item>
      <title>vLLM vs Ollama: which should you use to serve an LLM?</title>
      <dc:creator>Viswaretas Kotra</dc:creator>
      <pubDate>Thu, 19 Nov 2026 00:26:32 +0000</pubDate>
      <link>https://dev.to/viswaretas_kotra/vllm-vs-ollama-which-should-you-use-to-serve-an-llm-33b8</link>
      <guid>https://dev.to/viswaretas_kotra/vllm-vs-ollama-which-should-you-use-to-serve-an-llm-33b8</guid>
      <description>&lt;p&gt;Use Ollama to run a model on your own laptop or workstation for yourself, and use vLLM when many users or programs hit the same GPU at once. Nodus covers the production side: it serves open models such as GPT-OSS through an OpenAI-compatible API, or runs vLLM with your own weights on GPUs such as an NVIDIA L40S listed at $0.793/hr on the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing page&lt;/a&gt;. The gap between the two engines is mostly memory accounting: Ollama reserves a full context window for every parallel slot, while vLLM gives each request only the KV cache it actually uses.&lt;/p&gt;

&lt;p&gt;That last point decides most real choices, so this post works it through with arithmetic for Qwen3 8B on one 48 GB GPU. We have not run our own vLLM vs Ollama benchmark. Every capacity figure below is calculated from the model's config and the two projects' docs, and every step is shown so you can redo it for your model.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is the difference between vLLM and Ollama?
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://github.com/ollama/ollama" rel="noopener noreferrer"&gt;Ollama&lt;/a&gt; is an MIT-licensed app for running open models on macOS, Windows and Linux. Its README lists &lt;a href="https://github.com/ggml-org/llama.cpp" rel="noopener noreferrer"&gt;llama.cpp&lt;/a&gt; as its supported backend, and it pulls models from its own library by short names. Two commands get you a chat:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -fsSL https://ollama.com/install.sh &lt;span class="o"&gt;|&lt;/span&gt; sh
ollama run gemma4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://github.com/vllm-project/vllm" rel="noopener noreferrer"&gt;vLLM&lt;/a&gt; is an Apache 2.0 serving engine built for throughput. Its &lt;a href="https://docs.vllm.ai/en/latest/getting_started/quickstart/" rel="noopener noreferrer"&gt;quickstart&lt;/a&gt; lists Linux and Python 3.11 to 3.14 as prerequisites and starts an OpenAI-compatible server on &lt;code&gt;http://localhost:8000&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;uv venv --python 3.12 --seed
&lt;span class="nb"&gt;source&lt;/span&gt; .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen2.5-1.5B-Instruct
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Our explainer &lt;a href="https://www.nodus-compute.ai/blog/what-is-vllm/" rel="noopener noreferrer"&gt;What is vLLM?&lt;/a&gt; covers PagedAttention and continuous batching in depth. This post is about choosing between the two.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Ollama&lt;/th&gt;
&lt;th&gt;vLLM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Built for&lt;/td&gt;
&lt;td&gt;Local use on your own machine&lt;/td&gt;
&lt;td&gt;Serving many requests per GPU&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Operating systems&lt;/td&gt;
&lt;td&gt;macOS, Windows, Linux&lt;/td&gt;
&lt;td&gt;Linux; macOS via the community vllm-metal plugin&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;NVIDIA GPUs&lt;/td&gt;
&lt;td&gt;Compute capability 5.0 and newer&lt;/td&gt;
&lt;td&gt;Compute capability 7.5 and newer&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model format&lt;/td&gt;
&lt;td&gt;Ollama library tags; imports GGUF files&lt;/td&gt;
&lt;td&gt;Hugging Face weights; GGUF through a plugin marked highly experimental&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Default port&lt;/td&gt;
&lt;td&gt;11434, bound to 127.0.0.1&lt;/td&gt;
&lt;td&gt;8000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI-compatible API&lt;/td&gt;
&lt;td&gt;A subset, at &lt;code&gt;/v1&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Parallel requests per model&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;OLLAMA_NUM_PARALLEL&lt;/code&gt;, default 1&lt;/td&gt;
&lt;td&gt;Continuous batching, limited by KV cache&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Models per server&lt;/td&gt;
&lt;td&gt;Several, up to &lt;code&gt;OLLAMA_MAX_LOADED_MODELS&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;One model per server&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;License&lt;/td&gt;
&lt;td&gt;MIT&lt;/td&gt;
&lt;td&gt;Apache 2.0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Sources: &lt;a href="https://docs.ollama.com/faq" rel="noopener noreferrer"&gt;Ollama FAQ&lt;/a&gt;, &lt;a href="https://docs.ollama.com/gpu" rel="noopener noreferrer"&gt;Ollama hardware support&lt;/a&gt;, &lt;a href="https://docs.ollama.com/api/openai-compatibility" rel="noopener noreferrer"&gt;Ollama OpenAI compatibility&lt;/a&gt;, &lt;a href="https://docs.ollama.com/import" rel="noopener noreferrer"&gt;Ollama import docs&lt;/a&gt;, &lt;a href="https://docs.vllm.ai/en/latest/getting_started/quickstart/" rel="noopener noreferrer"&gt;vLLM quickstart&lt;/a&gt;, &lt;a href="https://docs.vllm.ai/en/latest/getting_started/installation/gpu/" rel="noopener noreferrer"&gt;vLLM GPU installation&lt;/a&gt;, &lt;a href="https://docs.vllm.ai/en/latest/getting_started/installation/cpu/" rel="noopener noreferrer"&gt;vLLM CPU installation&lt;/a&gt;, &lt;a href="https://docs.vllm.ai/en/latest/features/quantization/gguf/" rel="noopener noreferrer"&gt;vLLM GGUF docs&lt;/a&gt;. Checked November 18, 2026.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why is vLLM faster than Ollama?
&lt;/h2&gt;

&lt;p&gt;For one person typing into one chat, it often is not meaningfully faster, and we have no measurement that says otherwise. The difference shows up under concurrency, and it starts with defaults.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://docs.ollama.com/faq" rel="noopener noreferrer"&gt;Ollama FAQ&lt;/a&gt; says each model processes &lt;code&gt;OLLAMA_NUM_PARALLEL&lt;/code&gt; requests at the same time, with a default of 1. Extra requests wait in a queue of up to 512 (&lt;code&gt;OLLAMA_MAX_QUEUE&lt;/code&gt;), and past that the server answers with a &lt;code&gt;503&lt;/code&gt; overloaded error. Raising the parallel setting works, but the same page warns that "Required RAM will scale by &lt;code&gt;OLLAMA_NUM_PARALLEL&lt;/code&gt; * &lt;code&gt;OLLAMA_CONTEXT_LENGTH&lt;/code&gt;". Every slot is sized for the longest conversation it might hold.&lt;/p&gt;

&lt;p&gt;vLLM does not size anything per slot. It stores KV cache in small blocks and hands them out as tokens arrive, and it adds and removes requests from the running batch at every step. A request holding 2,000 tokens uses memory for 2,000 tokens, even if the server allows 32,768. When the cache does fill up, the &lt;a href="https://docs.vllm.ai/en/latest/configuration/optimization/" rel="noopener noreferrer"&gt;vLLM optimization docs&lt;/a&gt; say it preempts a request and recomputes it later, which costs latency but keeps the server up.&lt;/p&gt;

&lt;p&gt;Decoding is limited by memory bandwidth, so a GPU produces far more total tokens when it works on many requests in each step. Our post on &lt;a href="https://www.nodus-compute.ai/blog/llm-tokens-per-second/" rel="noopener noreferrer"&gt;LLM tokens per second&lt;/a&gt; shows how large that batching effect is. More requests in memory means a bigger batch, and that is where vLLM's production speed comes from.&lt;/p&gt;

&lt;h2&gt;
  
  
  How many concurrent users fit on one GPU with each?
&lt;/h2&gt;

&lt;p&gt;Here is the arithmetic for Qwen3 8B, which both projects can serve. Its &lt;a href="https://huggingface.co/Qwen/Qwen3-8B/blob/main/config.json" rel="noopener noreferrer"&gt;config.json&lt;/a&gt; lists 36 layers, 8 key-value heads and a head dimension of 128, in bfloat16. Its &lt;a href="https://huggingface.co/Qwen/Qwen3-8B" rel="noopener noreferrer"&gt;model page&lt;/a&gt; reports 8,190,735,360 parameters. We use 16-bit weights in both engines so only the KV accounting differs; Ollama's library has this as the &lt;a href="https://ollama.com/library/qwen3:8b-fp16" rel="noopener noreferrer"&gt;qwen3:8b-fp16&lt;/a&gt; tag.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;KV cache per token: 2 (key and value) x 36 layers x 8 heads x 128 dims x 2 bytes = 147,456 bytes.&lt;/li&gt;
&lt;li&gt;Weights: 8,190,735,360 x 2 bytes = 16.38 GB.&lt;/li&gt;
&lt;li&gt;GPU: an NVIDIA L40S, treated as 48 GB. That leaves 48 minus 16.38 = 31.62 GB for KV cache. This ignores activations and CUDA graph memory, so every count below is an upper bound.&lt;/li&gt;
&lt;li&gt;One Ollama slot at a 32,768-token context: 32,768 x 147,456 bytes = 4.83 GB, reserved whether the conversation uses it or not.&lt;/li&gt;
&lt;li&gt;One Ollama slot at 4,096 tokens: 4,096 x 147,456 bytes = 0.604 GB.&lt;/li&gt;
&lt;li&gt;One vLLM request that averages 2,000 tokens of prompt plus output: 2,000 x 147,456 bytes = 0.295 GB.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The 2,000-token average is our assumption for a typical chat turn with some history. Replace it with your own logs.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F04b1rnmfcmcgxoldf6d6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F04b1rnmfcmcgxoldf6d6.png" alt="KV cache memory per concurrent request for Qwen3 8B in 16-bit: 4.83 GB for an Ollama slot at a 32,768-token context, 0.604 GB at a 4,096-token context, and 0.295 GB for a vLLM request averaging 2,000 tokens." width="800" height="304"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Ollama pays for the maximum context on every slot; vLLM pays for the tokens a request actually holds.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Now divide the 31.62 GB budget. At a 32,768-token context, Ollama fits 6 parallel slots. Cut the context to 16,384 and it fits 13, at 8,192 it fits 26, and at 4,096 it fits 52. vLLM fits about 107 requests averaging 2,000 tokens, and that number does not change when you raise the maximum context, because the maximum is no longer reserved up front.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F99gm7l4cn89ds9jl56ns.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F99gm7l4cn89ds9jl56ns.png" alt="Concurrent requests that fit in 31.62 GB of KV cache on a 48 GB GPU with Qwen3 8B: Ollama fits 52, 26, 13 and 6 slots at 4,096, 8,192, 16,384 and 32,768 tokens of context, while vLLM fits about 107 requests averaging 2,000 tokens at every maximum context." width="800" height="416"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Doubling Ollama's context halves its parallel slots; vLLM's ceiling depends on average use, not the maximum.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;This is why Ollama's own advice pulls against concurrency. Its &lt;a href="https://docs.ollama.com/context-length" rel="noopener noreferrer"&gt;context length page&lt;/a&gt; recommends at least 64,000 tokens for agents, web search and coding tools, and every extra token of context is multiplied by every slot.&lt;/p&gt;

&lt;p&gt;Ollama can stretch the budget. The FAQ says &lt;code&gt;OLLAMA_KV_CACHE_TYPE=q8_0&lt;/code&gt; uses about half the memory of the default &lt;code&gt;f16&lt;/code&gt;, with a very small loss in precision. That would take the 32,768-token case from 6 slots to about 13. vLLM has a comparable option in &lt;code&gt;--kv-cache-dtype fp8&lt;/code&gt;, listed in its &lt;a href="https://docs.vllm.ai/en/latest/configuration/engine_args/" rel="noopener noreferrer"&gt;engine arguments&lt;/a&gt;, so quantizing the cache helps both engines rather than closing the gap.&lt;/p&gt;
&lt;h2&gt;
  
  
  What does each concurrent user cost per hour?
&lt;/h2&gt;

&lt;p&gt;Spread the same GPU across those slots. On an L40S listed at $0.793/hr on the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing page&lt;/a&gt;, the cost of holding room for one concurrent request for an hour is the hourly price divided by the slot count.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Setup on one L40S, Qwen3 8B&lt;/th&gt;
&lt;th&gt;Concurrent requests&lt;/th&gt;
&lt;th&gt;$ per concurrent request-hour&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Ollama, 32,768-token context&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;$0.132&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ollama, 32,768 context, q8_0 KV cache&lt;/td&gt;
&lt;td&gt;about 13&lt;/td&gt;
&lt;td&gt;$0.061&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ollama, 4,096-token context&lt;/td&gt;
&lt;td&gt;52&lt;/td&gt;
&lt;td&gt;$0.015&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;vLLM, any max context, 2,000-token average&lt;/td&gt;
&lt;td&gt;about 107&lt;/td&gt;
&lt;td&gt;$0.0074&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Prices checked November 18, 2026. For example, $0.793 / 6 = $0.132, and $0.793 / 107 = $0.0074.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbkedmq9povg4ownfvv8x.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbkedmq9povg4ownfvv8x.png" alt="Hourly cost per concurrent request on an L40S at $0.793/hr serving Qwen3 8B: $0.132 for Ollama at a 32,768-token context, $0.061 with a q8_0 KV cache, $0.015 at a 4,096-token context, and $0.0074 for vLLM at a 2,000-token average." width="800" height="328"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;At a production-length context, room for one concurrent user costs about 18 times more under Ollama's fixed slots.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Two caveats keep this honest. Holding a request in memory is not the same as serving it fast, since per-user speed drops as the batch grows. And vLLM's 107 assumes your average holds; a burst of long conversations fills the cache sooner and triggers preemption.&lt;/p&gt;
&lt;h2&gt;
  
  
  Which should you choose?
&lt;/h2&gt;

&lt;p&gt;Use this rule, worked from the arithmetic above:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Work out your KV budget: GPU memory minus weights.&lt;/li&gt;
&lt;li&gt;Work out one Ollama slot: the context you need x KV bytes per token.&lt;/li&gt;
&lt;li&gt;If your peak concurrent requests fit in budget / slot, Ollama is fine, and simpler to run.&lt;/li&gt;
&lt;li&gt;If they do not, or your context is long and your average use is short, use vLLM. Its capacity is roughly the budget / (average tokens x KV bytes per token), so its advantage is about your maximum context divided by your average tokens. In our example that ratio is 32,768 / 2,000, or about 16x.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A developer on a Mac, a private notebook, or an internal tool with a handful of users belongs on Ollama. An API behind a product, an agent fleet, or a batch job over millions of prompts belongs on vLLM.&lt;/p&gt;
&lt;h2&gt;
  
  
  Where is Ollama the better choice?
&lt;/h2&gt;

&lt;p&gt;Ollama is much easier on a personal machine. It installs as an app on macOS and Windows. Its &lt;a href="https://docs.ollama.com/gpu" rel="noopener noreferrer"&gt;hardware docs&lt;/a&gt; say it uses Apple's Metal API for GPU acceleration and supports NVIDIA cards back to compute capability 5.0. vLLM's quickstart lists Linux, its GPU builds start at compute capability 7.5, and its GPU path on a Mac goes through a community plugin.&lt;/p&gt;

&lt;p&gt;It also juggles models better on one box. The FAQ says Ollama can keep up to 3 models per GPU loaded by default if they fit, and unloads idle ones to make room. The vLLM quickstart says its server hosts one model at a time. For trying five models in an afternoon, that matters more than throughput.&lt;/p&gt;

&lt;p&gt;Quantized GGUF files are native to Ollama: its &lt;a href="https://docs.ollama.com/import" rel="noopener noreferrer"&gt;import docs&lt;/a&gt; load one from a one-line Modelfile. vLLM's &lt;a href="https://docs.vllm.ai/en/latest/features/quantization/gguf/" rel="noopener noreferrer"&gt;GGUF docs&lt;/a&gt; call its support highly experimental and under-optimized, and now require a separate plugin. If you only have a GGUF file, Ollama is the natural home.&lt;/p&gt;

&lt;p&gt;Privacy is simple too. The FAQ says Ollama does not see your prompts or data when you run locally. For data that cannot leave a laptop, our guide on &lt;a href="https://www.nodus-compute.ai/blog/run-llm-locally-vs-cloud/" rel="noopener noreferrer"&gt;running an LLM locally vs in the cloud&lt;/a&gt; covers when that trade is worth it.&lt;/p&gt;
&lt;h2&gt;
  
  
  How do you move from Ollama to production on Nodus?
&lt;/h2&gt;

&lt;p&gt;Both Ollama and Nodus speak the OpenAI API, so the smallest move is a base URL and a key. This is the local example from &lt;a href="https://docs.ollama.com/api/openai-compatibility" rel="noopener noreferrer"&gt;Ollama's OpenAI compatibility page&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;base_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'http://localhost:11434/v1/'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;api_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'ollama'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# required but ignored&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And this is the client from the &lt;a href="https://www.nodus-compute.ai/docs/guides/inference/" rel="noopener noreferrer"&gt;Nodus inference guide&lt;/a&gt;, which marks the feature Beta:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"https://platform-inference.nodus-compute.ai/v1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;api_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"YOUR_NODUS_API_KEY"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;reply&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"nodus/gpt-oss-120b"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"What is the capital of France?"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reply&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ollama's docs use &lt;code&gt;gpt-oss:20b&lt;/code&gt; locally, and the Nodus catalog serves the same model as &lt;code&gt;nodus/gpt-oss-20b&lt;/code&gt;. The guide says Nodus adds no request-rate, token-rate or concurrency caps to inference, though available model capacity and balance checks still apply. There are no slots to size.&lt;/p&gt;

&lt;p&gt;If you need your own fine-tuned weights or exact engine settings, run vLLM itself on a Nodus GPU you choose. Our guide to &lt;a href="https://www.nodus-compute.ai/blog/host-your-own-llm-api/" rel="noopener noreferrer"&gt;hosting your own LLM behind an OpenAI-style API&lt;/a&gt; walks through &lt;code&gt;vllm serve&lt;/code&gt; as a Nodus Job, including the API-key gap in vLLM's server.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Can Ollama run in production?
&lt;/h3&gt;

&lt;p&gt;It can serve a small internal tool well, especially with &lt;code&gt;OLLAMA_NUM_PARALLEL&lt;/code&gt; raised and the context kept short. Watch the queue: once 512 requests are waiting, the server returns &lt;code&gt;503&lt;/code&gt;, so plan the slot count from the arithmetic above rather than discovering it under load.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does Ollama protect its API with a key?
&lt;/h3&gt;

&lt;p&gt;Not locally: its OpenAI compatibility page says the client needs an API key value but Ollama ignores it. The server binds to 127.0.0.1 by default, so put an authenticating proxy in front before you set &lt;code&gt;OLLAMA_HOST&lt;/code&gt; to expose it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can vLLM load the models I downloaded with Ollama?
&lt;/h3&gt;

&lt;p&gt;vLLM can serve GGUF files after you install &lt;code&gt;vllm-gguf-plugin&lt;/code&gt;, and its docs recommend passing the base model's tokenizer with &lt;code&gt;--tokenizer&lt;/code&gt;. Support is marked highly experimental, so for production it is usually better to serve the original Hugging Face weights.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does Ollama use so much memory when I raise the context?
&lt;/h3&gt;

&lt;p&gt;The KV cache for every parallel slot is allocated for the full context. For Qwen3 8B in 16-bit, each 32,768-token slot holds 4.83 GB before a single token arrives, and four slots need 19.3 GB.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is vLLM worth it for a single user?
&lt;/h3&gt;

&lt;p&gt;Usually not. Continuous batching pays off when there are several requests to batch, and for one user Ollama's simpler setup wins.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.nodus-compute.ai/blog/vllm-vs-ollama/" rel="noopener noreferrer"&gt;Nodus blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>vllm</category>
      <category>ollama</category>
      <category>ai</category>
    </item>
    <item>
      <title>How many tokens per second should an LLM serve on one GPU?</title>
      <dc:creator>Viswaretas Kotra</dc:creator>
      <pubDate>Wed, 18 Nov 2026 21:35:18 +0000</pubDate>
      <link>https://dev.to/viswaretas_kotra/how-many-tokens-per-second-should-an-llm-serve-on-one-gpu-27b8</link>
      <guid>https://dev.to/viswaretas_kotra/how-many-tokens-per-second-should-an-llm-serve-on-one-gpu-27b8</guid>
      <description>&lt;p&gt;One NVIDIA H100 serving an 8B model at full load produces about 1,300 to 26,000 output tokens per second in total, depending on prompt length, while any single user sees only a few dozen to a few hundred, capped by memory bandwidth. On an H100 SXM listed at $2.60/hr on the Nodus pricing page, NVIDIA's published 14,992 tokens per second for Llama 3.1 8B works out to about $0.05 per million output tokens, and serving one user at a time costs about 72 times more. So the right target is not one number: set a per-user floor for chat (around 30 to 50 tokens per second), then pack in as many users as the GPU allows while that floor holds.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does "tokens per second" actually measure?
&lt;/h2&gt;

&lt;p&gt;It measures two different things, and most confusion comes from mixing them. NVIDIA's &lt;a href="https://docs.nvidia.com/nim/benchmarking/llm/metrics" rel="noopener noreferrer"&gt;LLM benchmarking metrics guide&lt;/a&gt; defines both.&lt;/p&gt;

&lt;p&gt;Total tokens per second (TPS) per system is all output tokens across every simultaneous request, divided by the benchmark's wall time. This is what decides your cost.&lt;/p&gt;

&lt;p&gt;TPS per user is one request's output length divided by its end-to-end latency. It approaches 1 divided by the inter-token latency (ITL), the gap between consecutive tokens. This is what a person watching a chat window feels.&lt;/p&gt;

&lt;p&gt;The guide states the tradeoff plainly: as concurrent requests increase, total system TPS rises while TPS per user falls. Two more numbers complete the picture:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;What it tells you&lt;/th&gt;
&lt;th&gt;Who cares&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Time to first token (TTFT)&lt;/td&gt;
&lt;td&gt;Queueing plus prefill plus network before the first token&lt;/td&gt;
&lt;td&gt;Chat users, voice agents&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Inter-token latency (ITL)&lt;/td&gt;
&lt;td&gt;Time between tokens while streaming&lt;/td&gt;
&lt;td&gt;Chat users&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TPS per user&lt;/td&gt;
&lt;td&gt;Output speed for one request&lt;/td&gt;
&lt;td&gt;Chat, agents waiting on a reply&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Total TPS per system&lt;/td&gt;
&lt;td&gt;Output across all requests&lt;/td&gt;
&lt;td&gt;Batch jobs, whoever pays the bill&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Definitions from &lt;a href="https://docs.nvidia.com/nim/benchmarking/llm/metrics" rel="noopener noreferrer"&gt;NVIDIA's metrics guide&lt;/a&gt;. Tools differ on whether TTFT is included in ITL, so compare numbers only when the definitions match.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why is one user's speed capped by memory bandwidth?
&lt;/h2&gt;

&lt;p&gt;Generating each new token means reading every model weight from GPU memory once. With a single request, there is nothing else to share that read with, so the ceiling is roughly memory bandwidth divided by the bytes of weights. Compute barely matters at this point.&lt;/p&gt;

&lt;p&gt;Here is that ceiling worked out for Llama 3.1 8B at 16-bit precision, about 16 GB of weights, using bandwidth from NVIDIA's own spec pages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;L4: 300 GB/s / 16 GB = 18.75 tokens per second&lt;/li&gt;
&lt;li&gt;L40S: 864 / 16 = 54 tokens per second&lt;/li&gt;
&lt;li&gt;A100 80 GB SXM: 2,039 / 16 = 127.4 tokens per second&lt;/li&gt;
&lt;li&gt;H100 SXM: 3,350 / 16 = 209.4 tokens per second&lt;/li&gt;
&lt;li&gt;H200: 4,800 / 16 = 300 tokens per second&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Bandwidth figures are from NVIDIA's pages for the &lt;a href="https://www.nvidia.com/en-us/data-center/l4/" rel="noopener noreferrer"&gt;L4&lt;/a&gt;, &lt;a href="https://www.nvidia.com/en-us/data-center/l40s/" rel="noopener noreferrer"&gt;L40S&lt;/a&gt;, &lt;a href="https://www.nvidia.com/en-us/data-center/a100/" rel="noopener noreferrer"&gt;A100&lt;/a&gt;, &lt;a href="https://www.nvidia.com/en-us/data-center/h100/" rel="noopener noreferrer"&gt;H100&lt;/a&gt; and &lt;a href="https://www.nvidia.com/en-us/data-center/h200/" rel="noopener noreferrer"&gt;H200&lt;/a&gt;. These are upper bounds we calculated, not measurements. Real decoding also reads the KV cache and pays kernel overhead, so expect less.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhi2gfn755db9smj7y0y0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhi2gfn755db9smj7y0y0.png" alt="Single-user decode ceiling for a 16 GB model (Llama 3.1 8B at 16-bit): L4 18.75 tokens/s, L40S 54, A100 80 GB 127.4, H100 SXM 209.4, H200 300." width="800" height="372"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Single-stream speed tracks memory bandwidth, so an H200 can stream one user about 16 times faster than an L4.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The same arithmetic gives two quick rules. Halving the bytes per weight, for example by serving in 8-bit (FP8), doubles the ceiling: 418.75 tokens per second for the 8B model on an H100. A 70B model in FP8 is about 70 GB, which leaves an H100 a ceiling of 3,350 / 70 = 47.9 tokens per second for one user, before any KV cache traffic.&lt;/p&gt;

&lt;h2&gt;
  
  
  How many tokens per second does one GPU serve with batching?
&lt;/h2&gt;

&lt;p&gt;Far more than the single-user ceiling, because a batch of requests shares each read of the weights. NVIDIA's &lt;a href="https://nvidia.github.io/TensorRT-LLM/performance/perf-overview.html" rel="noopener noreferrer"&gt;TensorRT-LLM performance overview&lt;/a&gt; publishes total output throughput at maximum load. For Llama 3.1 8B in FP8 on one H100 SXM:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Input, output tokens per request&lt;/th&gt;
&lt;th&gt;Output tokens per second, one H100&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;128, 128&lt;/td&gt;
&lt;td&gt;26,401&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1,000, 1,000&lt;/td&gt;
&lt;td&gt;14,992&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5,000, 500&lt;/td&gt;
&lt;td&gt;3,276&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;20,000, 2,000&lt;/td&gt;
&lt;td&gt;1,341&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Source: &lt;a href="https://nvidia.github.io/TensorRT-LLM/performance/perf-overview.html" rel="noopener noreferrer"&gt;TensorRT-LLM performance overview&lt;/a&gt;, throughput tables, checked November 18, 2026. NVIDIA describes these as reference points, not peak performance.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F20htrzyy9maudp55pezb.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F20htrzyy9maudp55pezb.png" alt="Llama 3.1 8B FP8 output throughput on one H100 at maximum load: 26,401 tokens/s at 128 in and 128 out, 14,992 at 1,000 and 1,000, 3,276 at 5,000 and 500, and 1,341 at 20,000 and 2,000." width="800" height="328"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Long prompts cut the same GPU's output throughput by up to 20 times, so ask any benchmark what input length it used.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Three things in that table matter more than the headline number. Prompt length dominates: long inputs spend GPU time on prefill and fill memory with KV cache, so fewer requests fit in a batch. Bigger models scale down fast: the same page lists Llama 3.3 70B FP8 at 4,181 tokens per second on two H100s at 1,000 in and 1,000 out, or about 2,091 per GPU. A faster GPU helps less than its spec suggests: the H200 row for 8B at 1,000 and 1,000 is 17,162, only 14% above the H100.&lt;/p&gt;

&lt;p&gt;These are maximum-throughput runs, and they hide the user's experience. The example output on the same page reports 5,531 total output tokens per second alongside a per-user output speed of 5.47 tokens per second and an average TTFT of 52.7 seconds. That is a fine result for an overnight batch job and an unusable one for chat. NVIDIA's &lt;a href="https://docs.nvidia.com/nim/benchmarking/llm/parameters" rel="noopener noreferrer"&gt;parameters guide&lt;/a&gt; explains why: throughput saturates near the maximum batch size while latency keeps rising.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is 40 tokens per second good?
&lt;/h2&gt;

&lt;p&gt;For one person reading a streamed chat reply, yes. At NVIDIA's &lt;a href="https://docs.nvidia.com/nim/benchmarking/llm/overview" rel="noopener noreferrer"&gt;rule of thumb&lt;/a&gt; of about 0.75 English words per token, 40 tokens per second is 30 words per second, with 25 ms between tokens. That is comfortably faster than people read, so pushing a chat endpoint much higher per user mostly costs money.&lt;/p&gt;

&lt;p&gt;It is not good in two other cases. Reasoning models and agents generate tokens nobody reads: a 2,000-token reasoning trace at 40 tokens per second is 50 seconds of waiting before the answer starts. For those, per-user speed is latency, and faster is worth paying for.&lt;/p&gt;

&lt;p&gt;It is also a poor number if it is your whole GPU. One H100 at $2.60/hr producing 40 tokens per second costs $2.60 x 1,000,000 / (40 x 3,600) = $18.06 per million output tokens. The same GPU at NVIDIA's 14,992 tokens per second costs $0.05.&lt;/p&gt;

&lt;p&gt;The decision rule we use:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Chat and voice: pick a per-user floor (30 to 50 tokens per second is comfortable for reading) and a TTFT limit, then raise concurrency until either one breaks.&lt;/li&gt;
&lt;li&gt;Agents and reasoning: set the floor from the total wait you can accept. Hidden tokens divided by tokens per second is seconds of waiting.&lt;/li&gt;
&lt;li&gt;Batch jobs: ignore per-user speed and TTFT. Maximize total throughput and judge by cost per million tokens.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  How do you convert tokens per second into cost per million tokens?
&lt;/h2&gt;

&lt;p&gt;Divide the hourly price by the tokens produced in an hour. We use the H100 SXM listed at $2.60/hr on the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing page&lt;/a&gt;, checked November 18, 2026, with NVIDIA's published 8B throughput:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;26,401 tokens per second: $2.60 x 1,000,000 / (26,401 x 3,600) = $0.027 per million output tokens&lt;/li&gt;
&lt;li&gt;14,992 tokens per second: $0.048&lt;/li&gt;
&lt;li&gt;3,276 tokens per second: $0.22&lt;/li&gt;
&lt;li&gt;1,341 tokens per second: $0.54&lt;/li&gt;
&lt;li&gt;One user at the 209.4 tokens per second bandwidth ceiling: $3.45&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3p6f3ozpu5n6o310d8b2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3p6f3ozpu5n6o310d8b2.png" alt="Cost per million output tokens for Llama 3.1 8B on one H100 at $2.60/hr: $3.45 for one user at the bandwidth ceiling, $0.54 at 20,000 in and 2,000 out, $0.22 at 5,000 and 500, $0.048 at 1,000 and 1,000, $0.027 at 128 and 128." width="800" height="372"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Batching, not the hourly rate, sets the price: the same H100 hour costs 72 times more per token when it serves one user.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The 70B case costs more because it needs two GPUs: two H100s at $5.20/hr and 4,181 tokens per second come to $0.35 per million output tokens. The H200, listed at $3.593/hr, comes to $0.058 at 17,162 tokens per second, slightly more per token than the H100 here because its price rises faster than this throughput.&lt;/p&gt;

&lt;p&gt;These figures assume the GPU stays saturated and count output tokens only. Real traffic has idle hours and input tokens. Our post on &lt;a href="https://www.nodus-compute.ai/blog/llm-inference-cost-per-token/" rel="noopener noreferrer"&gt;LLM inference cost per million tokens&lt;/a&gt; covers input, cached and reasoning token pricing, and &lt;a href="https://www.nodus-compute.ai/blog/self-hosting-llm-vs-api-cost/" rel="noopener noreferrer"&gt;whether self-hosting is cheaper than an API&lt;/a&gt; works through break-even volume and how busy the GPU must be.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you measure tokens per second for your own workload on Nodus?
&lt;/h2&gt;

&lt;p&gt;Run a concurrency sweep with your real prompt and output lengths, not synthetic 128-token prompts. NVIDIA's &lt;a href="https://docs.nvidia.com/nim/benchmarking/llm/parameters" rel="noopener noreferrer"&gt;parameters guide&lt;/a&gt; recommends controlling load by concurrency, the number of requests in flight, and recording TTFT, ITL and total TPS at each level. The useful point is the highest concurrency where your per-user floor and TTFT limit still hold. Total TPS at that point, divided into the hourly price, is your real cost per token.&lt;/p&gt;

&lt;p&gt;To stand up the server itself, our guide to &lt;a href="https://www.nodus-compute.ai/blog/host-your-own-llm-api/" rel="noopener noreferrer"&gt;hosting your own LLM behind an OpenAI-style API&lt;/a&gt; has the vLLM commands, and &lt;a href="https://www.nodus-compute.ai/blog/vllm-vs-sglang/" rel="noopener noreferrer"&gt;vLLM vs SGLang&lt;/a&gt; compares the two engines. On Nodus, a &lt;a href="https://www.nodus-compute.ai/docs/guides/sweeps/" rel="noopener noreferrer"&gt;Sweep&lt;/a&gt; runs the same Job across GPU types and parameter values, then reports each cell's cost, wall time and throughput, so a single &lt;code&gt;nodus apply -f sweep.yaml&lt;/code&gt; can be set up to compare, say, an L40S against an H100 at several concurrency levels under a &lt;code&gt;maxCostUSD&lt;/code&gt; cap. We have not published such a sweep ourselves yet. Throughput there counts the units your program reports, so log one unit per request and compute tokens per second from your benchmark client.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Why do published tokens per second numbers disagree so much?
&lt;/h3&gt;

&lt;p&gt;They usually differ in input and output length, precision (16-bit, FP8, FP4), engine, concurrency and whether they report per-user or total throughput. NVIDIA's own H100 table spans 1,341 to 26,401 tokens per second for one model on one GPU just by changing lengths.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does a B200 with FP4 lower the cost per token for a 70B model?
&lt;/h3&gt;

&lt;p&gt;In NVIDIA's published numbers, yes. The same TensorRT-LLM page lists Llama 3.3 70B in FP4 at 6,434 output tokens per second on one B200 at 1,000 in and 1,000 out. At the B200's listed $5.50/hr on Nodus, that is $0.24 per million output tokens, against $0.35 for FP8 on two H100s, provided FP4 quality passes your evals.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can a small GPU like an L4 serve chat?
&lt;/h3&gt;

&lt;p&gt;For a small model and few users, yes, if the model is quantized. The 16 GB 8B model has an 18.75 tokens per second single-user ceiling on an L4, below a comfortable chat floor, while 8-bit weights double that to 37.5. The L4 is listed at $0.430/hr on the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing page&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.nodus-compute.ai/blog/llm-tokens-per-second/" rel="noopener noreferrer"&gt;Nodus blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>machinelearning</category>
      <category>gpu</category>
      <category>performance</category>
    </item>
    <item>
      <title>What are the best Shadeform alternatives for buying GPU time?</title>
      <dc:creator>Viswaretas Kotra</dc:creator>
      <pubDate>Wed, 18 Nov 2026 19:05:58 +0000</pubDate>
      <link>https://dev.to/viswaretas_kotra/what-are-the-best-shadeform-alternatives-for-buying-gpu-time-38dn</link>
      <guid>https://dev.to/viswaretas_kotra/what-are-the-best-shadeform-alternatives-for-buying-gpu-time-38dn</guid>
      <description>&lt;p&gt;Shadeform is good at one job: finding a GPU virtual machine across 30+ clouds and launching it from one account, so the right alternative depends on whether you also want someone to finish the run for you. Nodus takes the job itself, runs it on the cheapest capacity that will finish it, and restores its checkpoint if a machine is lost, at $2.60 per NVIDIA H100 SXM hour on the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing page&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;SkyPilot is the open-source choice for your own cloud accounts, Vast.ai has some of the lowest H100 sticker prices, and Runpod suits per-second container pods. We build Nodus, so read with that in mind. Every fact about another company links to its own docs or pricing page, and every sum is written out.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does Shadeform actually do for you?
&lt;/h2&gt;

&lt;p&gt;Shadeform is a broker for VMs, not a job runner. Its &lt;a href="https://docs.shadeform.ai/getting-started/introduction" rel="noopener noreferrer"&gt;docs&lt;/a&gt; describe a cloud GPU marketplace where you compare availability, price and specs across 30+ providers, then launch through one console, REST API, &lt;code&gt;shade&lt;/code&gt; CLI or MCP server. Each instance is a plain VM you SSH into, with an optional startup script, container and persistent volume.&lt;/p&gt;

&lt;p&gt;The pricing model is simple. Shadeform's &lt;a href="https://docs.shadeform.ai/getting-started/faq" rel="noopener noreferrer"&gt;FAQ&lt;/a&gt; says it charges the same rate as going direct to the provider, with no extra fee, and passes your payment through. By default it launches into its own account with each provider (Shade Cloud), so you never sign up with them. You can also link your own provider account to spend existing credits or a negotiated rate.&lt;/p&gt;

&lt;p&gt;The same FAQ is clear about the limits. Shadeform inherits each provider's SLA, and its &lt;a href="https://docs.shadeform.ai/getting-started/concepts" rel="noopener noreferrer"&gt;Docker launch option&lt;/a&gt; starts a container on your VM rather than running a serverless one. On &lt;a href="https://docs.shadeform.ai/guides/spotinstances" rel="noopener noreferrer"&gt;spot instances&lt;/a&gt;, a reclaimed machine is terminated, and the guide asks you to checkpoint to a volume, make jobs resumable, and poll and relaunch yourself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which Shadeform alternative fits which buyer?
&lt;/h2&gt;

&lt;p&gt;Pick by who handles the run after launch. Shadeform and Vast.ai hand you a machine. SkyPilot and Nodus take a job and keep it going.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Option&lt;/th&gt;
&lt;th&gt;What you buy&lt;/th&gt;
&lt;th&gt;Who restarts a lost run&lt;/th&gt;
&lt;th&gt;H100 price signal&lt;/th&gt;
&lt;th&gt;Good fit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Shadeform&lt;/td&gt;
&lt;td&gt;VMs from 30+ clouds, one bill&lt;/td&gt;
&lt;td&gt;You, from your own checkpoints&lt;/td&gt;
&lt;td&gt;Single H100 listings $1.66 to $5.99, average $3.47&lt;/td&gt;
&lt;td&gt;Finding the cheapest live VM across clouds&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nodus&lt;/td&gt;
&lt;td&gt;Jobs, Workspaces, Sandboxes, Functions&lt;/td&gt;
&lt;td&gt;Nodus, from the last committed checkpoint&lt;/td&gt;
&lt;td&gt;H100 SXM $2.60&lt;/td&gt;
&lt;td&gt;Runs that should finish without babysitting&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SkyPilot&lt;/td&gt;
&lt;td&gt;Open-source scheduler over your accounts&lt;/td&gt;
&lt;td&gt;SkyPilot managed jobs&lt;/td&gt;
&lt;td&gt;Whatever your clouds charge&lt;/td&gt;
&lt;td&gt;Teams with several cloud accounts and credits&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vast.ai&lt;/td&gt;
&lt;td&gt;Marketplace GPU instances&lt;/td&gt;
&lt;td&gt;You&lt;/td&gt;
&lt;td&gt;H100 SXM from $1.70, median $1.73&lt;/td&gt;
&lt;td&gt;Lowest sticker price, flexible hosts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runpod&lt;/td&gt;
&lt;td&gt;Per-second container pods&lt;/td&gt;
&lt;td&gt;You&lt;/td&gt;
&lt;td&gt;H100 SXM $4.39 Secure, from $2.69 Community&lt;/td&gt;
&lt;td&gt;Containers on one provider&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Sources: Shadeform &lt;a href="https://shadeform.com/directory/instances/latitude/H100" rel="noopener noreferrer"&gt;H100 directory&lt;/a&gt; and &lt;a href="https://docs.shadeform.ai/getting-started/faq" rel="noopener noreferrer"&gt;FAQ&lt;/a&gt;, &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing&lt;/a&gt; and &lt;a href="https://www.nodus-compute.ai/docs/concepts/attempts-and-recovery/" rel="noopener noreferrer"&gt;attempts and recovery&lt;/a&gt;, &lt;a href="https://docs.skypilot.co/en/latest/examples/managed-jobs.html" rel="noopener noreferrer"&gt;SkyPilot managed jobs&lt;/a&gt;, &lt;a href="https://vast.ai/pricing" rel="noopener noreferrer"&gt;Vast.ai pricing&lt;/a&gt;, &lt;a href="https://www.runpod.io/pricing" rel="noopener noreferrer"&gt;Runpod pricing&lt;/a&gt;. Prices checked November 18, 2026.&lt;/p&gt;

&lt;p&gt;One caveat on the Shadeform row. Its cheapest single-H100 listing is a PCIe VM, while the Nodus rate is for the H100 SXM, so the cards are close but not identical. Shadeform's directory also changes as providers come and go, and its own &lt;a href="https://docs.shadeform.ai/guides/mostaffordablegpus" rel="noopener noreferrer"&gt;cheapest-GPU guide&lt;/a&gt; shows that the lowest listed price is not always available.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F23pzcks9thyhtzfqrnf2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F23pzcks9thyhtzfqrnf2.png" alt="Listed price for one NVIDIA H100 per GPU-hour on November 18, 2026: Shadeform lowest listing $1.66, Vast.ai H100 SXM median $1.73, Nodus H100 SXM $2.60, Runpod Community from $2.69, Shadeform average listing $3.47, Runpod Secure $4.39, Shadeform highest listing $5.99." width="800" height="460"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Shadeform's spread is the story: its listings run from below Nodus to more than twice the Nodus rate.&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  Is Nodus cheaper than Shadeform for an 80-hour H100 run?
&lt;/h2&gt;

&lt;p&gt;Nodus is cheaper than the average Shadeform H100 listing and dearer than the cheapest one. Take a fine-tune that needs one H100 for 80 hours, priced at each listed rate.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Option&lt;/th&gt;
&lt;th&gt;Rate per GPU-hour&lt;/th&gt;
&lt;th&gt;80 GPU-hours&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Shadeform, lowest listing&lt;/td&gt;
&lt;td&gt;$1.66&lt;/td&gt;
&lt;td&gt;$132.80&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vast.ai H100 SXM median&lt;/td&gt;
&lt;td&gt;$1.73&lt;/td&gt;
&lt;td&gt;$138.40&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nodus H100 SXM&lt;/td&gt;
&lt;td&gt;$2.60&lt;/td&gt;
&lt;td&gt;$208.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runpod Community, from&lt;/td&gt;
&lt;td&gt;$2.69&lt;/td&gt;
&lt;td&gt;$215.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Shadeform, average listing&lt;/td&gt;
&lt;td&gt;$3.47&lt;/td&gt;
&lt;td&gt;$277.60&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runpod Secure&lt;/td&gt;
&lt;td&gt;$4.39&lt;/td&gt;
&lt;td&gt;$351.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Shadeform, highest listing&lt;/td&gt;
&lt;td&gt;$5.99&lt;/td&gt;
&lt;td&gt;$479.20&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each row is the rate times 80: for example $2.60 x 80 = $208.00 and $3.47 x 80 = $277.60. Against the average Shadeform listing, Nodus saves $69.60 on this run. Against the cheapest listing, Nodus costs $75.20 more, if that listing is live when you need it and the run never loses its machine.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdcn1mr8b5jhpizcm9arh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdcn1mr8b5jhpizcm9arh.png" alt="One NVIDIA H100 for 80 hours at listed rates: Shadeform lowest $132.80, Vast.ai median $138.40, Nodus $208.00, Runpod Community $215.20, Shadeform average $277.60, Runpod Secure $351.20, Shadeform highest $479.20." width="800" height="460"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;At list prices the cheapest VM wins a clean run; the average Shadeform listing costs 33% more than Nodus.&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  How much rework can a cheaper listing absorb before Nodus wins?
&lt;/h2&gt;

&lt;p&gt;The cheapest rate only stays cheapest while the run goes cleanly. When a VM is lost and your scripts restart from scratch, you pay again for the hours you redo. That gives a simple break-even rule we use when pricing runs:&lt;/p&gt;

&lt;p&gt;A listing at price p beats Nodus only if p x (1 + r) is below $2.60, where r is the share of the run you expect to redo.&lt;/p&gt;

&lt;p&gt;Solving for r gives the rework each Shadeform price tier can absorb. These assume the Nodus run redoes nothing, which is close to true for a checkpointed Job that loses only the minutes since its last checkpoint.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;$1.66 listing: 2.60 / 1.66 = 1.566, so it stays cheaper until you redo 56.6% of the run.&lt;/li&gt;
&lt;li&gt;$1.99 listing: 2.60 / 1.99 = 1.307, so the limit is 30.7%.&lt;/li&gt;
&lt;li&gt;$2.48 listing: 2.60 / 2.48 = 1.048, so the limit is 4.8%.&lt;/li&gt;
&lt;li&gt;$2.50 listing: 2.60 / 2.50 = 1.040, so the limit is 4.0%.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On the 80-hour run, 4.0% is 3.2 hours. Losing one machine late in the run without a good resume path wipes out the saving from a $2.50 listing. The $1.66 listing has far more room, which is why it is a fair pick for short, restartable work.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frbb8b8je5xfm16kgs7g6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frbb8b8je5xfm16kgs7g6.png" alt="Share of a run you can redo before a cheaper listing costs more than Nodus at $2.60 per H100-hour: $1.66 listing 56.6%, $1.99 listing 30.7%, $2.48 listing 4.8%, $2.50 listing 4.0%." width="800" height="328"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Below about $2 a GPU-hour, a listing can survive real rework; near $2.50, one bad restart erases the gap.&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  What happens on Nodus when a machine is lost mid-run?
&lt;/h2&gt;

&lt;p&gt;Nodus starts a new attempt somewhere else and restores your files. Its &lt;a href="https://www.nodus-compute.ai/docs/concepts/attempts-and-recovery/" rel="noopener noreferrer"&gt;attempts and recovery&lt;/a&gt; page describes the sequence. On a reclaim notice, Nodus asks your program for an urgent checkpoint while it prepares a replacement, which starts only once the old machine is provably gone. A machine that stops sending heartbeats gets a replacement prepared after 20 seconds and is declared lost after 60 seconds on shared capacity.&lt;/p&gt;

&lt;p&gt;The contract is files, not memory. Your program writes weights, optimizer state and its step counter to &lt;code&gt;/nodus/state&lt;/code&gt;, and the &lt;a href="https://www.nodus-compute.ai/docs/guides/checkpoints/" rel="noopener noreferrer"&gt;checkpoint guide&lt;/a&gt; shows the Job spec defaults:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;recovery&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;continuity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Checkpointed&lt;/span&gt;      &lt;span class="c1"&gt;# restore the latest checkpoint into each new attempt
&lt;/span&gt;  &lt;span class="na"&gt;checkpoint&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;paths&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;/nodus/state&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;       &lt;span class="c1"&gt;# up to 64 absolute paths, all saved in one checkpoint
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the work Shadeform's spot guide leaves to you: the polling, the relaunch and the restore. For a wider view of the same pattern across providers, see &lt;a href="https://www.nodus-compute.ai/blog/gpu-capacity-failover/" rel="noopener noreferrer"&gt;how GPU capacity failover works&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where is Shadeform the better choice?
&lt;/h2&gt;

&lt;p&gt;Shadeform is the better choice when you want a VM, not a managed run. If your team lives in SSH, tmux and its own scripts, a plain machine with no new job format is the shortest path. Shadeform gets you one from many providers without opening many accounts.&lt;/p&gt;

&lt;p&gt;It also wins on price discovery. One API call sorted by price across 30+ clouds is hard to beat for finding the cheapest live H100, and its lowest listing in this check, $1.66, is below the Nodus rate. Short jobs that restart cleanly can bank that difference.&lt;/p&gt;

&lt;p&gt;Three more cases favour it. You can link your own provider accounts and launch into them to spend existing credits. Shadeform offers reserved contracts, with a 12-month minimum term on its &lt;a href="https://shadeform.com/prices" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt;. And if you already use SkyPilot, Shadeform has a &lt;a href="https://docs.shadeform.ai/guides/skypilot" rel="noopener noreferrer"&gt;SkyPilot guide&lt;/a&gt; that adds its catalog as one more cloud.&lt;/p&gt;

&lt;h2&gt;
  
  
  Are SkyPilot, Vast.ai and Runpod better fits for some teams?
&lt;/h2&gt;

&lt;p&gt;Yes, each one fits a narrower buyer than Shadeform or Nodus.&lt;/p&gt;

&lt;p&gt;SkyPilot is the closest to Nodus in spirit. It is &lt;a href="https://github.com/skypilot-org/skypilot" rel="noopener noreferrer"&gt;open source under Apache 2.0&lt;/a&gt; and runs jobs on Kubernetes, Slurm, reserved GPUs or 20+ clouds that you hold accounts with. Its &lt;a href="https://docs.skypilot.co/en/latest/examples/managed-jobs.html" rel="noopener noreferrer"&gt;managed jobs&lt;/a&gt; recover from preemptions, node crashes and GPU errors, and your job can reload its latest checkpoint. You run the software and keep the provider bills.&lt;/p&gt;

&lt;p&gt;Vast.ai is the sticker-price option. Its &lt;a href="https://vast.ai/pricing" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt; lists the H100 SXM from $1.70 per hour, with a 30-day median of $1.73. Like a Shadeform VM, you handle restarts yourself. Our &lt;a href="https://www.nodus-compute.ai/blog/vast-ai-alternatives-2026/" rel="noopener noreferrer"&gt;Vast.ai alternatives&lt;/a&gt; post goes deeper.&lt;/p&gt;

&lt;p&gt;Runpod fits teams that want per-second container pods on one provider. Its &lt;a href="https://www.runpod.io/pricing" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt; lists the H100 SXM at $4.39 per hour on Secure Cloud and from $2.69 on Community Cloud. See our &lt;a href="https://www.nodus-compute.ai/blog/runpod-alternatives-2026/" rel="noopener noreferrer"&gt;Runpod alternatives&lt;/a&gt; guide for the trade-offs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Can you connect a Shadeform account to Nodus?
&lt;/h2&gt;

&lt;p&gt;No. Nodus's &lt;a href="https://www.nodus-compute.ai/docs/guides/pools/cloud-accounts/" rel="noopener noreferrer"&gt;cloud accounts docs&lt;/a&gt; list AWS, GCP, Azure and a set of API-key clouds such as Lambda, Nebius and Runpod, and Shadeform is not among them. If you hold credits with one of the listed clouds, you can connect that account directly and have Nodus spend those credits before Nodus capacity (Beta).&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you move a Shadeform workflow to Nodus?
&lt;/h2&gt;

&lt;p&gt;Keep your training script and point its saves at &lt;code&gt;/nodus/state&lt;/code&gt;. Then submit it as a Job from the project directory, with a dry run first. The commands come from the &lt;a href="https://www.nodus-compute.ai/docs/getting-started/install/" rel="noopener noreferrer"&gt;install&lt;/a&gt; and &lt;a href="https://www.nodus-compute.ai/docs/getting-started/quickstart/" rel="noopener noreferrer"&gt;quickstart&lt;/a&gt; pages:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install nodus-compute
nodus login
nodus run --name finetune --gpu H100 --image nodus/pytorch \
  --max-cost 25 --timeout 8h --dry-run -- python train.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The dry run uploads the directory and shows the estimate without starting anything. Remove &lt;code&gt;--dry-run&lt;/code&gt; to submit. &lt;code&gt;--max-cost&lt;/code&gt; sets a hard dollar limit for the Job, and &lt;code&gt;nodus describe job/finetune&lt;/code&gt; shows its attempts, placement and cost. Your SSH key, startup script and polling loop go away; the state directory replaces them.&lt;/p&gt;

&lt;p&gt;If you want to compare GPU types before committing, a &lt;a href="https://www.nodus-compute.ai/docs/guides/sweeps/" rel="noopener noreferrer"&gt;Sweep&lt;/a&gt; runs the same Job across several GPUs and reports the cheapest and fastest cell. Our post on &lt;a href="https://www.nodus-compute.ai/blog/run-training-across-multiple-gpu-clouds/" rel="noopener noreferrer"&gt;running one training job across several GPU clouds&lt;/a&gt; covers the wider design.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Does Shadeform have more GPUs than going to a provider directly?
&lt;/h3&gt;

&lt;p&gt;No. Its FAQ says it taps each provider's on-demand pool, so what you see on Shadeform is what you would see at the provider. What it adds is one search, one account and one bill across them.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does Nodus work the same way as Shadeform?
&lt;/h3&gt;

&lt;p&gt;No. Shadeform lists VMs for you to launch and manage, while Nodus runs the work itself. You submit a Job, Function or TrainingJob, Nodus picks the capacity, and recovery and billing are per run.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do Shadeform spot instances resume automatically?
&lt;/h3&gt;

&lt;p&gt;No. Its spot guide says a reclaimed instance is terminated and marked &lt;code&gt;preempted&lt;/code&gt;, and billing stops. Restarting it, on spot or on-demand, is up to your own polling and relaunch code.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can an AI agent buy GPU time through Shadeform or Nodus?
&lt;/h3&gt;

&lt;p&gt;Both have a hosted MCP server. Shadeform's lets Claude Code, Cursor or Codex find and manage instances. The Nodus server creates resources only after an accepted dry run and runs under the person's own projects and spending limits.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which data centers do Shadeform GPUs run in?
&lt;/h3&gt;

&lt;p&gt;Shadeform's FAQ says it guarantees all GPUs are in Tier 3 or Tier 4 data centers. Machine SLAs are the underlying provider's, which Shadeform shares on request.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.nodus-compute.ai/blog/shadeform-alternatives/" rel="noopener noreferrer"&gt;Nodus blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>gpu</category>
      <category>cloud</category>
      <category>machinelearning</category>
      <category>ai</category>
    </item>
    <item>
      <title>How do you host your own LLM behind an OpenAI-style API?</title>
      <dc:creator>Viswaretas Kotra</dc:creator>
      <pubDate>Wed, 18 Nov 2026 16:45:43 +0000</pubDate>
      <link>https://dev.to/viswaretas_kotra/how-do-you-host-your-own-llm-behind-an-openai-style-api-4pm6</link>
      <guid>https://dev.to/viswaretas_kotra/how-do-you-host-your-own-llm-behind-an-openai-style-api-4pm6</guid>
      <description>&lt;p&gt;To host your own LLM behind an OpenAI-style API, run vLLM's server (&lt;code&gt;vllm serve &amp;lt;model&amp;gt;&lt;/code&gt;) on a GPU that holds the weights and cache, require an API key behind a proxy, and point your OpenAI client at the new &lt;code&gt;base_url&lt;/code&gt;. On Nodus you can skip the server with a named inference endpoint for a catalog open model, or run &lt;code&gt;vllm serve&lt;/code&gt; with your own weights as a Job on a GPU such as an L4 listed at $0.430/hr. Use a managed endpoint when a catalog model passes your evals, and run vLLM yourself only when you need your own weights or engine settings.&lt;/p&gt;

&lt;p&gt;This guide gives the vLLM commands, the API key gap most tutorials skip, monthly GPU costs for four model sizes, and both routes on Nodus. We have not run every command here end to end; where a step is assembled from docs rather than tested by us, we say so.&lt;/p&gt;

&lt;h2&gt;
  
  
  Can you host your own LLM?
&lt;/h2&gt;

&lt;p&gt;Yes, if three things line up: the model's weights are downloadable, its license allows your use, and you have a GPU that holds it. Open-weight families such as Llama, Qwen and gpt-oss meet the first condition. Licenses differ by model, so read the model card before you ship.&lt;/p&gt;

&lt;p&gt;The GPU is the part people get wrong. A model needs memory for its weights plus a KV cache that grows with every concurrent request. Our guide to &lt;a href="https://www.nodus-compute.ai/blog/how-to-calculate-llm-vram/" rel="noopener noreferrer"&gt;calculating LLM VRAM&lt;/a&gt; has the full formula and worked numbers, and we only use its results below.&lt;/p&gt;

&lt;p&gt;You also need a serving engine. vLLM is the common choice because it ships an HTTP server that speaks the OpenAI API, so clients written for OpenAI work once you change their base URL. If you want to know why it batches requests so well, read &lt;a href="https://www.nodus-compute.ai/blog/what-is-vllm/" rel="noopener noreferrer"&gt;what vLLM is&lt;/a&gt;; this post is about getting it online.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you start vLLM's OpenAI-compatible server?
&lt;/h2&gt;

&lt;p&gt;Install vLLM on a Linux machine with an NVIDIA GPU, then start the server with one command. The &lt;a href="https://docs.vllm.ai/en/latest/getting_started/quickstart/" rel="noopener noreferrer"&gt;vLLM quickstart&lt;/a&gt; installs it with uv:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;uv venv --python 3.12 --seed
&lt;span class="nb"&gt;source&lt;/span&gt; .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then serve a model:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;vllm serve Qwen/Qwen2.5-1.5B-Instruct
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The quickstart says the server starts at &lt;code&gt;http://localhost:8000&lt;/code&gt; by default and accepts &lt;code&gt;--host&lt;/code&gt; and &lt;code&gt;--port&lt;/code&gt; to change that. Check it with a request:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl http://localhost:8000/v1/chat/completions \
    -H &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; \
    -d &lt;span class="s1"&gt;'{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Who won the world series in 2020?"}
        ]
    }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you prefer containers, the &lt;a href="https://docs.vllm.ai/en/latest/deployment/docker/" rel="noopener noreferrer"&gt;vLLM Docker guide&lt;/a&gt; runs the prebuilt &lt;code&gt;vllm/vllm-openai&lt;/code&gt; image. The image's start command is the server, so the model goes in as an argument:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env &lt;span class="s2"&gt;"HF_TOKEN=$HF_TOKEN"&lt;/span&gt; \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-0.6B
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The mounted cache keeps downloaded weights between restarts. &lt;code&gt;HF_TOKEN&lt;/code&gt; is only needed for gated models such as Llama.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you add an API key, and is it enough?
&lt;/h2&gt;

&lt;p&gt;Pass &lt;code&gt;--api-key&lt;/code&gt; or set the &lt;code&gt;VLLM_API_KEY&lt;/code&gt; environment variable, and the server checks the bearer token on its OpenAI routes. The quickstart notes that you can pass several keys after &lt;code&gt;--api-key&lt;/code&gt;, which helps when you rotate them.&lt;/p&gt;

&lt;p&gt;That key does not protect the whole server. vLLM's &lt;a href="https://docs.vllm.ai/en/latest/usage/security/" rel="noopener noreferrer"&gt;security guide&lt;/a&gt; says &lt;code&gt;--api-key&lt;/code&gt; only covers paths under &lt;code&gt;/v1&lt;/code&gt;, &lt;code&gt;/v2&lt;/code&gt;, &lt;code&gt;/inference&lt;/code&gt; and &lt;code&gt;/cohere&lt;/code&gt;. Other routes on the same port, including an &lt;code&gt;/invocations&lt;/code&gt; path that reaches the same inference functions as &lt;code&gt;/v1&lt;/code&gt;, answer without a key. The guide's own advice is not to rely on &lt;code&gt;--api-key&lt;/code&gt; alone.&lt;/p&gt;

&lt;p&gt;What to do instead:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Bind vLLM to localhost or a private network, never straight to the internet.&lt;/li&gt;
&lt;li&gt;Put a reverse proxy in front that forwards only the &lt;code&gt;/v1/...&lt;/code&gt; routes your clients use and terminates TLS.&lt;/li&gt;
&lt;li&gt;Keep the key in a secret store or environment variable, not in a command line that ends up in shell history.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  How do you switch an existing OpenAI client to your server?
&lt;/h2&gt;

&lt;p&gt;Change the &lt;code&gt;base_url&lt;/code&gt; and the model name; the rest of your code stays the same. This is the client from the vLLM quickstart, with the key you set on the server in place of its &lt;code&gt;"EMPTY"&lt;/code&gt; placeholder:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;api_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"YOUR_VLLM_KEY"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;base_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"http://localhost:8000/v1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;chat_response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"Qwen/Qwen2.5-1.5B-Instruct"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"system"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"You are a helpful assistant."&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Tell me a joke."&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chat_response&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details trip people up. The &lt;code&gt;model&lt;/code&gt; field must match the name the server loaded, which &lt;code&gt;GET /v1/models&lt;/code&gt; returns. And the URL ends in &lt;code&gt;/v1&lt;/code&gt;, because the SDK appends &lt;code&gt;/chat/completions&lt;/code&gt; itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which GPU do you need, and what does it cost to keep it on?
&lt;/h2&gt;

&lt;p&gt;Pick the smallest GPU class whose usable memory covers the weights plus the cache you need, then multiply its hourly price by the hours the API stays up. The fit for each model below comes from the GPU-class table in our &lt;a href="https://www.nodus-compute.ai/blog/how-to-calculate-llm-vram/" rel="noopener noreferrer"&gt;VRAM guide&lt;/a&gt;. We priced the cheapest GPU the Nodus pricing page lists in each class on two schedules. Always on is 730 hours, an average month; business hours are 10 hours a day for 22 days, or 220 hours.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model and precision&lt;/th&gt;
&lt;th&gt;GPU class&lt;/th&gt;
&lt;th&gt;Cheapest listed GPU in class&lt;/th&gt;
&lt;th&gt;Always on, 730 h&lt;/th&gt;
&lt;th&gt;Business hours, 220 h&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Llama 3.1 8B, BF16&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;td&gt;RTX 3090, $0.180/hr&lt;/td&gt;
&lt;td&gt;$131.40&lt;/td&gt;
&lt;td&gt;$39.60&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-32B, INT4&lt;/td&gt;
&lt;td&gt;48 GB&lt;/td&gt;
&lt;td&gt;RTX A6000, $0.333/hr&lt;/td&gt;
&lt;td&gt;$243.09&lt;/td&gt;
&lt;td&gt;$73.26&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama 3.3 70B, INT4&lt;/td&gt;
&lt;td&gt;80 GB&lt;/td&gt;
&lt;td&gt;A100 80 GB, $1.090/hr&lt;/td&gt;
&lt;td&gt;$795.70&lt;/td&gt;
&lt;td&gt;$239.80&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama 3.3 70B, FP8&lt;/td&gt;
&lt;td&gt;141 GB&lt;/td&gt;
&lt;td&gt;H200 SXM, $3.593/hr&lt;/td&gt;
&lt;td&gt;$2,622.89&lt;/td&gt;
&lt;td&gt;$790.46&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Prices checked November 18, 2026, on the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing page&lt;/a&gt;. The arithmetic is price x hours: for example, $1.090 x 730 = $795.70 and $1.090 x 220 = $239.80.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ff2lrsxnhz996jx9pd28p.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ff2lrsxnhz996jx9pd28p.png" alt="Monthly cost of one always-on GPU at Nodus listed prices: Llama 3.1 8B on an RTX 3090 $131.40, Qwen3-32B INT4 on an RTX A6000 $243.09, Llama 3.3 70B INT4 on an A100 80 GB $795.70, Llama 3.3 70B FP8 on an H200 $2,622.89." width="800" height="328"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Model size sets the bill: the same API costs 20 times more per month at 70B FP8 than at 8B.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The cheapest card in a class is not always the right one. The RTX 3090 and RTX A6000 are older consumer and workstation cards. Data-center options in the same classes cost more, such as the L4 at $0.430/hr and the L40S at $0.793/hr, so a busy production endpoint may justify the step up. The 70B rows also show the precision trade: FP8 needs the 141 GB class and costs over three times as much as INT4 on an 80 GB card.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw3l90gcgz2hze3dt0ask.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw3l90gcgz2hze3dt0ask.png" alt="Always-on versus business-hours monthly cost: 8B $131.40 vs $39.60, 32B INT4 $243.09 vs $73.26, 70B INT4 $795.70 vs $239.80, 70B FP8 $2,622.89 vs $790.46." width="800" height="416"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Turning the server off outside working hours cuts each bill by about 70%, if nobody needs it at night.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;These are GPU costs only. They leave out the engineer who keeps the server patched and the idle hours when it serves nothing, which is why "can I afford the GPU" is the wrong question. Our &lt;a href="https://www.nodus-compute.ai/blog/self-hosting-llm-vs-api-cost/" rel="noopener noreferrer"&gt;self-hosting vs API cost&lt;/a&gt; post works out how many tokens a day a GPU must serve before it beats a per-token API.&lt;/p&gt;
&lt;h2&gt;
  
  
  How do you host an LLM API on Nodus?
&lt;/h2&gt;

&lt;p&gt;There are two routes on Nodus, and the choice depends on whose weights you serve.&lt;/p&gt;

&lt;p&gt;If a model in the Nodus catalog does the job, create a named inference endpoint. The &lt;a href="https://www.nodus-compute.ai/docs/guides/inference/" rel="noopener noreferrer"&gt;Nodus inference guide&lt;/a&gt; marks this feature Beta. An endpoint gives one model its own base URL, a list of allowed API keys and an optional dollar cap, and there is no server to run. The commands, exactly as the docs show them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;$ nodus create inferenceendpoint support-bot --model nodus/gpt-oss-120b --max-cost 50
$ nodus &lt;span class="nb"&gt;wait&lt;/span&gt; ep/support-bot --for=jsonpath=&lt;span class="o"&gt;{&lt;/span&gt;.status.phase&lt;span class="o"&gt;}&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;Running
$ nodus get ep
$ nodus inference chat --endpoint support-bot &lt;span class="s2"&gt;"Where is my order?"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Your OpenAI client then uses &lt;code&gt;https://inference.nodus-compute.ai/endpoints/&amp;lt;project&amp;gt;/support-bot/v1&lt;/code&gt; as its &lt;code&gt;base_url&lt;/code&gt;. A key created for an endpoint can call only that endpoint, and once its requests have spent the cap, the next one gets &lt;code&gt;402 BudgetExceeded&lt;/code&gt;. That covers the key and spending problems from the vLLM section without a proxy.&lt;/p&gt;

&lt;p&gt;If you need your own weights, a fine-tune or specific engine flags, run &lt;code&gt;vllm serve&lt;/code&gt; as a Nodus Job. The &lt;a href="https://www.nodus-compute.ai/docs/getting-started/quickstart/" rel="noopener noreferrer"&gt;quickstart&lt;/a&gt; and &lt;a href="https://www.nodus-compute.ai/docs/reference/cli/nodus_run/" rel="noopener noreferrer"&gt;&lt;code&gt;nodus run&lt;/code&gt; reference&lt;/a&gt; document every flag below, the &lt;a href="https://www.nodus-compute.ai/docs/guides/images/" rel="noopener noreferrer"&gt;images guide&lt;/a&gt; allows public registry images such as &lt;code&gt;vllm/vllm-openai&lt;/code&gt;, and the &lt;a href="https://www.nodus-compute.ai/docs/guides/secrets/" rel="noopener noreferrer"&gt;secrets guide&lt;/a&gt; passes secret keys in as environment variables. We assembled this from those pages and have not run this exact combination ourselves:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;nodus secret create hf-token --from-literal &lt;span class="nv"&gt;HF_TOKEN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;hf_xxx
nodus secret create vllm-key --from-literal &lt;span class="nv"&gt;VLLM_API_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;your-key
nodus run --name vllm-api --gpu L4 --image vllm/vllm-openai:latest \
  --secret hf-token --secret vllm-key --no-source \
  --max-cost 5 --timeout 8h -d -- vllm serve Qwen/Qwen2.5-1.5B-Instruct
nodus logs -f job/vllm-api
nodus port-forward job/vllm-api 8000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because vLLM reads &lt;code&gt;VLLM_API_KEY&lt;/code&gt; from the environment, the key never appears in the command. &lt;code&gt;nodus port-forward&lt;/code&gt; listens on &lt;code&gt;127.0.0.1&lt;/code&gt; by default, so your OpenAI client keeps &lt;code&gt;http://localhost:8000/v1&lt;/code&gt; while the GPU runs on Nodus. Eight hours of an L4 at $0.430/hr is $3.44, inside the $5 cap. Check the first lines of &lt;code&gt;nodus logs&lt;/code&gt; to confirm the server started before you send traffic.&lt;/p&gt;

&lt;p&gt;That port-forward suits development, evals and internal tools. Nodus Functions do not expose public web endpoints; the &lt;a href="https://www.nodus-compute.ai/docs/getting-started/modal-users/" rel="noopener noreferrer"&gt;Modal migration guide&lt;/a&gt; says to use an inference endpoint for model serving. For a public, always-on API on your own weights, Nodus offers dedicated capacity plans through the inference console's Contact us option.&lt;/p&gt;

&lt;h2&gt;
  
  
  When should you not host your own LLM API?
&lt;/h2&gt;

&lt;p&gt;Do not host your own when a per-token API serves the same model and your traffic is low or bursty. You pay for every hour the GPU is up, busy or not, while a per-token API charges only for tokens. Our post on &lt;a href="https://www.nodus-compute.ai/blog/llm-inference-cost-per-token/" rel="noopener noreferrer"&gt;LLM inference cost per token&lt;/a&gt; shows how wide those per-token prices range.&lt;/p&gt;

&lt;p&gt;The decision rule we use:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Try the model on a per-token API or a Nodus endpoint first. If it passes your evals and the bill is fine, stop there.&lt;/li&gt;
&lt;li&gt;Host vLLM yourself when you need weights the catalog does not have, engine settings an API will not expose, or data that must stay on hardware you control.&lt;/li&gt;
&lt;li&gt;Keep the GPU on 24/7 only when traffic fills it across the day. Otherwise run it on a schedule or as a Job you start when work arrives.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Can one vLLM server host several models?
&lt;/h3&gt;

&lt;p&gt;No. The vLLM quickstart says the server hosts one model at a time. To offer two models, run two servers on separate ports or GPUs and route between them in your proxy by the &lt;code&gt;model&lt;/code&gt; field.&lt;/p&gt;

&lt;h3&gt;
  
  
  What if my model does not fit on one GPU?
&lt;/h3&gt;

&lt;p&gt;Split it across GPUs on one machine with vLLM's &lt;code&gt;--tensor-parallel-size&lt;/code&gt; option from its &lt;a href="https://docs.vllm.ai/en/stable/configuration/engine_args/" rel="noopener noreferrer"&gt;engine arguments&lt;/a&gt;, or quantize it to the next GPU class down. On Nodus, &lt;code&gt;--gpu H100:8&lt;/code&gt; asks for eight GPUs in one Job.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do you retry a request on a Nodus endpoint without paying twice?
&lt;/h3&gt;

&lt;p&gt;Send an &lt;code&gt;Idempotency-Key&lt;/code&gt; header. The inference guide says a repeat within 24 hours returns the same answer with no second charge, and a request that failed without a charge frees the key so the retry runs again.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does streaming work the same as OpenAI?
&lt;/h3&gt;

&lt;p&gt;For the Nodus inference API, yes: its guide says streaming works as in the OpenAI API with &lt;code&gt;stream: true&lt;/code&gt;, and completed streams end with &lt;code&gt;data: [DONE]&lt;/code&gt;. vLLM's &lt;a href="https://docs.vllm.ai/en/latest/serving/online_serving/" rel="noopener noreferrer"&gt;online serving docs&lt;/a&gt; include streaming chat examples for its server, so test your own client against it once before switching.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.nodus-compute.ai/blog/host-your-own-llm-api/" rel="noopener noreferrer"&gt;Nodus blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>api</category>
      <category>llm</category>
      <category>vllm</category>
      <category>selfhosted</category>
    </item>
    <item>
      <title>What is a serverless GPU, and when should you use one?</title>
      <dc:creator>Viswaretas Kotra</dc:creator>
      <pubDate>Wed, 18 Nov 2026 02:34:08 +0000</pubDate>
      <link>https://dev.to/viswaretas_kotra/what-is-a-serverless-gpu-and-when-should-you-use-one-1960</link>
      <guid>https://dev.to/viswaretas_kotra/what-is-a-serverless-gpu-and-when-should-you-use-one-1960</guid>
      <description>&lt;p&gt;A serverless GPU is one you never provision: the platform starts your container when work arrives, bills while it runs, and scales to zero when traffic stops. Nodus Functions work this way, starting workers when calls arrive and scaling them back down, while long training runs go to Nodus TrainingJobs, which checkpoint and resume instead of hitting a call timeout. Use serverless for bursty inference, short batch jobs and agent tool calls, and an always-on GPU or a training job once the GPU would be busy most of the day.&lt;/p&gt;

&lt;h2&gt;
  
  
  How does a serverless GPU actually work?
&lt;/h2&gt;

&lt;p&gt;You hand the platform a container image and a function, and it handles the machines. When a call comes in and no worker is free, it places a worker on a GPU, pulls the image, loads your code and runs the call. That wait is the cold start.&lt;/p&gt;

&lt;p&gt;After the call, the worker stays warm for a short idle window in case another call arrives. If none does, it is released and you stop paying.&lt;/p&gt;

&lt;p&gt;On Nodus that window is &lt;code&gt;scaledown_window&lt;/code&gt;, one minute by default and up to 20 minutes, as the &lt;a href="https://www.nodus-compute.ai/docs/guides/functions/scaling/" rel="noopener noreferrer"&gt;Functions scaling guide&lt;/a&gt; describes. Modal keeps containers alive for 60 seconds after the last input by default and bills for that time, according to its &lt;a href="https://modal.com/pricing" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt;. Runpod Serverless waits 5 seconds by default before scaling a worker down, per its &lt;a href="https://docs.runpod.io/serverless/pricing" rel="noopener noreferrer"&gt;serverless pricing docs&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Four settings decide most of the behavior on any platform:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Setting&lt;/th&gt;
&lt;th&gt;What it controls&lt;/th&gt;
&lt;th&gt;Nodus Functions default&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Minimum workers&lt;/td&gt;
&lt;td&gt;Workers kept warm with no traffic (0 means scale to zero)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;min_workers&lt;/code&gt; 0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Maximum workers&lt;/td&gt;
&lt;td&gt;The ceiling on parallel GPUs&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;max_workers&lt;/code&gt; 10 (1 to 1,000)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Idle window&lt;/td&gt;
&lt;td&gt;How long a finished worker waits before release&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;scaledown_window&lt;/code&gt; 1m&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Call timeout&lt;/td&gt;
&lt;td&gt;The longest one call may run&lt;/td&gt;
&lt;td&gt;5 minutes, 24 hours at most&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Defaults from the &lt;a href="https://www.nodus-compute.ai/docs/guides/functions/" rel="noopener noreferrer"&gt;Nodus Functions guide&lt;/a&gt; and &lt;a href="https://www.nodus-compute.ai/docs/guides/functions/scaling/" rel="noopener noreferrer"&gt;scaling guide&lt;/a&gt;, checked November 17, 2026.&lt;/p&gt;

&lt;h2&gt;
  
  
  What exactly do you pay for?
&lt;/h2&gt;

&lt;p&gt;You pay for worker time, not for requests, and worker time includes more than your code running. Runpod's docs list three billed phases: the cold start until your handler is ready, execution, and the idle wait after the last job. Runpod bills from worker start to full stop, rounded up to the nearest second, and does not bill image pulls that happen while the worker is scaled down (&lt;a href="https://docs.runpod.io/serverless/pricing" rel="noopener noreferrer"&gt;Runpod serverless pricing&lt;/a&gt;). Modal bills during application load time and while inputs are processed, per second (&lt;a href="https://modal.com/pricing" rel="noopener noreferrer"&gt;Modal pricing&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;On Nodus, the estimate tells you the rate and the start times before you deploy. Calling &lt;code&gt;embed.estimate("hello")&lt;/code&gt; returns the expected cost, the cold and warm start bands and the worker's hourly rate, and every finished call reports &lt;code&gt;status.costUSD&lt;/code&gt;, the worker time attributed to it (&lt;a href="https://www.nodus-compute.ai/docs/guides/functions/calls/" rel="noopener noreferrer"&gt;Calling Functions&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;So a serverless bill is busy time plus overhead. If each call loads a large model from scratch, the overhead can exceed the work.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is the difference between a request and a job on a serverless GPU?
&lt;/h2&gt;

&lt;p&gt;A request is a call someone is waiting on, and a job is work you start and collect later. On Nodus, &lt;code&gt;.remote()&lt;/code&gt; blocks until the result comes back, &lt;code&gt;.spawn()&lt;/code&gt; starts a call and returns a handle, and &lt;code&gt;.map()&lt;/code&gt; fans one Function out over many inputs and returns results in input order (&lt;a href="https://www.nodus-compute.ai/docs/guides/functions/" rel="noopener noreferrer"&gt;Functions guide&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;The difference matters for timeouts. Synchronous requests should finish in seconds, and jobs can run longer, but every platform caps a single call.&lt;/p&gt;

&lt;p&gt;A Nodus Function call that runs past its &lt;code&gt;timeout&lt;/code&gt; ends as &lt;code&gt;Failed&lt;/code&gt; with &lt;code&gt;DeadlineExceeded&lt;/code&gt;. Modal's default execution timeout is 300 seconds and can be raised to 24 hours (&lt;a href="https://modal.com/docs/guide/timeouts" rel="noopener noreferrer"&gt;Modal timeouts&lt;/a&gt;). Runpod's console default is 600 seconds, with a range of 5 seconds to 7 days (&lt;a href="https://docs.runpod.io/serverless/endpoints/endpoint-configurations" rel="noopener noreferrer"&gt;Runpod endpoint settings&lt;/a&gt;).&lt;/p&gt;

&lt;h2&gt;
  
  
  Why are cold starts the real limit for inference?
&lt;/h2&gt;

&lt;p&gt;Cold starts, not price, are what make serverless awkward for latency-sensitive endpoints. Most of a GPU cold start is loading model weights into GPU memory, and that grows with the model.&lt;/p&gt;

&lt;p&gt;Nodus documents one measurement on an NVIDIA A10. A vLLM engine serving Qwen3-0.6B took about 90 seconds to cold start, and about 6 seconds to restore from a GPU snapshot with its weights and KV cache kept out (&lt;a href="https://www.nodus-compute.ai/docs/guides/functions/gpu-snapshots/" rel="noopener noreferrer"&gt;GPU snapshots (Beta)&lt;/a&gt;). Snapshots are in Beta. If your users cannot wait a minute and a half on the first request after a quiet period, you either keep a worker warm with &lt;code&gt;min_workers=1&lt;/code&gt; or make that restore path part of the design.&lt;/p&gt;

&lt;p&gt;Keeping one worker warm trades latency for an always-on bill. The Nodus scaling guide shows the pattern:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nd"&gt;@app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;function&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cpu&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;memory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"4Gi"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;min_workers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_workers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;scaledown_window&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"5m"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;target_concurrency&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That keeps one worker up, adds one worker for every four calls waiting or running, and holds idle extras for five minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  When does serverless cost less than an always-on GPU?
&lt;/h2&gt;

&lt;p&gt;Serverless wins when the GPU would sit idle most of the day, and loses once it is busy more than the ratio of the two prices. That gives a rule you can check against any price list.&lt;/p&gt;

&lt;p&gt;Break-even busy fraction = always-on hourly price / (serverless hourly price x (1 + overhead))&lt;/p&gt;

&lt;p&gt;Overhead is the billed cold start and idle time on top of real work, as a fraction of busy time. We derived this rule from each vendor's published billing model; it ignores storage and egress.&lt;/p&gt;

&lt;p&gt;Runpod makes a clean worked example because it sells both models for the H100. Its &lt;a href="https://www.runpod.io/pricing" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt; lists an H100 PRO serverless flex worker at $5.89/hr and an H100 SXM Pod on Secure Cloud at $4.39/hr.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Bursty, 4 busy hours a day: serverless costs 4 x $5.89 = $23.56. An always-on Pod costs 24 x $4.39 = $105.36.&lt;/li&gt;
&lt;li&gt;Steady, 20 busy hours a day: serverless costs 20 x $5.89 = $117.80, more than the Pod's $105.36.&lt;/li&gt;
&lt;li&gt;Break-even with zero overhead: $4.39 / $5.89 = 74.5% busy, or about 17.9 hours a day.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffulaaf7ckw4ex3yx2r3p.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffulaaf7ckw4ex3yx2r3p.png" alt="On Runpod, Serverless flex costs $23.56 a day at 4 busy hours and $117.80 at 20 busy hours, while an always-on H100 Pod costs $105.36 a day." width="800" height="416"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Under about 18 busy hours a day, paying only for busy time is cheaper even at a higher hourly rate.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Overhead pulls the break-even down. If cold starts and idle tails add 25% to busy time, break-even falls to 74.5% / 1.25 = 59.6%, about 14.3 hours a day. At 50% overhead it is 74.5% / 1.5 = 49.7%, about 11.9 hours. These overhead levels are illustrations, not measurements; read your own from the platform's per-call cost or usage records.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqf76lrr48ztmh96i5fpy.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqf76lrr48ztmh96i5fpy.png" alt="Break-even busy hours per day for Runpod Serverless versus an always-on Pod: 17.9 hours at 0% overhead, 14.3 hours at 25% and 11.9 hours at 50%." width="800" height="304"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Slow cold starts make serverless stop paying off sooner, so shrink load time before you compare prices.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The rule changes shape when a platform charges the same rate for warm and busy workers. Modal lists an NVIDIA H100 at $0.001097 per second, which is about $3.95/hr, whether a container is serving or held warm (&lt;a href="https://modal.com/pricing" rel="noopener noreferrer"&gt;Modal pricing&lt;/a&gt;). On Nodus, an idle warm worker is billed at the Function's worker rate, the same as a busy one, and the H100 SXM is listed at $2.60/hr on the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt;. When warm and busy time cost the same, scale to zero never costs more for the same busy time; the trade is cold-start latency against the cost of &lt;code&gt;min_workers&lt;/code&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Option&lt;/th&gt;
&lt;th&gt;H100 list price&lt;/th&gt;
&lt;th&gt;How idle time is billed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Nodus&lt;/td&gt;
&lt;td&gt;$2.60/hr&lt;/td&gt;
&lt;td&gt;Warm idle window billed, released after &lt;code&gt;scaledown_window&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Modal&lt;/td&gt;
&lt;td&gt;$3.95/hr ($0.001097/s)&lt;/td&gt;
&lt;td&gt;Idle window billed, 60 s by default&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runpod Pod, Secure Cloud&lt;/td&gt;
&lt;td&gt;$4.39/hr&lt;/td&gt;
&lt;td&gt;Always billed while running&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runpod Serverless flex&lt;/td&gt;
&lt;td&gt;$5.89/hr&lt;/td&gt;
&lt;td&gt;Idle timeout billed, 5 s by default&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Sources: &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing&lt;/a&gt;, &lt;a href="https://modal.com/pricing" rel="noopener noreferrer"&gt;Modal pricing&lt;/a&gt;, &lt;a href="https://www.runpod.io/pricing" rel="noopener noreferrer"&gt;Runpod pricing&lt;/a&gt;, &lt;a href="https://docs.runpod.io/serverless/pricing" rel="noopener noreferrer"&gt;Runpod serverless docs&lt;/a&gt;. Prices checked November 17, 2026.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F417yuztf00qgb8k72wi8.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F417yuztf00qgb8k72wi8.png" alt="H100 list prices per hour: Nodus $2.60, Modal $3.95, Runpod Pod $4.39 and Runpod Serverless flex $5.89." width="800" height="328"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The serverless premium varies by platform, and on some it is zero.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;For a ranked look at platforms, see &lt;a href="https://www.nodus-compute.ai/blog/best-serverless-gpu-platforms-2026/" rel="noopener noreferrer"&gt;the best serverless GPU platforms in 2026&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;
  
  
  When is serverless the wrong choice?
&lt;/h2&gt;

&lt;p&gt;Serverless is the wrong tool for work that keeps a GPU busy for hours. Steady high-utilization serving crosses the break-even above, and long training runs into the call timeout. A 30-hour fine-tune does not fit in a 24-hour call, and a call that fails partway has no built-in progress to resume.&lt;/p&gt;

&lt;p&gt;On Nodus that work belongs in a Job or a TrainingJob. A TrainingJob runs a managed runtime such as &lt;code&gt;nodus/sft&lt;/code&gt; or &lt;code&gt;nodus/grpo-lora&lt;/code&gt;, prices itself in a dry run, takes a &lt;code&gt;maxCostUSD&lt;/code&gt; cap, and recovers from full checkpoints with optimizer, scheduler and RNG state (&lt;a href="https://www.nodus-compute.ai/docs/guides/training/" rel="noopener noreferrer"&gt;Training (Beta)&lt;/a&gt;). We cover the long-run case in detail in &lt;a href="https://www.nodus-compute.ai/blog/serverless-gpu-for-training/" rel="noopener noreferrer"&gt;can you use serverless GPUs for long training runs&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;A middle case is worth knowing. A Nodus GPU class that scales to zero leaves its GPU idle between workers, and with gap-fill (Beta) a checkpointed one-GPU Job from your organization can use that idle time. On an A10, such a Job kept 91% of a dedicated GPU's steps at one request a minute and 34% at eight, per the &lt;a href="https://www.nodus-compute.ai/docs/guides/functions/gpu-snapshots/" rel="noopener noreferrer"&gt;GPU snapshots guide&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;
  
  
  Which Nodus feature should you pick?
&lt;/h2&gt;

&lt;p&gt;Pick by how long the work runs and whether someone waits on it. The &lt;a href="https://www.nodus-compute.ai/docs/guides/choose-compute/" rel="noopener noreferrer"&gt;compute feature guide&lt;/a&gt; maps the full set; for GPU work it comes down to this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Calls in seconds to minutes, bursty or fanned out over many inputs: Functions, with &lt;code&gt;min_workers=0&lt;/code&gt; unless latency demands a warm worker.&lt;/li&gt;
&lt;li&gt;A script or batch that should run to completion: a Job.&lt;/li&gt;
&lt;li&gt;Fine-tuning, preference training, distillation or RL over hours or days: a TrainingJob.&lt;/li&gt;
&lt;li&gt;Calling a hosted open-weight model rather than serving your own: Inference (Beta).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To start, install the SDK and sign in, as the docs show:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;$ pip install nodus-compute
$ nodus login
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Is a serverless GPU the same as a spot GPU?
&lt;/h3&gt;

&lt;p&gt;No. Serverless describes how you are billed and scaled, per unit of worker time with scale to zero. Spot describes capacity that can be reclaimed at a discount. A platform can combine them, so check both before you plan a long run.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does scale to zero mean my model reloads on every request?
&lt;/h3&gt;

&lt;p&gt;Only on requests that arrive after the idle window has closed. Calls inside the window reuse the warm worker, which is why the idle window is the cheapest latency control you have.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I measure my real overhead?
&lt;/h3&gt;

&lt;p&gt;Compare billed worker time with time spent inside your handler over a day of real traffic. On Nodus, each call's &lt;code&gt;status.costUSD&lt;/code&gt; and the usage records give the billed side.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can agents use serverless GPUs?
&lt;/h3&gt;

&lt;p&gt;Yes, and agent tool calls are a good fit because they are short and irregular. A coding agent such as Claude Code can also start Nodus work through MCP after &lt;code&gt;nodus mcp install&lt;/code&gt;, following the &lt;a href="https://www.nodus-compute.ai/connect/" rel="noopener noreferrer"&gt;connect guide&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.nodus-compute.ai/blog/what-is-serverless-gpu/" rel="noopener noreferrer"&gt;Nodus blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>serverless</category>
      <category>gpu</category>
      <category>machinelearning</category>
      <category>mlops</category>
    </item>
    <item>
      <title>What are the best fal.ai alternatives for model inference?</title>
      <dc:creator>Viswaretas Kotra</dc:creator>
      <pubDate>Wed, 18 Nov 2026 00:01:14 +0000</pubDate>
      <link>https://dev.to/viswaretas_kotra/what-are-the-best-falai-alternatives-for-model-inference-26i0</link>
      <guid>https://dev.to/viswaretas_kotra/what-are-the-best-falai-alternatives-for-model-inference-26i0</guid>
      <description>&lt;p&gt;Which fal.ai alternative fits comes down to how you use fal: calling its hosted image, video and audio models, or running your own model on fal Serverless. If you deploy your own open-weight model, Nodus lists the NVIDIA H100 at $2.60/hr on its &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt;, against fal's $4.50 list price, and Nodus Functions scale to zero between bursts. If you call hosted generative media models by the image or by the second of video, Replicate is the closest swap, and fal is often still the right place to stay.&lt;/p&gt;

&lt;p&gt;We build Nodus, so read this with that in mind. Every competitor fact links to that company's own pricing page or docs, the arithmetic is shown once, and one section covers where fal is the better choice.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does fal.ai actually sell?
&lt;/h2&gt;

&lt;p&gt;fal sells two different things, and the right alternative differs for each. Its &lt;a href="https://fal.ai/docs/documentation/serverless/pricing.md" rel="noopener noreferrer"&gt;docs&lt;/a&gt; describe Model APIs, which run hosted models behind an API key, and fal Serverless, which deploys your own Python apps on fal GPUs.&lt;/p&gt;

&lt;p&gt;Model APIs are priced by output. On the &lt;a href="https://fal.ai/pricing" rel="noopener noreferrer"&gt;fal pricing page&lt;/a&gt;, FLUX.1 [schnell] costs $0.003 per megapixel, Nano Banana costs $0.0398 per image, and Kling Video v3 Pro costs $0.14 per second of video. Several of those models are closed, so no other platform can host them for you.&lt;/p&gt;

&lt;p&gt;fal Serverless is priced by the GPU-second. The &lt;a href="https://fal.ai/docs/documentation/serverless/pricing.md" rel="noopener noreferrer"&gt;billing docs&lt;/a&gt; say runners are billed while alive, including the &lt;code&gt;setup()&lt;/code&gt; phase that loads your model and idle time while a runner waits for requests. The same docs say Serverless deploys need access that the fal team approves per account.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which fal.ai alternatives are worth a look?
&lt;/h2&gt;

&lt;p&gt;Five platforms cover most of what fal does, and they split the same way fal does: hosted model APIs on one side, your own container on the other.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Best for&lt;/th&gt;
&lt;th&gt;H100 per hour (listed)&lt;/th&gt;
&lt;th&gt;How billing works&lt;/th&gt;
&lt;th&gt;Hosted media model APIs&lt;/th&gt;
&lt;th&gt;Idle workers&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;fal.ai&lt;/td&gt;
&lt;td&gt;Hosted image, video and audio APIs; fal Serverless&lt;/td&gt;
&lt;td&gt;$4.50 list, "as low as" $2.49&lt;/td&gt;
&lt;td&gt;Output-based for Model APIs; per second for Serverless&lt;/td&gt;
&lt;td&gt;Yes, large catalog&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;keep_alive&lt;/code&gt; default 60 s, billed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nodus&lt;/td&gt;
&lt;td&gt;Your own open-weight model as a Function; open LLMs by API (Beta)&lt;/td&gt;
&lt;td&gt;$2.60 (H100 SXM)&lt;/td&gt;
&lt;td&gt;Prepaid credits; Functions bill while workers run or stay warm&lt;/td&gt;
&lt;td&gt;No image or video catalog&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;scaledown_window&lt;/code&gt; default 1 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Replicate&lt;/td&gt;
&lt;td&gt;Calling public models; packaging your own with Cog&lt;/td&gt;
&lt;td&gt;$5.49&lt;/td&gt;
&lt;td&gt;Per second; public models bill only active time&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Private models also pay setup and idle&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Modal&lt;/td&gt;
&lt;td&gt;Python functions that scale up and down&lt;/td&gt;
&lt;td&gt;$3.95 (H100 SXM5)&lt;/td&gt;
&lt;td&gt;Per second&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Scales to zero; &lt;code&gt;scaledown_window&lt;/code&gt; sets the idle time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runpod Serverless&lt;/td&gt;
&lt;td&gt;Custom inference workers&lt;/td&gt;
&lt;td&gt;$5.89 (H100 PRO)&lt;/td&gt;
&lt;td&gt;Per second, from worker start to full stop&lt;/td&gt;
&lt;td&gt;Public Endpoints for some models&lt;/td&gt;
&lt;td&gt;Idle timeout default 5 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Baseten&lt;/td&gt;
&lt;td&gt;Dedicated deployments packaged with Truss&lt;/td&gt;
&lt;td&gt;$6.50 ($0.10833/min)&lt;/td&gt;
&lt;td&gt;Per minute&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Not stated on the pricing page&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Sources: &lt;a href="https://fal.ai/pricing" rel="noopener noreferrer"&gt;fal pricing&lt;/a&gt;, &lt;a href="https://fal.ai/docs/documentation/serverless/pricing.md" rel="noopener noreferrer"&gt;fal Serverless billing&lt;/a&gt;, &lt;a href="https://fal.ai/docs/documentation/deployment/scale-your-application.md" rel="noopener noreferrer"&gt;fal scaling reference&lt;/a&gt;, &lt;a href="https://replicate.com/pricing" rel="noopener noreferrer"&gt;Replicate pricing&lt;/a&gt;, &lt;a href="https://modal.com/pricing" rel="noopener noreferrer"&gt;Modal pricing&lt;/a&gt;, &lt;a href="https://www.runpod.io/pricing" rel="noopener noreferrer"&gt;Runpod pricing&lt;/a&gt;, &lt;a href="https://docs.runpod.io/serverless/pricing" rel="noopener noreferrer"&gt;Runpod Serverless pricing docs&lt;/a&gt;, &lt;a href="https://docs.runpod.io/public-endpoints/overview" rel="noopener noreferrer"&gt;Runpod Public Endpoints&lt;/a&gt;, &lt;a href="https://modal.com/docs/guide/scale" rel="noopener noreferrer"&gt;Modal scaling guide&lt;/a&gt;, &lt;a href="https://replicate.com/docs/topics/billing" rel="noopener noreferrer"&gt;Replicate billing docs&lt;/a&gt;, &lt;a href="https://www.baseten.co/pricing/" rel="noopener noreferrer"&gt;Baseten pricing&lt;/a&gt;, the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing page&lt;/a&gt; and the &lt;a href="https://www.nodus-compute.ai/docs/guides/functions/scaling/" rel="noopener noreferrer"&gt;Nodus scaling docs&lt;/a&gt;. Prices checked November 17, 2026.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo4xqrcsdxo8exp1f3sa2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo4xqrcsdxo8exp1f3sa2.png" alt="Listed price for one H100 per hour on November 17, 2026: fal &amp;quot;as low as&amp;quot; $2.49, Nodus $2.60, Modal $3.95, fal list $4.50, Replicate $5.49, Runpod Serverless $5.89, Baseten $6.50." width="800" height="460"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;fal's discounted rate is the only H100 price below Nodus; its list price sits in the middle of the pack.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;fal's pricing page does not say what qualifies a deployment for the "as low as" rate, and it links to sales. We compare against both numbers below so you can see the range.&lt;/p&gt;
&lt;h2&gt;
  
  
  Which is cheapest for running your own model?
&lt;/h2&gt;

&lt;p&gt;On listed rates, Nodus is cheapest unless you qualify for fal's discounted H100 price. We priced one H100 deployment that is alive for 200 hours in a month, which is roughly a busy business-hours service.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Platform&lt;/th&gt;
&lt;th&gt;Rate used&lt;/th&gt;
&lt;th&gt;1 x H100 alive for 200 hours&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;fal, "as low as"&lt;/td&gt;
&lt;td&gt;$2.49/hr&lt;/td&gt;
&lt;td&gt;$498.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nodus&lt;/td&gt;
&lt;td&gt;$2.60/hr&lt;/td&gt;
&lt;td&gt;$520.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Modal&lt;/td&gt;
&lt;td&gt;$0.001097/s&lt;/td&gt;
&lt;td&gt;$789.84&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;fal, list&lt;/td&gt;
&lt;td&gt;$4.50/hr&lt;/td&gt;
&lt;td&gt;$900.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Replicate&lt;/td&gt;
&lt;td&gt;$0.001525/s&lt;/td&gt;
&lt;td&gt;$1,098.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runpod Serverless&lt;/td&gt;
&lt;td&gt;$5.89/hr&lt;/td&gt;
&lt;td&gt;$1,178.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Baseten&lt;/td&gt;
&lt;td&gt;$0.10833/min&lt;/td&gt;
&lt;td&gt;$1,299.96&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The per-second and per-minute rates were multiplied out exactly: 720,000 seconds at $0.001097 is $789.84, 720,000 seconds at $0.001525 is $1,098.00, and 12,000 minutes at $0.10833 is $1,299.96. Against fal's list price, Nodus is $380.00 less, or 42%. Against fal's best advertised rate, Nodus is $22.00 more.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxm63lg11gusjifxq3fcj.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxm63lg11gusjifxq3fcj.png" alt="One H100 alive for 200 hours at listed rates: fal &amp;quot;as low as&amp;quot; $498.00, Nodus $520.00, Modal $789.84, fal list $900.00, Replicate $1,098.00, Runpod Serverless $1,178.00, Baseten $1,299.96." width="800" height="460"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;At list prices the same 200 H100-hours cost 42% less on Nodus than on fal.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;These rows are GPU time only. Replicate's H100 includes 13 vCPUs and 144 GB of memory, while the Nodus pricing page lists CPU and memory as separate lines. A preprocessing-heavy service adds to the Nodus figure, so price your real container.&lt;/p&gt;
&lt;h2&gt;
  
  
  When does self-hosting beat paying per image?
&lt;/h2&gt;

&lt;p&gt;Self-hosting wins once one GPU produces more images per hour than its hourly price divided by the per-image price. We worked this rule out from the listed prices below, and it needs only two numbers you can check yourself.&lt;/p&gt;

&lt;p&gt;Take FLUX.1 [schnell] at fal's $0.003 per megapixel. One H100 on Nodus at $2.60/hr breaks even at 2.60 ÷ 0.003 = 867 one-megapixel images per hour, or one image every 4.15 seconds. The same H100 at fal's $4.50 list price needs 1,500 images per hour to break even. At Modal's $3.95 per hour it needs 1,316, and at Replicate's $5.49 it needs 1,830.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl9m0euq0rt6iu7726nia.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl9m0euq0rt6iu7726nia.png" alt="Images per hour one H100 must produce to match FLUX.1 schnell at $0.003 per megapixel: Nodus 867, Modal 1,316, fal Serverless list 1,500, Replicate 1,830." width="800" height="328"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The cheaper the GPU-hour, the lower the throughput you need before your own deployment pays off.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Utilization matters as much as the GPU price. If your traffic keeps the GPU busy only half the hours it is alive, the busy hours must reach 1,733 images each on Nodus.&lt;/p&gt;

&lt;p&gt;We have not benchmarked FLUX throughput for this post, so measure your own images per GPU-second before you move. For a cheap model like schnell, staying on the per-image API is often right. For a model with a higher per-output price, the break-even throughput drops in proportion.&lt;/p&gt;
&lt;h2&gt;
  
  
  Can Nodus serve the models I call on fal?
&lt;/h2&gt;

&lt;p&gt;Partly. Nodus serves open-weight models through one OpenAI-compatible API, now in &lt;a href="https://www.nodus-compute.ai/docs/guides/inference/" rel="noopener noreferrer"&gt;Beta&lt;/a&gt;, and the &lt;a href="https://www.nodus-compute.ai/docs/guides/inference/" rel="noopener noreferrer"&gt;docs&lt;/a&gt; name GPT-OSS, Whisper and Gemma among eligible models. Closed models are not in the public catalog, and Nodus has no hosted image or video model catalog today.&lt;/p&gt;

&lt;p&gt;For image and video work, the Nodus path is your own model as a &lt;a href="https://www.nodus-compute.ai/docs/guides/functions/" rel="noopener noreferrer"&gt;Function&lt;/a&gt;. A GPU class loads weights once in an &lt;code&gt;@nodus.enter()&lt;/code&gt; hook and then serves many calls. With &lt;a href="https://www.nodus-compute.ai/docs/guides/functions/gpu-snapshots/" rel="noopener noreferrer"&gt;GPU snapshots&lt;/a&gt; (Beta, enabled per organization), later workers restore the loaded model instead of loading it again. That targets the same cold-start cost fal bills during &lt;code&gt;setup()&lt;/code&gt;.&lt;/p&gt;
&lt;h2&gt;
  
  
  Where is fal.ai the better choice?
&lt;/h2&gt;

&lt;p&gt;fal is the better choice when you want a hosted generative media model with no deployment at all. Its catalog includes closed models like Nano Banana and Kling, which you cannot self-host anywhere. If your volume is low or spiky, paying $0.003 per megapixel beats keeping any GPU warm.&lt;/p&gt;

&lt;p&gt;fal also wins on H100 price if your account qualifies for its "as low as" rate of $2.49, which is $0.11 below the Nodus list. And its docs and examples are built around media workloads: ComfyUI workflows, video models and cold-start tuning for large diffusion weights. A team that lives in that world may move faster staying put.&lt;/p&gt;
&lt;h2&gt;
  
  
  How do you move a fal Serverless app to Nodus?
&lt;/h2&gt;

&lt;p&gt;The concepts map almost one to one. fal's &lt;code&gt;setup()&lt;/code&gt; becomes an &lt;code&gt;@nodus.enter()&lt;/code&gt; method, &lt;code&gt;keep_alive&lt;/code&gt; becomes &lt;code&gt;scaledown_window&lt;/code&gt;, and &lt;code&gt;min_concurrency&lt;/code&gt; becomes &lt;code&gt;min_workers&lt;/code&gt;. This shape comes from the &lt;a href="https://www.nodus-compute.ai/docs/guides/functions/gpu-snapshots/" rel="noopener noreferrer"&gt;Nodus GPU snapshot example&lt;/a&gt;; swap in your own model loading and handler.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;nodus&lt;/span&gt;

&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;nodus&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;App&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"image-api"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;image&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;nodus&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Image&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;debian_slim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pip_install&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"torch==2.6.0"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"transformers==4.53.3"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="nd"&gt;@app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cls&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gpu&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"A10"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;image&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;gpu_snapshot&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;scaledown_window&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"30s"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Generator&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nd"&gt;@nodus&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;enter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;  &lt;span class="c1"&gt;# load your weights onto the GPU once per cold worker&lt;/span&gt;

    &lt;span class="nd"&gt;@nodus&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;method&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;generate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;  &lt;span class="c1"&gt;# run one request&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Deploy it with &lt;code&gt;nodus deploy app.py&lt;/code&gt;, and the App stays up after your program exits. Set the GPU type to the one your model needs from the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt;. If you also call open LLMs, the &lt;a href="https://www.nodus-compute.ai/docs/guides/python/inference/" rel="noopener noreferrer"&gt;Python inference guide&lt;/a&gt; shows &lt;code&gt;nodus.llm.openai()&lt;/code&gt;, which returns the official OpenAI client pointed at Nodus.&lt;/p&gt;

&lt;p&gt;For more options beyond fal, our &lt;a href="https://www.nodus-compute.ai/blog/replicate-alternatives-2026/" rel="noopener noreferrer"&gt;Replicate alternatives&lt;/a&gt; and &lt;a href="https://www.nodus-compute.ai/blog/best-serverless-gpu-platforms-2026/" rel="noopener noreferrer"&gt;serverless GPU platforms&lt;/a&gt; posts compare the same field from other angles.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Is fal.ai only for image generation?
&lt;/h3&gt;

&lt;p&gt;No. Its pricing page lists video, audio and 3D models alongside image models, and fal Serverless runs any Python app you deploy. Most of its catalog is generative media rather than text LLMs.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does fal charge for cold starts?
&lt;/h3&gt;

&lt;p&gt;Partly. fal's billing docs say scheduling and image pulls are not billed, but the &lt;code&gt;setup()&lt;/code&gt; phase that loads your model is, along with idle time. HTTP 500-range errors are not charged.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I run LLMs on Nodus instead of a media model?
&lt;/h3&gt;

&lt;p&gt;Yes. Nodus inference (Beta) serves open-weight LLMs such as GPT-OSS through an OpenAI-compatible API, and named endpoints add rate limits and a spending cap. Our &lt;a href="https://www.nodus-compute.ai/blog/llm-inference-cost-per-token/" rel="noopener noreferrer"&gt;LLM cost per token guide&lt;/a&gt; covers how to compare those prices.&lt;/p&gt;

&lt;h3&gt;
  
  
  What does an idle worker cost on Nodus?
&lt;/h3&gt;

&lt;p&gt;A warm Nodus Function worker is billed at its rate while it waits, for as long as &lt;code&gt;scaledown_window&lt;/code&gt; keeps it, which defaults to one minute. Set &lt;code&gt;min_workers&lt;/code&gt; to 0 and the Function scales to zero when nothing is queued.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.nodus-compute.ai/blog/fal-ai-alternatives/" rel="noopener noreferrer"&gt;Nodus blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>inference</category>
      <category>serverless</category>
      <category>gpu</category>
    </item>
    <item>
      <title>How much does LLM inference cost per million tokens?</title>
      <dc:creator>Viswaretas Kotra</dc:creator>
      <pubDate>Tue, 17 Nov 2026 21:34:16 +0000</pubDate>
      <link>https://dev.to/viswaretas_kotra/how-much-does-llm-inference-cost-per-million-tokens-1j1</link>
      <guid>https://dev.to/viswaretas_kotra/how-much-does-llm-inference-cost-per-million-tokens-1j1</guid>
      <description>&lt;p&gt;One million tokens costs about $0.09 to $0.15 of input and $0.18 to $0.60 of output on small and open-model APIs, and $2 to $8 of input and $10 to $40 of output on mid-tier and frontier models. Nodus serves open-weight models such as gpt-oss-120b through a per-token inference API (Beta) that prices input, cached input and output separately and reports the charge on every request. The list price is only half the answer: in a 20-turn agent loop we priced, resent history made up 98% of the tokens, and prompt caching cut the session cost by 78%.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does 1 million tokens cost today?
&lt;/h2&gt;

&lt;p&gt;It depends on the tier far more than on the provider. Within a tier, the big labs price almost identically. Between the top and bottom tiers, input prices differ by 20 to 90 times and output prices by 20 to over 200 times.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tier&lt;/th&gt;
&lt;th&gt;Model (provider)&lt;/th&gt;
&lt;th&gt;Input per 1M&lt;/th&gt;
&lt;th&gt;Cached input per 1M&lt;/th&gt;
&lt;th&gt;Output per 1M&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Frontier&lt;/td&gt;
&lt;td&gt;Claude Opus 5.5 (Anthropic)&lt;/td&gt;
&lt;td&gt;$4.00&lt;/td&gt;
&lt;td&gt;$0.20&lt;/td&gt;
&lt;td&gt;$20.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Frontier&lt;/td&gt;
&lt;td&gt;gpt-6.2-astra (OpenAI)&lt;/td&gt;
&lt;td&gt;$8.00&lt;/td&gt;
&lt;td&gt;$0.40&lt;/td&gt;
&lt;td&gt;$40.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mid&lt;/td&gt;
&lt;td&gt;Claude Sonnet 5.5 (Anthropic)&lt;/td&gt;
&lt;td&gt;$2.00&lt;/td&gt;
&lt;td&gt;$0.10&lt;/td&gt;
&lt;td&gt;$10.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mid&lt;/td&gt;
&lt;td&gt;gpt-6.2-sol (OpenAI)&lt;/td&gt;
&lt;td&gt;$2.00&lt;/td&gt;
&lt;td&gt;$0.10&lt;/td&gt;
&lt;td&gt;$10.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Small&lt;/td&gt;
&lt;td&gt;Claude Haiku 5.5 (Anthropic)&lt;/td&gt;
&lt;td&gt;$0.10&lt;/td&gt;
&lt;td&gt;$0.01&lt;/td&gt;
&lt;td&gt;$0.50&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Small&lt;/td&gt;
&lt;td&gt;gpt-6-luna (OpenAI)&lt;/td&gt;
&lt;td&gt;$0.10&lt;/td&gt;
&lt;td&gt;$0.01&lt;/td&gt;
&lt;td&gt;$0.50&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Open model&lt;/td&gt;
&lt;td&gt;gpt-oss-120B (Together AI)&lt;/td&gt;
&lt;td&gt;$0.15&lt;/td&gt;
&lt;td&gt;not listed&lt;/td&gt;
&lt;td&gt;$0.60&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Open model&lt;/td&gt;
&lt;td&gt;DeepSeek-V4-Flash (DeepInfra)&lt;/td&gt;
&lt;td&gt;$0.09&lt;/td&gt;
&lt;td&gt;$0.018&lt;/td&gt;
&lt;td&gt;$0.18&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Prices checked November 17, 2026, standard tier, from &lt;a href="https://www.anthropic.com/pricing" rel="noopener noreferrer"&gt;Anthropic's pricing page&lt;/a&gt; (Haiku rates for prompts up to 100K tokens, cache rates for the 5-minute cache), &lt;a href="https://developers.openai.com/api/docs/pricing" rel="noopener noreferrer"&gt;OpenAI's API pricing&lt;/a&gt; (short context), &lt;a href="https://www.together.ai/pricing" rel="noopener noreferrer"&gt;Together AI's pricing&lt;/a&gt; and &lt;a href="https://deepinfra.com/pricing" rel="noopener noreferrer"&gt;DeepInfra's pricing&lt;/a&gt;. Batch and flex tiers cost less. Anthropic and OpenAI both list batch processing at half the standard rate.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fchur4jduyzt876kteftx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fchur4jduyzt876kteftx.png" alt="Output price per million tokens: gpt-6.2-astra $40, Claude Opus 5.5 $20, Claude Sonnet 5.5 and gpt-6.2-sol $10, gpt-oss-120B on Together AI $0.60, Claude Haiku 5.5 and gpt-6-luna $0.50, DeepSeek-V4-Flash on DeepInfra $0.18." width="800" height="504"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Output tokens from a frontier model cost about 33 to 220 times more than from a small or open model.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Two patterns hold across the table. Output costs 5 times input at Anthropic and OpenAI, and 2 to 4 times input on the open-model rows. Cached input costs 5% to 20% of the input rate wherever it is listed.&lt;/p&gt;
&lt;h2&gt;
  
  
  Why are input, output, cached and reasoning tokens priced differently?
&lt;/h2&gt;

&lt;p&gt;Each token type uses a different amount of GPU time. The price follows the work.&lt;/p&gt;

&lt;p&gt;Input tokens are processed together in one parallel pass, called prefill. The GPU reads the model weights once and handles thousands of prompt tokens at the same time, so each input token is cheap.&lt;/p&gt;

&lt;p&gt;Output tokens are generated one at a time. Every new token needs another pass over the weights and the growing cache of earlier tokens. That serial work is why output costs several times more.&lt;/p&gt;

&lt;p&gt;Cached input is a prompt prefix the provider has already processed and stored. A cache hit skips the prefill work, so Anthropic and OpenAI charge 5% to 10% of the input rate for it ($0.10 against $2.00 on Sonnet 5.5). Writing the cache can cost extra: Anthropic lists $2.50 per million for Sonnet 5.5 cache writes, 1.25 times its input rate, and OpenAI lists the same $2.50 as a 30-minute cache write for gpt-6.2-sol.&lt;/p&gt;

&lt;p&gt;Reasoning tokens are the model's hidden thinking. You never see them, but OpenAI's &lt;a href="https://developers.openai.com/api/docs/guides/reasoning" rel="noopener noreferrer"&gt;reasoning guide&lt;/a&gt; says they are billed as output tokens. The &lt;a href="https://www.nodus-compute.ai/docs/guides/inference/" rel="noopener noreferrer"&gt;Nodus inference guide&lt;/a&gt; notes the same for its API: reasoning uses the output-token budget even though the reasoning text is not returned. A model that thinks for 2,000 tokens before a 200-token answer bills 2,200 output tokens.&lt;/p&gt;
&lt;h2&gt;
  
  
  How do you turn token prices into the bill for a workload?
&lt;/h2&gt;

&lt;p&gt;Count each token type per request, multiply by its rate, then multiply by requests. The formula we use:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cost per request = ( uncached input x input price
                   + cache writes   x cache-write price
                   + cache reads    x cached-input price
                   + (visible output + reasoning) x output price ) / 1,000,000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The hard part is counting input honestly. Chat and agent APIs are stateless, so every turn resends the whole conversation: the system prompt, tool definitions, every earlier message and every earlier answer. Input grows with each turn while output stays flat.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does a 20-turn agent session actually cost?
&lt;/h2&gt;

&lt;p&gt;We priced one agent session ourselves to show how this plays out. The setup: an 8,000-token system prompt and tool definitions, 1,500 new input tokens per turn (tool results and instructions) and 500 output tokens per turn, for 20 turns. We left reasoning tokens out to keep the arithmetic visible.&lt;/p&gt;

&lt;p&gt;Turn 1 sends 9,500 input tokens. Each later turn sends the previous context plus the last 500-token answer plus 1,500 new tokens, so turn 20 sends 47,500. The session totals 570,000 input tokens and 10,000 output tokens. That is 57 input tokens for every output token, and input is 98% of all tokens.&lt;/p&gt;

&lt;p&gt;On Claude Sonnet 5.5 without caching:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Input: 570,000 x $2.00 / 1M = $1.14&lt;/li&gt;
&lt;li&gt;Output: 10,000 x $10.00 / 1M = $0.10&lt;/li&gt;
&lt;li&gt;Total: $1.24 per session, with input 92% of the bill&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;With prompt caching, each turn reads the previous context from the cache and writes only the 2,000 new tokens. Across the session that is 522,500 cached reads and 47,500 cache writes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Cache reads: 522,500 x $0.10 / 1M = $0.052&lt;/li&gt;
&lt;li&gt;Cache writes: 47,500 x $2.50 / 1M = $0.119&lt;/li&gt;
&lt;li&gt;Output: $0.10, unchanged&lt;/li&gt;
&lt;li&gt;Total: $0.271 per session, 78% less&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This assumes every turn lands within the cache lifetime, which is 5 minutes in Anthropic's listed rates. A slow human between turns can miss it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmxxa5b06c9tp11sp5y2q.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmxxa5b06c9tp11sp5y2q.png" alt="Cost of a 20-turn Claude Sonnet 5.5 agent session: without caching $1.140 input and $0.100 output; with caching $0.119 cache writes, $0.052 cache reads and $0.100 output." width="800" height="304"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Without caching, resent history is 92% of the bill; caching turns it into the smallest line.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The gap widens every turn. Without caching, turn 1 costs $0.024, turn 5 $0.040, turn 10 $0.060, turn 15 $0.080 and turn 20 $0.100. With caching, turn 1 costs $0.0288 because of the cache write, then turn 5 costs $0.0116, turn 10 $0.0126, turn 15 $0.0136 and turn 20 $0.0146.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fefykjs6p3wqxraw81tve.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fefykjs6p3wqxraw81tve.png" alt="Per-turn cost of a Claude Sonnet 5.5 agent session: without caching it climbs from $0.024 at turn 1 to $0.100 at turn 20; with caching it is $0.0288 at turn 1, then $0.0116 at turn 5 and $0.0146 at turn 20." width="800" height="416"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Uncached sessions get more expensive every turn; cached ones stay nearly flat after the first.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The same session on other models, using the rates in the table:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;No caching&lt;/th&gt;
&lt;th&gt;With caching&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Claude Opus 5.5&lt;/td&gt;
&lt;td&gt;$2.48&lt;/td&gt;
&lt;td&gt;$0.542&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Sonnet 5.5&lt;/td&gt;
&lt;td&gt;$1.24&lt;/td&gt;
&lt;td&gt;$0.271&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Haiku 5.5&lt;/td&gt;
&lt;td&gt;$0.0620&lt;/td&gt;
&lt;td&gt;$0.0162&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120B (Together AI)&lt;/td&gt;
&lt;td&gt;$0.0915&lt;/td&gt;
&lt;td&gt;no cached rate listed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek-V4-Flash (DeepInfra)&lt;/td&gt;
&lt;td&gt;$0.0531&lt;/td&gt;
&lt;td&gt;$0.0155&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For DeepInfra we priced cache writes at the normal input rate, because its page lists no separate write price. gpt-6.2-sol and gpt-6-luna list the same input, cached and output rates as Sonnet 5.5 and Haiku 5.5. Per million tokens processed, the uncached Sonnet session works out to $2.14 and the cached one to $0.467.&lt;/p&gt;
&lt;h2&gt;
  
  
  Which price should you compare when picking a model?
&lt;/h2&gt;

&lt;p&gt;Compare the price of the token type that dominates your traffic. A simple rule follows from the formula: divide the output price by the input price to get a ratio R. If your requests send more than R input tokens per output token, input and caching drive your bill. If they send fewer, output does.&lt;/p&gt;

&lt;p&gt;At Anthropic and OpenAI, R is 5. Our agent session sent 57 input tokens per output token, so the cached-input rate mattered more than anything else. A long-form writer that sends 1,000 tokens and gets 3,000 back sits on the other side, and output price decides.&lt;/p&gt;

&lt;p&gt;So for chat, agents, RAG and coding assistants, check whether caching exists, what a cache write costs and how long the cache lives. For generation-heavy work and reasoning models, compare output prices and count reasoning tokens. If you can wait for results, batch tiers halve the bill at Anthropic and OpenAI.&lt;/p&gt;
&lt;h2&gt;
  
  
  How do you check per-token prices on Nodus?
&lt;/h2&gt;

&lt;p&gt;The Nodus inference API is OpenAI-compatible and serves open models such as gpt-oss-120b and gpt-oss-20b. Its console shows Input, Cached input and Output prices in USD per million tokens for each model, and &lt;code&gt;GET /v1/models&lt;/code&gt; returns the same rates. We do not repeat them here because they move as the catalog updates. Catalog cards use a reference mix of 3 input tokens per output token, so check the full rate card if your traffic looks like the agent session above.&lt;/p&gt;

&lt;p&gt;Three details from the Nodus &lt;a href="https://www.nodus-compute.ai/docs/guides/inference/" rel="noopener noreferrer"&gt;inference docs&lt;/a&gt; matter for budgeting. The accepted rate is fixed for each request, so a price change or a pricier fallback does not raise it. Every successful chat response includes &lt;code&gt;usage.cost&lt;/code&gt; and the count of cached tokens. A named endpoint can carry a spending cap:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;$ nodus create inferenceendpoint support-bot --model nodus/gpt-oss-120b --max-cost 50
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once requests through it have spent $50, the next one gets &lt;code&gt;402 BudgetExceeded&lt;/code&gt;. The guide's example call to gpt-oss-20b shows the receipt format: 84 input plus 21 output tokens, charged $0.000015.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you convert a GPU-hour price into dollars per million tokens?
&lt;/h2&gt;

&lt;p&gt;Divide the hourly cost by the tokens the GPU serves in that hour:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ per 1M tokens = hourly price x 1,000,000 / (tokens per second x 3,600 x share of hours busy)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Throughput and utilization dominate this number, not the hourly rate. Our post on &lt;a href="https://www.nodus-compute.ai/blog/self-hosting-llm-vs-api-cost/" rel="noopener noreferrer"&gt;whether self-hosting an LLM is cheaper than an API&lt;/a&gt; works through break-even volumes with H100 rates from the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing page&lt;/a&gt;. For a single machine at your desk, see &lt;a href="https://www.nodus-compute.ai/blog/run-llm-locally-vs-cloud/" rel="noopener noreferrer"&gt;running an LLM locally vs in the cloud&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Is a token the same size across models?
&lt;/h3&gt;

&lt;p&gt;No. Each model family has its own tokenizer, so the same text can be a different number of tokens. Compare costs on your own prompts with each provider's token counter, not on words or characters.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do I pay for input tokens if the request fails?
&lt;/h3&gt;

&lt;p&gt;It depends on how it fails. OpenAI's reasoning guide warns that a response cut off by &lt;code&gt;max_output_tokens&lt;/code&gt; can bill input and reasoning tokens without any visible answer. On Nodus, a request that fails with no tokens is &lt;code&gt;Released&lt;/code&gt; and not charged.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does a longer context window cost more per token?
&lt;/h3&gt;

&lt;p&gt;Sometimes. Anthropic charges Haiku 5.5 $0.50 per million input tokens above 100K tokens of prompt, five times its shorter-prompt rate, and OpenAI lists separate long-context prices.&lt;/p&gt;

&lt;h3&gt;
  
  
  How much does a million output tokens cost on a reasoning model?
&lt;/h3&gt;

&lt;p&gt;The same as any output token at that model's rate, but you buy more of them. Hidden reasoning can be several times the visible answer, so budget output by measured total output tokens per request.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.nodus-compute.ai/blog/llm-inference-cost-per-token/" rel="noopener noreferrer"&gt;Nodus blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>cloud</category>
    </item>
    <item>
      <title>What are the best Massed Compute alternatives for AI work?</title>
      <dc:creator>Viswaretas Kotra</dc:creator>
      <pubDate>Tue, 17 Nov 2026 19:04:29 +0000</pubDate>
      <link>https://dev.to/viswaretas_kotra/what-are-the-best-massed-compute-alternatives-for-ai-work-3l39</link>
      <guid>https://dev.to/viswaretas_kotra/what-are-the-best-massed-compute-alternatives-for-ai-work-3l39</guid>
      <description>&lt;p&gt;Massed Compute sells hourly virtual machines on hardware it owns, so a good alternative either costs less per GPU-hour or takes over work you now do by hand. Nodus lists the NVIDIA H100 SXM at $2.60 per GPU-hour on its &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt;, against $3.14 at Massed Compute, and restarts checkpointed jobs on a new machine when one is lost. Runpod fits per-second containers, Lambda fits dedicated 8-GPU instances, Modal fits Python functions and Vast.ai fits buyers who want market prices.&lt;/p&gt;

&lt;p&gt;We build Nodus, so weigh that as you read. Every fact about another company links to its own pricing page or docs, and each sum is written out so you can redo it later.&lt;/p&gt;

&lt;h2&gt;
  
  
  What do you get with Massed Compute today?
&lt;/h2&gt;

&lt;p&gt;Massed Compute gives you a full GPU virtual machine, billed by the hour with no contract. Its &lt;a href="https://massedcompute.com/pricing/" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt; says it owns and operates all of its infrastructure, charges nothing for bandwidth and deploys instances in under 4 minutes. The same page lists SOC 2, HIPAA and GDPR compliance and NVIDIA Preferred Partner status.&lt;/p&gt;

&lt;p&gt;The catalog is wide: one NVIDIA H100 SXM5 with 14 vCPUs and 122 GB of RAM is $3.14/hr, and eight of them are $25.12/hr. Its plain "H100 (80GB)" configuration starts at $2.73/hr for one GPU. Older and smaller cards go far lower: an RTX A6000 is $0.57/hr and an A30 is $0.35/hr. B200 and B300 machines come in 8-GPU form on request.&lt;/p&gt;

&lt;p&gt;Its &lt;a href="https://massedcompute.com/products/on-demand/" rel="noopener noreferrer"&gt;on-demand page&lt;/a&gt; adds the working details. Instances ship with NVIDIA drivers and CUDA installed, a virtual desktop is included, and billing is hourly. The &lt;a href="https://massedcompute.com/products/inventory-api/" rel="noopener noreferrer"&gt;Inventory API page&lt;/a&gt; lists a minimum term of 1 hour, a REST API and an MCP server, so an AI agent can launch machines too.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which Massed Compute alternative fits which kind of work?
&lt;/h2&gt;

&lt;p&gt;Choose by how your work runs, not only by the H100 price. This table prices one H100 SXM on each platform.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Platform&lt;/th&gt;
&lt;th&gt;1 x H100 SXM per GPU-hour&lt;/th&gt;
&lt;th&gt;Billing&lt;/th&gt;
&lt;th&gt;What you run&lt;/th&gt;
&lt;th&gt;Good fit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Massed Compute&lt;/td&gt;
&lt;td&gt;$3.14&lt;/td&gt;
&lt;td&gt;Hourly, 1-hour minimum term&lt;/td&gt;
&lt;td&gt;Full VMs, bare metal, clusters&lt;/td&gt;
&lt;td&gt;Long sessions on a machine you control&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nodus&lt;/td&gt;
&lt;td&gt;$2.60&lt;/td&gt;
&lt;td&gt;Prepaid credits&lt;/td&gt;
&lt;td&gt;Jobs, Workspaces, sandboxes, functions&lt;/td&gt;
&lt;td&gt;Runs that should restart and stop on their own&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runpod&lt;/td&gt;
&lt;td&gt;$2.69 Community Cloud, $4.39 Secure Cloud&lt;/td&gt;
&lt;td&gt;Per second&lt;/td&gt;
&lt;td&gt;Container pods&lt;/td&gt;
&lt;td&gt;Containers at a fixed per-second price&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lambda&lt;/td&gt;
&lt;td&gt;$4.89 on a 1-GPU instance, $4.59 on 8 GPUs&lt;/td&gt;
&lt;td&gt;Per GPU-hour&lt;/td&gt;
&lt;td&gt;Instances of 1 to 8 GPUs&lt;/td&gt;
&lt;td&gt;Dedicated multi-GPU machines&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Modal&lt;/td&gt;
&lt;td&gt;$3.95&lt;/td&gt;
&lt;td&gt;Per second&lt;/td&gt;
&lt;td&gt;Python functions in containers&lt;/td&gt;
&lt;td&gt;Code already written as Modal functions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vast.ai&lt;/td&gt;
&lt;td&gt;Set by supply and demand&lt;/td&gt;
&lt;td&gt;Per second&lt;/td&gt;
&lt;td&gt;Containers on market hosts&lt;/td&gt;
&lt;td&gt;Cheapest hourly rate, if you can shop for it&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Prices checked November 17, 2026: &lt;a href="https://massedcompute.com/pricing/" rel="noopener noreferrer"&gt;Massed Compute pricing&lt;/a&gt;, &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing&lt;/a&gt;, &lt;a href="https://www.runpod.io/pricing" rel="noopener noreferrer"&gt;Runpod pricing&lt;/a&gt;, &lt;a href="https://lambda.ai/pricing" rel="noopener noreferrer"&gt;Lambda pricing&lt;/a&gt;, &lt;a href="https://modal.com/pricing" rel="noopener noreferrer"&gt;Modal pricing&lt;/a&gt; and &lt;a href="https://vast.ai/pricing" rel="noopener noreferrer"&gt;Vast.ai pricing&lt;/a&gt;. Modal's $0.001097 per second is multiplied out to an hour. Vast.ai publishes live market rates rather than a fixed price.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu0f52eo4paj24z0xsdk9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu0f52eo4paj24z0xsdk9.png" alt="Listed price for one H100 SXM per GPU-hour on November 17, 2026: Nodus $2.60, Runpod Community Cloud $2.69, Massed Compute $3.14, Modal $3.95, Runpod Secure Cloud $4.39 and Lambda $4.89." width="800" height="416"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Nodus and Runpod Community Cloud list the H100 SXM below Massed Compute; Modal, Runpod Secure Cloud and Lambda list it above.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Vast.ai deserves a word on its model. Its pricing page says prices are set by the market across 40+ data centers, with on-demand, interruptible and reserved tiers, and billing down to the second. That can be the lowest rate on a given day, but you pick the host yourself. Our &lt;a href="https://www.nodus-compute.ai/blog/vast-ai-alternatives-2026/" rel="noopener noreferrer"&gt;Vast.ai alternatives guide&lt;/a&gt; covers that tradeoff, and our &lt;a href="https://www.nodus-compute.ai/blog/runpod-alternatives-2026/" rel="noopener noreferrer"&gt;Runpod alternatives guide&lt;/a&gt; goes deeper on Runpod.&lt;/p&gt;
&lt;h2&gt;
  
  
  What does an 8 x H100 training run cost on each?
&lt;/h2&gt;

&lt;p&gt;On list prices, a 10-hour run on eight H100 SXM GPUs costs $208.00 on Nodus and $251.20 on Massed Compute. We worked this from each pricing page: 80 GPU-hours, nothing else added.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Platform&lt;/th&gt;
&lt;th&gt;Rate used&lt;/th&gt;
&lt;th&gt;80 GPU-hours&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Nodus&lt;/td&gt;
&lt;td&gt;$2.60 per GPU-hour&lt;/td&gt;
&lt;td&gt;$208.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runpod Community Cloud&lt;/td&gt;
&lt;td&gt;$2.69 per GPU-hour&lt;/td&gt;
&lt;td&gt;$215.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Massed Compute&lt;/td&gt;
&lt;td&gt;$25.12/hr for 8 GPUs&lt;/td&gt;
&lt;td&gt;$251.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Modal&lt;/td&gt;
&lt;td&gt;$0.001097 per GPU-second&lt;/td&gt;
&lt;td&gt;$315.94&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runpod Secure Cloud&lt;/td&gt;
&lt;td&gt;$4.39 per GPU-hour&lt;/td&gt;
&lt;td&gt;$351.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lambda&lt;/td&gt;
&lt;td&gt;$4.59 per GPU-hour on 8 GPUs&lt;/td&gt;
&lt;td&gt;$367.20&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The sums: 8 x 10 x $2.60 = $208.00 on Nodus, and 10 x $25.12 = $251.20 on Massed Compute, so Nodus is $43.20 or 17% lower. Modal is 0.001097 x 3,600 x 80 = $315.94. These are compute only. Massed Compute's rate includes the vCPUs, RAM and disk on its page and no bandwidth fees, while Modal and others bill some storage and egress separately.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa2z7xghyzlaabzy2urxe.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa2z7xghyzlaabzy2urxe.png" alt="For 80 GPU-hours on 8 x H100 SXM at list prices: Nodus $208.00, Runpod Community Cloud $215.20, Massed Compute $251.20, Modal $315.94, Runpod Secure Cloud $351.20 and Lambda $367.20." width="800" height="416"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The same 80 GPU-hours range from $208.00 to $367.20 across six list prices.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The rate is only half the bill. On an hourly machine, every hour it sits running before or after the work costs the full rate. If a run on Massed Compute starts two hours late because nobody was watching, that is another $50.24. Price your idle and restart hours before you compare rates.&lt;/p&gt;
&lt;h2&gt;
  
  
  How do Massed Compute GPU prices compare with Nodus card by card?
&lt;/h2&gt;

&lt;p&gt;Nodus lists a lower price than Massed Compute on every card in this comparison. The gap is widest on the RTX PRO 6000 and the RTX A6000.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;GPU&lt;/th&gt;
&lt;th&gt;Massed Compute, 1 GPU&lt;/th&gt;
&lt;th&gt;Nodus list price&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;H100 SXM 80 GB&lt;/td&gt;
&lt;td&gt;$3.14&lt;/td&gt;
&lt;td&gt;$2.60&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A100 SXM 80 GB&lt;/td&gt;
&lt;td&gt;$1.38&lt;/td&gt;
&lt;td&gt;$1.09&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;L40S 48 GB&lt;/td&gt;
&lt;td&gt;$0.97&lt;/td&gt;
&lt;td&gt;$0.793&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RTX PRO 6000 96 GB&lt;/td&gt;
&lt;td&gt;$2.19&lt;/td&gt;
&lt;td&gt;$1.35&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RTX 6000 Ada 48 GB&lt;/td&gt;
&lt;td&gt;$0.79&lt;/td&gt;
&lt;td&gt;$0.743&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RTX A6000 48 GB&lt;/td&gt;
&lt;td&gt;$0.57&lt;/td&gt;
&lt;td&gt;$0.333&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Prices checked November 17, 2026, per GPU-hour: &lt;a href="https://massedcompute.com/pricing/" rel="noopener noreferrer"&gt;Massed Compute pricing&lt;/a&gt; and &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing&lt;/a&gt;. Massed Compute's rates include the vCPUs, RAM and storage listed with each configuration.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8y8jjcndrxuzs1d5vxy8.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8y8jjcndrxuzs1d5vxy8.png" alt="Listed per-GPU-hour prices on November 17, 2026, Massed Compute vs Nodus: H100 SXM $3.14 vs $2.60, A100 80GB $1.38 vs $1.09, L40S $0.97 vs $0.793, RTX PRO 6000 $2.19 vs $1.35 and RTX A6000 $0.57 vs $0.333." width="800" height="416"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Nodus lists the RTX PRO 6000 38% below Massed Compute and the H100 SXM 17% below.&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  What happens when a long run loses its machine?
&lt;/h2&gt;

&lt;p&gt;On a plain virtual machine, your own script owns recovery: it saves checkpoints, and someone restarts it after a failure. That is true on Massed Compute and on most hourly VMs. It works well when a person is watching, and it costs hours when the failure lands at 3 a.m.&lt;/p&gt;

&lt;p&gt;Nodus handles recovery for you on a job. The &lt;a href="https://www.nodus-compute.ai/docs/concepts/attempts-and-recovery/" rel="noopener noreferrer"&gt;attempts and recovery docs&lt;/a&gt; say a machine that stops sending heartbeats gets a replacement prepared after 20 seconds. On shared capacity it is declared lost after 60 seconds and replaced.&lt;/p&gt;

&lt;p&gt;A &lt;code&gt;Checkpointed&lt;/code&gt; run then restarts from its latest committed checkpoint, so it loses at most the work since that save. Nodus restores files, never process memory, so your script still loads its own checkpoint when it starts.&lt;/p&gt;

&lt;p&gt;For interactive work, a Nodus Workspace stops itself when nobody is using it. The &lt;a href="https://www.nodus-compute.ai/docs/guides/workspaces/" rel="noopener noreferrer"&gt;Workspaces docs&lt;/a&gt; set the default &lt;code&gt;idleTimeout&lt;/code&gt; to 1 hour with no SSH session, editor traffic or running process. Its home directory, &lt;code&gt;/home/nodus&lt;/code&gt;, is saved on every stop, and while stopped you pay only for those saved files. Our &lt;a href="https://www.nodus-compute.ai/blog/spot-vs-on-demand-gpus/" rel="noopener noreferrer"&gt;spot vs on-demand guide&lt;/a&gt; covers when cheaper, interruptible capacity is worth it.&lt;/p&gt;
&lt;h2&gt;
  
  
  Can you connect your Massed Compute account to Nodus?
&lt;/h2&gt;

&lt;p&gt;Yes, in Beta. Our &lt;a href="https://www.nodus-compute.ai/docs/guides/pools/cloud-accounts/" rel="noopener noreferrer"&gt;cloud accounts docs&lt;/a&gt; list &lt;code&gt;MassedCompute&lt;/code&gt; as an API-key connection. You can view your running Massed Compute instances in Nodus, or separately enable running work in that account.&lt;/p&gt;

&lt;p&gt;Viewing is read-only, and for Massed Compute it shows running instances only, not stopped or terminated ones. Running work is an explicit upgrade with a credit limit.&lt;/p&gt;

&lt;p&gt;Nodus then places your Jobs, Workspaces and other single-node work in your account first, and creates, labels and deletes only its own machines there. Massed Compute bills you for those machines directly, and Nodus adds a route fee of $0.02 per device-hour on GPU jobs. If you hold Massed Compute credit, this lets you spend it before Nodus capacity.&lt;/p&gt;
&lt;h2&gt;
  
  
  Where is Massed Compute the better choice?
&lt;/h2&gt;

&lt;p&gt;Massed Compute wins when you want a whole machine on hardware its provider runs end to end. Its pricing page says it owns and operates all its infrastructure, with no bandwidth charges and SOC 2, HIPAA and GDPR listed. For a team moving large datasets in and out, free bandwidth can matter more than the hourly rate.&lt;/p&gt;

&lt;p&gt;It also covers ground Nodus does not. Nodus does not list an A30 or the H100 NVL, and Massed Compute lists both, the A30 at $0.35/hr. Its virtual desktop suits people who want a GPU workstation rather than a terminal, including VFX and rendering work its site names. For larger needs it offers single-tenant bare metal and GPU clusters of 8 to 512+ nodes over InfiniBand, custom-quoted.&lt;/p&gt;

&lt;p&gt;Its pricing is also simple to read. One hourly price per configuration includes vCPUs, RAM and disk, and the Inventory API exposes the same catalog to code and agents. If you run long sessions, restart them yourself and like owning the whole VM, moving will save you less than the table suggests.&lt;/p&gt;
&lt;h2&gt;
  
  
  How do you move a Massed Compute workflow to Nodus?
&lt;/h2&gt;

&lt;p&gt;Package the training command you already run on your VM as a Nodus job. Install the CLI and sign in, as the &lt;a href="https://www.nodus-compute.ai/docs/getting-started/install/" rel="noopener noreferrer"&gt;install docs&lt;/a&gt; show:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install nodus-compute
nodus login
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then submit the run. This line is the example from the &lt;a href="https://www.nodus-compute.ai/docs/reference/cli/nodus_run/" rel="noopener noreferrer"&gt;&lt;code&gt;nodus run&lt;/code&gt; reference&lt;/a&gt;: eight H100s, a $40 spending cap, detached so it keeps running after you log off.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;nodus run --gpu H100:8 --max-cost 40 -d -- torchrun train.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For notebook-style work, the Workspaces docs give &lt;code&gt;nodus create workspace lab --gpu H100&lt;/code&gt;, then &lt;code&gt;nodus ssh workspace/lab&lt;/code&gt;. To keep using Massed Compute credit, the &lt;a href="https://www.nodus-compute.ai/docs/reference/cli/nodus_cloud_connect/" rel="noopener noreferrer"&gt;&lt;code&gt;nodus cloud connect&lt;/code&gt; reference&lt;/a&gt; accepts &lt;code&gt;massedcompute&lt;/code&gt; as the cloud name, with a required &lt;code&gt;--credit-limit&lt;/code&gt; and the API key read from standard input or &lt;code&gt;--key-env&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Is Massed Compute cheaper than Nodus?
&lt;/h3&gt;

&lt;p&gt;Not on the cards both list. Nodus lists the H100 SXM at $2.60 per GPU-hour against $3.14 at Massed Compute, and is lower on the A100, L40S, RTX PRO 6000, RTX 6000 Ada and RTX A6000 too. Massed Compute's price includes the CPU, RAM and disk listed with each machine, and charges no bandwidth fees.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does Massed Compute charge for egress?
&lt;/h3&gt;

&lt;p&gt;Its pricing page says there are no bandwidth charges. That is a real advantage over platforms that bill network egress per GiB, such as Modal at $0.04 per GiB beyond its included allowance.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can an AI agent launch Massed Compute machines?
&lt;/h3&gt;

&lt;p&gt;Yes. Its Inventory API page shows a REST endpoint and an MCP server, with Python, Node.js, Go and Terraform SDKs. Nodus has its own MCP server too, which &lt;code&gt;nodus mcp install&lt;/code&gt; adds to Claude Code, Cursor or Codex.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does Massed Compute offer multi-node training clusters?
&lt;/h3&gt;

&lt;p&gt;Yes, as a custom-quoted product: its pricing page lists GPU clusters that connect nodes over InfiniBand, from 8 to 512+ nodes. On-demand instances on the same page go up to 8 GPUs per machine.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.nodus-compute.ai/blog/massed-compute-alternatives/" rel="noopener noreferrer"&gt;Nodus blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>gpu</category>
      <category>machinelearning</category>
      <category>ai</category>
      <category>cloud</category>
    </item>
    <item>
      <title>How do you run training longer than Modal's function timeout?</title>
      <dc:creator>Viswaretas Kotra</dc:creator>
      <pubDate>Tue, 17 Nov 2026 16:45:38 +0000</pubDate>
      <link>https://dev.to/viswaretas_kotra/how-do-you-run-training-longer-than-modals-function-timeout-1aip</link>
      <guid>https://dev.to/viswaretas_kotra/how-do-you-run-training-longer-than-modals-function-timeout-1aip</guid>
      <description>&lt;p&gt;To train past Modal's 24-hour function timeout, make the run resumable: checkpoint to a Modal Volume, stop each call just before the limit, and spawn a fresh call that resumes. Nodus takes the other route: a Nodus Job's time limit is optional, and Nodus checkpoints and restores your state directory itself, so a 72-hour run is one Job instead of a chain of calls.&lt;/p&gt;

&lt;p&gt;The chain works well on Modal once it is built. The rest of this post is the working pattern, the three ways it loses money, the cost of those losses on a 72-hour run, and when we would stop building it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What exactly does Modal's timeout limit?
&lt;/h2&gt;

&lt;p&gt;Every Modal Function execution has a default timeout of 300 seconds, and you can set anything from 1 second to 24 hours (&lt;a href="https://modal.com/docs/guide/timeouts" rel="noopener noreferrer"&gt;Modal timeouts guide&lt;/a&gt;). The limit is per execution attempt. With &lt;code&gt;modal.Retries&lt;/code&gt;, each retry gets a fresh timeout, and once retries run out the caller gets a &lt;code&gt;modal.exception.FunctionTimeoutError&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Two other limits shape the design. Modal's own &lt;a href="https://modal.com/docs/examples/long-training" rel="noopener noreferrer"&gt;long-training example&lt;/a&gt; notes that 24-hour timeouts with 10 retries give a job about 10 days before someone has to restart it by hand. The same example warns that calls made with &lt;code&gt;.remote()&lt;/code&gt; expire after 24 hours, so a multi-day job should be started with &lt;code&gt;.spawn()&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Modal's &lt;a href="https://modal.com/docs/guide/preemption" rel="noopener noreferrer"&gt;preemption guide&lt;/a&gt; adds the third constraint. All Functions can be preempted, the chance grows with run length, and the non-preemptible option is not supported for GPU Functions. A preempted Function is restarted on the same input, from the beginning of your code.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does a working Modal pattern look like?
&lt;/h2&gt;

&lt;p&gt;The pattern that holds up is a self-stopping chain: each call trains until a time budget you set, saves, commits the Volume, spawns its successor and returns. Retries stay in place for crashes and preemptions, not as the main way past the 24-hour wall. We stop at 23 hours rather than 24 so the last save and commit finish well before the hard limit.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;signal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;modal&lt;/span&gt;

&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;modal&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;App&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"long-train"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;image&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;modal&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Image&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;debian_slim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;python_version&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"3.12"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;uv_pip_install&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"torch"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;vol&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;modal&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Volume&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ckpts"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;create_if_missing&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="no"&gt;CKPT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/ckpts"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="no"&gt;TIME_BUDGET_S&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;23&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt;   &lt;span class="c1"&gt;# stop well inside the 24 h ceiling&lt;/span&gt;
&lt;span class="no"&gt;SAVE_EVERY_S&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;15&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;

&lt;span class="nd"&gt;@app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;function&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gpu&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"H100:8"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;image&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;volumes&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="no"&gt;CKPT&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;vol&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
              &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;single_use_containers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
              &lt;span class="n"&gt;retries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;modal&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Retries&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;initial_delay&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;train_chunk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;started&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;last_save&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;run_dir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;CKPT&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;run_id&lt;/span&gt;
    &lt;span class="n"&gt;run_dir&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mkdir&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parents&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;exist_ok&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sched&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;build&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                     &lt;span class="c1"&gt;# your model, optimizer, scheduler&lt;/span&gt;
    &lt;span class="n"&gt;step&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;load_latest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_dir&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sched&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# 0 on a fresh run&lt;/span&gt;

    &lt;span class="n"&gt;interrupted&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;on_sigint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;nonlocal&lt;/span&gt; &lt;span class="n"&gt;interrupted&lt;/span&gt;
        &lt;span class="n"&gt;interrupted&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;                          &lt;span class="c1"&gt;# Modal sends an interrupt signal on preemption&lt;/span&gt;
    &lt;span class="n"&gt;signal&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;signal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;signal&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;SIGINT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;on_sigint&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="no"&gt;TOTAL_STEPS&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;train_step&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sched&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;step&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="n"&gt;out_of_time&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;started&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="no"&gt;TIME_BUDGET_S&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;interrupted&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;out_of_time&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;last_save&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="no"&gt;SAVE_EVERY_S&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="nf"&gt;save_atomic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_dir&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sched&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;vol&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;commit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                            &lt;span class="c1"&gt;# durable before we move on&lt;/span&gt;
            &lt;span class="n"&gt;last_save&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;interrupted&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;KeyboardInterrupt&lt;/span&gt;                 &lt;span class="c1"&gt;# end this attempt; Modal reruns the input&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;out_of_time&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;train_chunk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;spawn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;               &lt;span class="c1"&gt;# a fresh call with a fresh 24 h timeout&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The save and load helpers carry the real state, not just a step counter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;save_atomic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_dir&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sched&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;tmp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;run_dir&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="s2"&gt;"last.pt.tmp"&lt;/span&gt;
    &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"model"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;state_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"opt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;state_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s2"&gt;"sched"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;sched&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;state_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"step"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s2"&gt;"rng"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_rng_state&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;run_dir&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="s2"&gt;"last.pt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;            &lt;span class="c1"&gt;# never leave a half-written file&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;load_latest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_dir&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sched&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;run_dir&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="s2"&gt;"last.pt"&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exists&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;map_location&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"cpu"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load_state_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"model"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load_state_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"opt"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;sched&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load_state_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"sched"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_rng_state&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"rng"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"step"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Deploy the app with &lt;code&gt;modal deploy&lt;/code&gt;, then start the chain with &lt;code&gt;modal.Function.from_name("long-train", "train_chunk").spawn("run-001")&lt;/code&gt;. Nothing then depends on your laptop staying awake. If you use a dataloader, save its position too, or each resumed chunk replays data it has already seen.&lt;/p&gt;

&lt;p&gt;A few details in that code come straight from Modal's docs.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;single_use_containers=True&lt;/code&gt; is what the long-training example uses so every retry starts in a fresh container that mounts the latest Volume state. The explicit &lt;code&gt;vol.commit()&lt;/code&gt; matters because, per the &lt;a href="https://modal.com/docs/guide/volumes" rel="noopener noreferrer"&gt;Volumes guide&lt;/a&gt;, changes become visible to other containers only after a commit.&lt;/p&gt;

&lt;p&gt;Background commits run every few seconds and a final commit runs at shutdown, but we would rather know the checkpoint is durable than hope the last background commit finished. Volumes v2 is the version Modal recommends for new Volumes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where does a chained run lose time and money?
&lt;/h2&gt;

&lt;p&gt;A chained run pays three costs a single long run does not: work lost since the last save, a startup on every new call, and duplicate work when two calls write the same run. Each one is billed at the full GPU rate.&lt;/p&gt;

&lt;p&gt;Lost progress is the big one. If you rely on retries alone, the 24-hour timeout kills the call mid-step, and everything since the last checkpoint runs again. The self-stopping chain avoids that at planned boundaries, but a preemption still loses up to one save interval.&lt;/p&gt;

&lt;p&gt;Startup is paid on every call. Each new call pulls the image, attaches 8 GPUs, loads the checkpoint from the Volume and warms up. Modal bills Functions by the second (&lt;a href="https://modal.com/pricing" rel="noopener noreferrer"&gt;Modal pricing&lt;/a&gt;), and the GPUs are billed through all of it.&lt;/p&gt;

&lt;p&gt;Duplicate chains are the quiet failure. If a call crashes after &lt;code&gt;spawn()&lt;/code&gt; but before it returns, its retry spawns a second successor, and two chains now write &lt;code&gt;last.pt&lt;/code&gt;. The Volumes guide says last write wins on concurrent writes to the same file. Write a small marker with the next chunk's ID before spawning, and have each new call exit if the marker names someone else.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does that cost on a 72-hour run?
&lt;/h2&gt;

&lt;p&gt;We priced a 72-hour run on 8 NVIDIA H100 GPUs from Modal's pricing page, where an H100 is $0.001097 per second, or $3.9492 per GPU-hour. Eight of them cost $31.5936 per hour of wall time on the GPU meter alone. For the restart we assume 10 minutes from new call to first useful step. That is our illustrative figure, not a measurement, so substitute your own.&lt;/p&gt;

&lt;p&gt;The GPU meter for 72 hours of useful training is 72 x 8 x $3.9492 = $2,274.74. Both chain designs need four calls, because each call fits only about 23 hours of useful work. Everything below is overhead on top of that $2,274.74.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Chain design&lt;/th&gt;
&lt;th&gt;Overhead per boundary&lt;/th&gt;
&lt;th&gt;Boundaries&lt;/th&gt;
&lt;th&gt;Overhead&lt;/th&gt;
&lt;th&gt;Cost at $31.5936/hr&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Self-stopping, 23 h budget&lt;/td&gt;
&lt;td&gt;10 min restart&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;30 min&lt;/td&gt;
&lt;td&gt;$15.80&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retries only, hourly saves (worst case)&lt;/td&gt;
&lt;td&gt;up to 60 min lost + 10 min restart&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;210 min&lt;/td&gt;
&lt;td&gt;$110.58&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The retries-only design needs a fourth call too: each 24-hour attempt can lose up to an hour at the timeout kill, so three attempts cover at most about 68.5 hours of saved work.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0lkj8zhrydqj4jr2leg5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0lkj8zhrydqj4jr2leg5.png" alt="Extra GPU cost of chaining a 72-hour 8 x H100 run on Modal: $15.80 for a self-stopping chain and up to $110.58 for retries only with hourly saves." width="800" height="304"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Stopping each call yourself turns the 24-hour wall into three short restarts instead of three lost hours.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Preemptions add to either design, and the save interval sets the price. On the same 8 x H100 run, one preemption costs up to the interval plus the 10-minute restart.&lt;/p&gt;

&lt;p&gt;With hourly saves that is up to 70 minutes, or $36.86. Saving every 30 minutes caps it at 40 minutes, or $21.06. Saving every 15 minutes caps it at 25 minutes, or $13.16. Modal calls preemptions rare and publishes no rate, so we do not multiply by a count.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv7cqmo38f225zi8alkl6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv7cqmo38f225zi8alkl6.png" alt="Worst-case cost of one preemption on 8 x H100 at Modal's list price: $36.86 with 60-minute saves, $21.06 with 30-minute saves, $13.16 with 15-minute saves." width="800" height="304"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Checkpoint every 15 minutes on multi-GPU runs; the save is cheap next to a lost hour on 8 GPUs.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Prices checked November 17, 2026.&lt;/p&gt;
&lt;h2&gt;
  
  
  How does Nodus run the same 72-hour job?
&lt;/h2&gt;

&lt;p&gt;On Nodus the same run is one Job with no chain, and the checkpoint handling moves from your code into the platform. The &lt;a href="https://www.nodus-compute.ai/docs/guides/jobs/" rel="noopener noreferrer"&gt;Jobs guide&lt;/a&gt; treats the wall-clock &lt;code&gt;timeout&lt;/code&gt; as an optional limit that you can raise, lower or remove on a running Job, so there is no 24-hour boundary to plan around.&lt;/p&gt;

&lt;p&gt;Your program still writes its state, but only to &lt;code&gt;/nodus/state&lt;/code&gt;, and loads it on start, exactly like &lt;code&gt;load_latest&lt;/code&gt; above. Per the &lt;a href="https://www.nodus-compute.ai/docs/guides/checkpoints/" rel="noopener noreferrer"&gt;checkpoints guide&lt;/a&gt;, Nodus copies that directory on an automatic interval, at least four times over the run's expected length. It also checkpoints when the capacity gives notice that it is about to be reclaimed, and restores the directory on the replacement machine. The command comes from our docs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;nodus run --gpu H100:8 --max-cost 1600 -d -- torchrun train.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The checkpoint path defaults to &lt;code&gt;/nodus/state&lt;/code&gt;, and &lt;code&gt;--max-cost&lt;/code&gt; suspends the Job with its state saved if it would spend more than $1,600. The guide also covers a Hugging Face Trainer integration, so a Trainer script only needs &lt;code&gt;output_dir&lt;/code&gt; pointed at the state directory.&lt;/p&gt;

&lt;p&gt;The rate is the other difference. An H100 SXM is listed at $2.60/hr on the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing page&lt;/a&gt;, so 72 hours on 8 GPUs is 72 x 8 x $2.60 = $1,497.60, against $2,274.74 on Modal's GPU meter. That is 34% less before the chain overhead.&lt;/p&gt;

&lt;p&gt;Modal also bills CPU and memory on separate meters. Interruptions still cost a restart on Nodus; what changes is that you do not write or maintain the handoff.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1m982n2ll0b3hwb5rk21.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1m982n2ll0b3hwb5rk21.png" alt="GPU-meter cost of a 72-hour run on 8 x H100: $2,274.74 on Modal at $3.9492 per GPU-hour, $1,497.60 on Nodus at $2.60." width="800" height="304"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The same 576 GPU-hours cost 34% less at the Nodus list rate, before any restart overhead.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If you are moving an existing Modal app, our &lt;a href="https://www.nodus-compute.ai/blog/migrate-from-modal-to-nodus/" rel="noopener noreferrer"&gt;migration guide&lt;/a&gt; shows the smallest code change, and the &lt;a href="https://www.nodus-compute.ai/blog/modal-pricing-explained/" rel="noopener noreferrer"&gt;Modal pricing explainer&lt;/a&gt; covers the rest of Modal's meters.&lt;/p&gt;

&lt;h2&gt;
  
  
  When is the Modal chain the right choice?
&lt;/h2&gt;

&lt;p&gt;Stay on Modal and build the chain when your training already lives there and runs last a few days at most. Per-second billing, Volumes and the rest of your app in one place are real advantages. The long-training example is short, and the code above adds a few dozen lines to it.&lt;/p&gt;

&lt;p&gt;Our decision rule, from the arithmetic above:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Under 20 hours of training: set &lt;code&gt;timeout&lt;/code&gt; to the expected time plus some headroom, checkpoint every 15 to 30 minutes anyway, and keep &lt;code&gt;retries&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;From 20 hours to a few days on Modal: build the self-stopping chain with a 23-hour budget, explicit commits and a handoff marker. Do not use retries as the chain.&lt;/li&gt;
&lt;li&gt;Multi-day runs on 8 or more GPUs, or many long runs a month: run them as Nodus Jobs. The rate gap on 576 GPU-hours is $777.14, more than any overhead in the table above.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For a wider look at where to put long training, see our &lt;a href="https://www.nodus-compute.ai/blog/modal-alternatives-for-training-2026/" rel="noopener noreferrer"&gt;Modal alternatives for training&lt;/a&gt; and &lt;a href="https://www.nodus-compute.ai/blog/resume-training-after-spot-preemption/" rel="noopener noreferrer"&gt;how to make training survive preemption&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What do checkpoints cost to keep on a Modal Volume?
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://modal.com/pricing" rel="noopener noreferrer"&gt;Modal's pricing page&lt;/a&gt; lists Volumes at $0.09 per GiB per month with the first 1 TiB free, so checkpoints cost nothing extra until your total Volume storage passes 1 TiB. The Volumes guide notes that deleted data can stay billed for up to four days, so pruning old checkpoints saves money a few days later, not at once.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I just catch &lt;code&gt;FunctionTimeoutError&lt;/code&gt; and call the Function again?
&lt;/h3&gt;

&lt;p&gt;You can, and the timeouts guide shows how to catch it. It is the weaker design: the work since the last save is already lost when the error arrives, and the caller has to stay alive for the whole multi-day run.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I make a Modal GPU Function non-preemptible?
&lt;/h3&gt;

&lt;p&gt;Not today. The preemption guide says the &lt;code&gt;nonpreemptible&lt;/code&gt; option is not supported for GPU Functions, so every GPU training run should expect a restart and checkpoint for it.&lt;/p&gt;

&lt;h3&gt;
  
  
  How often does Nodus checkpoint a long Job?
&lt;/h3&gt;

&lt;p&gt;With the default &lt;code&gt;interval: auto&lt;/code&gt;, Nodus checkpoints at least four times over the Job's expected run time and keeps the time spent checkpointing to about a tenth of it. You can set a fixed interval from 1 minute to 6 hours instead, and a suspend or reclaim notice always triggers a checkpoint.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.nodus-compute.ai/blog/modal-function-timeout-long-training/" rel="noopener noreferrer"&gt;Nodus blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>machinelearning</category>
      <category>modal</category>
      <category>devops</category>
    </item>
    <item>
      <title>Is self-hosting an LLM cheaper than paying for an API?</title>
      <dc:creator>Viswaretas Kotra</dc:creator>
      <pubDate>Mon, 16 Nov 2026 16:35:22 +0000</pubDate>
      <link>https://dev.to/viswaretas_kotra/is-self-hosting-an-llm-cheaper-than-paying-for-an-api-14ae</link>
      <guid>https://dev.to/viswaretas_kotra/is-self-hosting-an-llm-cheaper-than-paying-for-an-api-14ae</guid>
      <description>&lt;p&gt;Self-hosting an open-weight model on one NVIDIA H100 is cheaper than a frontier API like Claude Sonnet 5.5 once you send it about 16 million tokens a day, but it takes roughly 240 to 400 million tokens a day to beat the cheap per-token APIs that serve the same open models. Nodus offers both sides, hourly GPUs and a per-token inference API (Beta) for open models such as gpt-oss-120b, so you can start per token and move to your own GPU when your volume crosses the line. What decides it is not the hourly price but how busy you keep the GPU.&lt;/p&gt;

&lt;h2&gt;
  
  
  What are you actually comparing?
&lt;/h2&gt;

&lt;p&gt;You are comparing a fixed hourly bill against a variable per-token bill. A GPU you keep running costs the same whether it serves a million tokens or none. An API charges only for tokens, at a rate that already includes someone else's idle hardware, redundancy and engineering.&lt;/p&gt;

&lt;p&gt;To make the two comparable, we turn every API price into one blended price per million tokens. We use the reference mix the &lt;a href="https://www.nodus-compute.ai/docs/guides/inference/" rel="noopener noreferrer"&gt;Nodus inference guide&lt;/a&gt; uses for its catalog: 3 input tokens for every output token. The blended price is (3 x input + output) / 4.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;API and model&lt;/th&gt;
&lt;th&gt;Input per 1M&lt;/th&gt;
&lt;th&gt;Output per 1M&lt;/th&gt;
&lt;th&gt;Blended per 1M (3:1)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Anthropic Claude Sonnet 5.5&lt;/td&gt;
&lt;td&gt;$2.00&lt;/td&gt;
&lt;td&gt;$10.00&lt;/td&gt;
&lt;td&gt;$4.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI gpt-6.1-sol&lt;/td&gt;
&lt;td&gt;$2.00&lt;/td&gt;
&lt;td&gt;$10.00&lt;/td&gt;
&lt;td&gt;$4.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anthropic Claude Haiku 5.5 / OpenAI gpt-6-luna&lt;/td&gt;
&lt;td&gt;$0.10&lt;/td&gt;
&lt;td&gt;$0.50&lt;/td&gt;
&lt;td&gt;$0.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Together AI, gpt-oss-120B&lt;/td&gt;
&lt;td&gt;$0.15&lt;/td&gt;
&lt;td&gt;$0.60&lt;/td&gt;
&lt;td&gt;$0.2625&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepInfra, Llama 3.3 70B Instruct Turbo&lt;/td&gt;
&lt;td&gt;$0.10&lt;/td&gt;
&lt;td&gt;$0.32&lt;/td&gt;
&lt;td&gt;$0.155&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Prices checked November 16, 2026, standard tier, from &lt;a href="https://www.anthropic.com/pricing" rel="noopener noreferrer"&gt;Anthropic's pricing page&lt;/a&gt; (Haiku rate for prompts up to 100K tokens), &lt;a href="https://developers.openai.com/api/docs/pricing" rel="noopener noreferrer"&gt;OpenAI's API pricing&lt;/a&gt; (short context), &lt;a href="https://www.together.ai/pricing" rel="noopener noreferrer"&gt;Together AI's pricing&lt;/a&gt; and &lt;a href="https://deepinfra.com/pricing" rel="noopener noreferrer"&gt;DeepInfra's pricing&lt;/a&gt;. Batch tiers and cached input lower all of these.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjcc1bb7dr3zq9r1gbj48.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjcc1bb7dr3zq9r1gbj48.png" alt="Blended API price per million tokens at a 3:1 input to output mix: Claude Sonnet 5.5 and gpt-6.1-sol $4.00, gpt-oss-120B on Together AI $0.2625, Claude Haiku 5.5 and gpt-6-luna $0.20, and Llama 3.3 70B on DeepInfra $0.155." width="800" height="328"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;A large frontier model costs about 15 to 26 times more per token than an open model served by a per-token API.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;That spread is the whole story. Against a $4.00 frontier price, almost any busy GPU wins. Against $0.155 to $0.2625, you are competing with providers whose job is packing many customers onto the same GPUs.&lt;/p&gt;
&lt;h2&gt;
  
  
  How many tokens a day does one GPU need to break even?
&lt;/h2&gt;

&lt;p&gt;Divide the GPU's daily cost by the blended price. We use one H100 because OpenAI's &lt;a href="https://huggingface.co/openai/gpt-oss-120b" rel="noopener noreferrer"&gt;gpt-oss-120b model card&lt;/a&gt; says the model fits on a single 80 GB GPU such as the H100. The H100 SXM is listed at $2.60/hr on the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing page&lt;/a&gt;, checked November 16, 2026.&lt;/p&gt;

&lt;p&gt;The arithmetic, once:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;One H100 for a day: $2.60 x 24 = $62.40. For an average month of 730 hours: $2.60 x 730 = $1,898.00.&lt;/li&gt;
&lt;li&gt;Against Sonnet 5.5 or gpt-6.1-sol: $62.40 / $4.00 = 15.6M tokens a day, or 474.5M a month.&lt;/li&gt;
&lt;li&gt;Against Haiku 5.5 or gpt-6-luna: $62.40 / $0.20 = 312.0M tokens a day, or 9,490M a month.&lt;/li&gt;
&lt;li&gt;Against gpt-oss-120B on Together AI: $62.40 / $0.2625 = 237.7M tokens a day, or 7,230.5M a month.&lt;/li&gt;
&lt;li&gt;Against Llama 3.3 70B on DeepInfra: $62.40 / $0.155 = 402.6M tokens a day, or 12,245.2M a month.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Femqrxszzmir3k5kcjw8v.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Femqrxszzmir3k5kcjw8v.png" alt="Tokens per day one H100 at $2.60/hr must serve to match each API: 15.6M against Sonnet 5.5 or gpt-6.1-sol, 237.7M against gpt-oss-120B on Together AI, 312.0M against Haiku 5.5 or gpt-6-luna, and 402.6M against Llama 3.3 70B on DeepInfra." width="800" height="328"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Replacing a frontier API pays off at small volume; undercutting an open-model API needs about 15 to 26 times more traffic.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Notice what the frontier row really compares. Switching from Sonnet 5.5 to a self-hosted gpt-oss-120b is a change of model, not just of hosting. It only saves money if the open model passes your evals. Our post on &lt;a href="https://www.nodus-compute.ai/blog/openai-fine-tuning-vs-open-models/" rel="noopener noreferrer"&gt;OpenAI fine-tuning vs open models&lt;/a&gt; runs the same kind of break-even for fine-tuned models, where the API side is pricier and the math favors your own GPU sooner.&lt;/p&gt;
&lt;h2&gt;
  
  
  How does utilization change the answer?
&lt;/h2&gt;

&lt;p&gt;Utilization decides it, because idle hours still bill. The daily totals above assume tokens arrive evenly around the clock. Real traffic does not. If your GPU is busy only 20% of the hours you pay for, it has to serve five times as fast while it is busy to hit the same daily total.&lt;/p&gt;

&lt;p&gt;Here is the throughput your H100 must sustain while busy, in input plus output tokens per second, to tie each API. It is the break-even daily total spread over the busy seconds only.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Share of paid hours the GPU is busy&lt;/th&gt;
&lt;th&gt;vs Sonnet 5.5 ($4.00)&lt;/th&gt;
&lt;th&gt;vs gpt-oss-120B on Together ($0.2625)&lt;/th&gt;
&lt;th&gt;vs Haiku 5.5 ($0.20)&lt;/th&gt;
&lt;th&gt;vs Llama 3.3 70B on DeepInfra ($0.155)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;181 tokens/s&lt;/td&gt;
&lt;td&gt;2,751 tokens/s&lt;/td&gt;
&lt;td&gt;3,611 tokens/s&lt;/td&gt;
&lt;td&gt;4,660 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;80%&lt;/td&gt;
&lt;td&gt;226 tokens/s&lt;/td&gt;
&lt;td&gt;3,439 tokens/s&lt;/td&gt;
&lt;td&gt;4,514 tokens/s&lt;/td&gt;
&lt;td&gt;5,824 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;50%&lt;/td&gt;
&lt;td&gt;361 tokens/s&lt;/td&gt;
&lt;td&gt;5,503 tokens/s&lt;/td&gt;
&lt;td&gt;7,222 tokens/s&lt;/td&gt;
&lt;td&gt;9,319 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;20%&lt;/td&gt;
&lt;td&gt;903 tokens/s&lt;/td&gt;
&lt;td&gt;13,757 tokens/s&lt;/td&gt;
&lt;td&gt;18,056 tokens/s&lt;/td&gt;
&lt;td&gt;23,298 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each cell is the daily break-even divided by 86,400 seconds, then divided by the busy share. For example, 237.7M / 86,400 = 2,751 tokens/s, and 2,751 / 0.2 = 13,757 tokens/s.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frz9e26vggdnw6gf0207w.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frz9e26vggdnw6gf0207w.png" alt="Throughput one H100 must sustain while busy to match each API: against Sonnet 5.5 it rises from 181 tokens/s at 100% busy to 903 at 20%, against gpt-oss-120B on Together AI from 2,751 to 13,757, and against Llama 3.3 70B on DeepInfra from 4,660 to 23,298." width="800" height="416"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Halving how busy the GPU is doubles the throughput it needs, so a part-time GPU struggles to beat a per-token API.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;We have not published a gpt-oss-120b throughput measurement of our own, so we will not tell you which cells an H100 can reach. That depends on your serving engine, prompt lengths, batch sizes and latency target. Measure it at your real traffic shape with an engine like vLLM (our &lt;a href="https://www.nodus-compute.ai/blog/what-is-vllm/" rel="noopener noreferrer"&gt;explainer on what vLLM does&lt;/a&gt; covers why batching matters) and read your number against the table.&lt;/p&gt;

&lt;p&gt;The same arithmetic gives an effective price for your own GPU. Suppose a load test shows 3,000 tokens per second while busy. At 30% busy, the cost is $2.60 / (3,000 x 3,600 x 0.3) x 1,000,000 = $0.80 per million tokens. At 90% busy it is $0.27.&lt;/p&gt;

&lt;p&gt;The GPU and model are the same, yet the price per token triples at the lower utilization. Only the 90% case gets close to the $0.155 to $0.2625 that open-model APIs charge.&lt;/p&gt;
&lt;h2&gt;
  
  
  What does self-hosting cost besides the GPU?
&lt;/h2&gt;

&lt;p&gt;More than the GPU line suggests, and none of it shows up in the break-even above. We leave these unpriced because they depend on your team, but count each one before you decide:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Redundancy. One GPU is one point of failure. Production serving usually means a second replica, which doubles the GPU bill unless traffic justifies both.&lt;/li&gt;
&lt;li&gt;Headroom for peaks. Sizing for your busiest hour lowers average utilization, which pushes you down the table above.&lt;/li&gt;
&lt;li&gt;Engineering time. Choosing an engine and quantization, tuning batch sizes, upgrading models and keeping latency in bounds is ongoing work.&lt;/li&gt;
&lt;li&gt;Startup and storage. Every new replica has to load tens of gigabytes of weights before it serves, and the weights and logs need storage.&lt;/li&gt;
&lt;li&gt;Monitoring, security patching and on-call for a service you now own.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A per-token API folds all of this into its price. That is why its rate can look high per token and still be the cheaper option for most teams.&lt;/p&gt;
&lt;h2&gt;
  
  
  When is self-hosting the better choice?
&lt;/h2&gt;

&lt;p&gt;Self-host when the volume is steady and high, or when something other than price requires it. We would pick a dedicated GPU in these cases:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You are replacing a frontier API with an open model that passes your evals, and you send more than about 16M tokens a day per GPU you would run.&lt;/li&gt;
&lt;li&gt;Your traffic is steady enough to keep a GPU busy most hours, for example batch classification, embeddings or overnight document processing that you can queue.&lt;/li&gt;
&lt;li&gt;You serve a fine-tuned model or several LoRA adapters that no per-token API hosts.&lt;/li&gt;
&lt;li&gt;Data rules, latency or guaranteed capacity matter more than the per-token rate.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Stay on a per-token API when traffic is spiky, below the break-even, or when an open-model API already serves the model you want. If you are weighing buying a workstation instead of paying by the hour, our post on &lt;a href="https://www.nodus-compute.ai/blog/run-llm-locally-vs-cloud/" rel="noopener noreferrer"&gt;running an LLM locally vs in the cloud&lt;/a&gt; does that hardware math.&lt;/p&gt;
&lt;h2&gt;
  
  
  How do you try both on Nodus?
&lt;/h2&gt;

&lt;p&gt;Start per token, then move the same model onto a GPU when your logs show the volume. The Nodus inference API is in Beta and is OpenAI-compatible. Its rates per model, including cached input, are shown in the console's Models catalog and by &lt;code&gt;GET /v1/models&lt;/code&gt;; we do not repeat them here because they update as the catalog changes. These commands are copied from the &lt;a href="https://www.nodus-compute.ai/docs/guides/inference/" rel="noopener noreferrer"&gt;inference guide&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;$ nodus inference models
$ nodus inference chat --model nodus/gpt-oss-20b --max-tokens 64 &lt;span class="s2"&gt;"What is the capital of France?"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;chat&lt;/code&gt; prints the token counts and the charge for the request, so a day of real traffic gives you the input to output mix and the daily volume this post's math needs. A named endpoint adds a spending cap: &lt;code&gt;nodus create inferenceendpoint support-bot --model nodus/gpt-oss-120b --max-cost 50&lt;/code&gt; answers the next request with &lt;code&gt;402 BudgetExceeded&lt;/code&gt; once its requests have spent $50. When your daily tokens clear the break-even and your load test clears the busy-throughput row you need, run the model on an hourly GPU from the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt; instead. For dedicated inference capacity, the guide says Nodus sets up a custom plan.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Is it cheaper to run a local LLM?
&lt;/h3&gt;

&lt;p&gt;For light personal use, yes, if the model already fits on hardware you own, since the extra cost is mostly electricity. For a product's traffic, a local or self-hosted model is only cheaper above the break-even volume, and a cheap open-model API usually wins below a few hundred million tokens a day per GPU.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does output-heavy traffic change the break-even?
&lt;/h3&gt;

&lt;p&gt;Yes. Output tokens cost 3.2 to 5 times more than input on every API above, so a chat or agent workload with long answers has a higher blended price and a lower break-even. Recompute the blend with your own measured mix instead of 3:1.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do batch APIs change the math?
&lt;/h3&gt;

&lt;p&gt;They move it toward the API. OpenAI and Anthropic both list batch processing at half the standard price on their pricing pages, which doubles the tokens a day your GPU must serve to break even on work that can wait.&lt;/p&gt;

&lt;h3&gt;
  
  
  What about models too big for one GPU?
&lt;/h3&gt;

&lt;p&gt;Multiply the GPU cost by the number of GPUs the model needs. A model that needs 8 GPUs needs 8 times the daily tokens to break even, which is why very large open models are usually cheapest through a per-token API, such as the Nodus inference API for models in its catalog, unless traffic is enormous.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.nodus-compute.ai/blog/self-hosting-llm-vs-api-cost/" rel="noopener noreferrer"&gt;Nodus blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ai</category>
      <category>machinelearning</category>
      <category>devops</category>
    </item>
    <item>
      <title>OpenAI fine-tuning vs open models: which costs less?</title>
      <dc:creator>Viswaretas Kotra</dc:creator>
      <pubDate>Mon, 16 Nov 2026 02:37:28 +0000</pubDate>
      <link>https://dev.to/viswaretas_kotra/openai-fine-tuning-vs-open-models-which-costs-less-4oj3</link>
      <guid>https://dev.to/viswaretas_kotra/openai-fine-tuning-vs-open-models-which-costs-less-4oj3</guid>
      <description>&lt;p&gt;Fine-tuning an open model costs far less: 50 million training tokens cost about $8 as a Llama 3.1 8B LoRA run on one H100 on Nodus, against $250 on OpenAI's gpt-4.1-mini and $1,250 on gpt-4.1. Serving is closer, because a dedicated GPU bills by the hour whether or not traffic arrives, so open models win there only above a monthly token volume you can calculate. OpenAI is also winding down self-serve fine-tuning, which settles the choice for most teams.&lt;/p&gt;

&lt;h2&gt;
  
  
  How does OpenAI price fine-tuning?
&lt;/h2&gt;

&lt;p&gt;OpenAI bills training per token, then bills the fine-tuned model per token when you call it. Training is charged on every token in your dataset, once per epoch. The &lt;a href="https://developers.openai.com/cookbook/examples/chat_finetuning_data_prep" rel="noopener noreferrer"&gt;data-prep cookbook&lt;/a&gt; states it as "you'll be charged for ~n_epochs x tokens in dataset". People searching for "ChatGPT fine-tuning cost" usually mean this API product.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Training per 1M tokens&lt;/th&gt;
&lt;th&gt;Input per 1M&lt;/th&gt;
&lt;th&gt;Cached input per 1M&lt;/th&gt;
&lt;th&gt;Output per 1M&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;gpt-4.1-2025-04-14&lt;/td&gt;
&lt;td&gt;$25.00&lt;/td&gt;
&lt;td&gt;$3.00&lt;/td&gt;
&lt;td&gt;$0.75&lt;/td&gt;
&lt;td&gt;$12.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-4.1-mini-2025-04-14&lt;/td&gt;
&lt;td&gt;$5.00&lt;/td&gt;
&lt;td&gt;$0.80&lt;/td&gt;
&lt;td&gt;$0.20&lt;/td&gt;
&lt;td&gt;$3.20&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Prices checked November 15, 2026 on &lt;a href="https://platform.openai.com/docs/pricing" rel="noopener noreferrer"&gt;OpenAI's pricing page&lt;/a&gt;, standard tier. The Batch tier halves input and output prices on fine-tuned models but keeps the same training price.&lt;/p&gt;

&lt;h2&gt;
  
  
  Can you still fine-tune on OpenAI?
&lt;/h2&gt;

&lt;p&gt;Only if your organization already was. OpenAI's &lt;a href="https://developers.openai.com/api/docs/deprecations#update-to-openais-self-serve-fine-tuning" rel="noopener noreferrer"&gt;deprecations page&lt;/a&gt; lists the timeline:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;May 7, 2026: organizations that had never run fine-tuning can no longer create jobs.&lt;/li&gt;
&lt;li&gt;July 2, 2026: organizations with no fine-tuned model inference in the past 60 days lose job creation.&lt;/li&gt;
&lt;li&gt;January 6, 2027: active existing customers can no longer create new fine-tuning jobs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Inference keeps working until the base model shuts down. The &lt;a href="https://developers.openai.com/api/docs/guides/supervised-fine-tuning" rel="noopener noreferrer"&gt;supervised fine-tuning guide&lt;/a&gt; says gpt-4.1, gpt-4.1-mini and their fine-tuned versions shut down on January 14, 2027. That date matters more than any price. A fine-tuned OpenAI model is a rented artifact with an end date, while a LoRA adapter you train on an open model is a file you keep.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does the same dataset cost on each side?
&lt;/h2&gt;

&lt;p&gt;We priced one dataset four ways. It is 25 million tokens of examples trained for 2 epochs, so 50 million training tokens. That is the worked example from our &lt;a href="https://www.nodus-compute.ai/blog/cheapest-way-to-fine-tune-llama-on-h100/" rel="noopener noreferrer"&gt;Llama on H100 cost post&lt;/a&gt;, which used a published torchtune throughput as a stand-in rather than a run we measured.&lt;/p&gt;

&lt;p&gt;The arithmetic, once:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;OpenAI gpt-4.1-mini: 50 x $5.00 = $250.00.&lt;/li&gt;
&lt;li&gt;OpenAI gpt-4.1: 50 x $25.00 = $1,250.00.&lt;/li&gt;
&lt;li&gt;Llama 3.1 8B LoRA on 1 x H100 on Nodus: 50,000,000 / 4,699 tokens per second = 2.96 hours, plus 15 minutes of startup, at $2.60/hr = $8.35.&lt;/li&gt;
&lt;li&gt;Llama 3.3 70B LoRA on 1 x B200 on Nodus: our &lt;a href="https://www.nodus-compute.ai/blog/fine-tune-70b-model-cost/" rel="noopener noreferrer"&gt;70B fine-tuning cost post&lt;/a&gt; estimates $19.16 of training for 20 million tokens. Scaled to 50 million that is $19.16 x 2.5 = $47.90, plus 15 minutes at $5.50/hr ($1.375), so $49.28.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhn9bwfvqtk7vzj49hhng.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhn9bwfvqtk7vzj49hhng.png" alt="Training cost for 50 million training tokens: OpenAI gpt-4.1 $1,250.00, OpenAI gpt-4.1-mini $250.00, Llama 3.3 70B LoRA on one B200 on Nodus $49.28 and Llama 3.1 8B LoRA on one H100 on Nodus $8.35." width="800" height="328"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;At list prices, a 70B open model fine-tunes for about a twenty-fifth of what gpt-4.1 charges for the same tokens.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Treat the Nodus rows as planning numbers. The 8B row rests on a throughput from different hardware and the 70B row on FLOP math at 30% utilization. Neither one compares model quality: an 8B Llama is not a drop-in for gpt-4.1-mini on every task. What the gap shows is scale.&lt;/p&gt;

&lt;p&gt;Even if your real throughput were five times worse, the open-model rows would still cost a fraction of the OpenAI ones. Nodus rates are listed on the &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing page&lt;/a&gt;, checked November 15, 2026.&lt;/p&gt;
&lt;h2&gt;
  
  
  How fast does a GPU have to train to beat per-token pricing?
&lt;/h2&gt;

&lt;p&gt;Very slowly. Divide the hourly rate by the per-token price and you get the training speed at which the two cost the same:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Comparison&lt;/th&gt;
&lt;th&gt;Break-even training speed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;H100 at $2.60/hr vs gpt-4.1-mini at $5.00 per 1M&lt;/td&gt;
&lt;td&gt;520,000 tokens/hr, about 144 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;H100 at $2.60/hr vs gpt-4.1 at $25.00 per 1M&lt;/td&gt;
&lt;td&gt;104,000 tokens/hr, about 29 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B200 at $5.50/hr vs gpt-4.1 at $25.00 per 1M&lt;/td&gt;
&lt;td&gt;220,000 tokens/hr, about 61 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For comparison, the torchtune figure behind our 8B example is 4,699 tokens per second. A LoRA run that trains slower than 144 tokens per second on an H100 has a setup problem worth fixing, not a pricing problem. Our &lt;a href="https://www.nodus-compute.ai/blog/best-platform-to-fine-tune-open-source-llms/" rel="noopener noreferrer"&gt;best fine-tuning platform guide&lt;/a&gt; runs the same division against cheaper per-token LoRA prices, where the bar is much higher.&lt;/p&gt;
&lt;h2&gt;
  
  
  When is serving your own fine-tuned model cheaper?
&lt;/h2&gt;

&lt;p&gt;Serving is where per-token pricing can win, because a GPU you keep up costs the same at 3 a.m. as at noon. Assume the 3 input tokens per output token that Nodus uses as its reference mix in the &lt;a href="https://www.nodus-compute.ai/docs/guides/inference/" rel="noopener noreferrer"&gt;inference guide&lt;/a&gt;. The blended price per 1M tokens is then:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fine-tuned gpt-4.1-mini: (3 x $0.80 + $3.20) / 4 = $1.40.&lt;/li&gt;
&lt;li&gt;Fine-tuned gpt-4.1: (3 x $3.00 + $12.00) / 4 = $5.25.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One NVIDIA L40S, listed at $0.793/hr on the Nodus pricing page, costs $0.793 x 720 hours = $570.96 a month. A 48 GB card holds an 8B model's 16 GB of 16-bit weights with room for the KV cache. The break-even volume is the monthly GPU cost divided by the blended price:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Serving GPU, 720 hours&lt;/th&gt;
&lt;th&gt;Monthly cost&lt;/th&gt;
&lt;th&gt;Break-even vs ft gpt-4.1-mini&lt;/th&gt;
&lt;th&gt;Break-even vs ft gpt-4.1&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;L40S at $0.793/hr&lt;/td&gt;
&lt;td&gt;$570.96&lt;/td&gt;
&lt;td&gt;408M tokens/month (157 tokens/s)&lt;/td&gt;
&lt;td&gt;109M tokens/month (42 tokens/s)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;H100 at $2.60/hr&lt;/td&gt;
&lt;td&gt;$1,872.00&lt;/td&gt;
&lt;td&gt;1,337M tokens/month (516 tokens/s)&lt;/td&gt;
&lt;td&gt;357M tokens/month (138 tokens/s)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flb78comte4ew2n49j2fp.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flb78comte4ew2n49j2fp.png" alt="Monthly serving cost from 0 to 600 million tokens: fine-tuned gpt-4.1 rises to $3,150, fine-tuned gpt-4.1-mini rises to $840, and one L40S on Nodus stays flat at $570.96." width="800" height="416"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Against fine-tuned gpt-4.1, one L40S pays for itself at about 109M tokens a month.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;At 600M tokens a month, fine-tuned gpt-4.1-mini costs $840 and fine-tuned gpt-4.1 costs $3,150, against $570.96 for the L40S. The tokens-per-second column is the average your GPU must sustain around the clock, so measure your own serving throughput at your real prompt lengths before you commit. Serving several adapters on one GPU moves the line further: vLLM &lt;a href="https://docs.vllm.ai/en/latest/features/lora.html" rel="noopener noreferrer"&gt;serves LoRA adapters per request&lt;/a&gt; on top of one base model.&lt;/p&gt;
&lt;h2&gt;
  
  
  Where is OpenAI fine-tuning the better choice?
&lt;/h2&gt;

&lt;p&gt;There are honest cases, mostly for teams that already have it.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You already run a fine-tuned gpt-4.1 or gpt-4.1-mini that passes your evals, and your volume is below the break-even above. Keep it until you have a replacement ready, and plan for the January 14, 2027 shutdown.&lt;/li&gt;
&lt;li&gt;Your dataset is tiny. OpenAI's guide reports improvements from 50 to 100 examples. At 50 examples of 1,000 tokens for 3 epochs, training is 150,000 tokens, or $0.75 on gpt-4.1-mini. At that size the cost difference is cents and the deciding factor is engineering time.&lt;/li&gt;
&lt;li&gt;Your traffic is low and spiky. Per-token serving bills nothing when nobody calls the model; a dedicated GPU bills every hour it is up.&lt;/li&gt;
&lt;li&gt;Your task needs the base model's quality and no open model you tested matches it on your held-out set. A cheaper model that fails your evals is not cheaper.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For new projects the first case no longer applies, and the last two are reasons to compare serving options, not reasons to wait for a platform that is closing.&lt;/p&gt;
&lt;h2&gt;
  
  
  How do you run the open-model version on Nodus?
&lt;/h2&gt;

&lt;p&gt;Nodus runs fine-tuning as a TrainingJob on the managed &lt;code&gt;nodus/sft&lt;/code&gt; runtime, which supports full fine-tuning, LoRA and QLoRA. This is the LoRA example from the &lt;a href="https://www.nodus-compute.ai/docs/guides/training/" rel="noopener noreferrer"&gt;training guide&lt;/a&gt;, copied as published. Swap in your model and data:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;nodus.dev/v1beta1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;TrainingJob&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;support-sft&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;runtime&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;nodus/sft&lt;/span&gt;
  &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;uri&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca&lt;/span&gt;
    &lt;span class="na"&gt;secret&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;hf-token&lt;/span&gt;            &lt;span class="c1"&gt;# only for gated repositories
&lt;/span&gt;  &lt;span class="na"&gt;data&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;volume&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;support-chats&lt;/span&gt;       &lt;span class="c1"&gt;# a Volume holding chats.jsonl
&lt;/span&gt;    &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;chats.jsonl&lt;/span&gt;
    &lt;span class="na"&gt;format&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;JSONL&lt;/span&gt;
    &lt;span class="na"&gt;columns&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;prompt:&lt;/span&gt; &lt;span class="s"&gt;question&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;completion:&lt;/span&gt; &lt;span class="s"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="na"&gt;parameters&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2000&lt;/span&gt;
    &lt;span class="na"&gt;lora&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;r:&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;alpha:&lt;/span&gt; &lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="na"&gt;maxCostUSD&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"20.00"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Price it before anything runs, then start it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;$ nodus apply -f sft.yaml --dry-run -o yaml     &lt;span class="c"&gt;# status.estimate: cost to completion, first hold, start time&lt;/span&gt;
$ nodus apply -f sft.yaml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;maxCostUSD&lt;/code&gt; caps the whole job and suspends it at a checkpoint rather than failing it. While it trains, &lt;code&gt;nodus describe&lt;/code&gt; shows tokens per second, which is the number to plug into the break-even tables above. The adapter lands under &lt;code&gt;outputs/adapter/&lt;/code&gt;, and you download it with &lt;code&gt;nodus cp&lt;/code&gt;, per the &lt;a href="https://www.nodus-compute.ai/docs/reference/runtimes/sft/" rel="noopener noreferrer"&gt;sft runtime reference&lt;/a&gt;. TrainingJobs are Beta, so read the dry run before the real run.&lt;/p&gt;

&lt;p&gt;A practical decision rule from the numbers above:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;New project, or an OpenAI fine-tune you would need to retrain after January 6, 2027: fine-tune an open model.&lt;/li&gt;
&lt;li&gt;Existing OpenAI fine-tune: compare its monthly tokens with the serving break-even. Above it, move now; below it, move before the shutdown, and look at per-token open-model serving.&lt;/li&gt;
&lt;li&gt;Either way, export the training data you used with OpenAI. It is the one asset that carries over unchanged.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  How much does fine-tuning gpt-4.1-mini cost for 1,000 examples?
&lt;/h3&gt;

&lt;p&gt;Multiply the tokens by the epochs. At 1,000 examples of 1,000 tokens and 3 epochs, that is 3 million training tokens, or $15 at $5.00 per 1M. The same run costs $75 on gpt-4.1.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do more epochs raise the OpenAI bill?
&lt;/h3&gt;

&lt;p&gt;Yes, linearly, because every epoch re-bills the whole dataset. OpenAI's cookbook estimate starts from 3 epochs and adjusts for very small or very large datasets, so check the epoch count before you launch.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is a fine-tuned open model as good as a fine-tuned GPT model?
&lt;/h3&gt;

&lt;p&gt;Sometimes, and only your held-out evals can say. Nodus can validate during training and fail a job that does not beat the starting model's validation loss. Still, the training guide says a separate untouched test set must establish task quality.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I keep using my fine-tuned OpenAI model after January 6, 2027?
&lt;/h3&gt;

&lt;p&gt;Yes, for inference, until its base model shuts down. For gpt-4.1 and gpt-4.1-mini that is January 14, 2027, so the window after the training cutoff is eight days.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is it cheaper to self-host than to pay per token for an open model?
&lt;/h3&gt;

&lt;p&gt;Only above the break-even volume for your GPU. Below it, a per-token hosted endpoint is usually cheaper, and the Nodus &lt;a href="https://www.nodus-compute.ai/docs/guides/inference/" rel="noopener noreferrer"&gt;inference API&lt;/a&gt; serves open-weight catalog models per token.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.nodus-compute.ai/blog/openai-fine-tuning-vs-open-models/" rel="noopener noreferrer"&gt;Nodus blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>openai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>ai</category>
    </item>
    <item>
      <title>What are the best Vertex AI alternatives for model training?</title>
      <dc:creator>Viswaretas Kotra</dc:creator>
      <pubDate>Mon, 16 Nov 2026 00:05:28 +0000</pubDate>
      <link>https://dev.to/viswaretas_kotra/what-are-the-best-vertex-ai-alternatives-for-model-training-4fb9</link>
      <guid>https://dev.to/viswaretas_kotra/what-are-the-best-vertex-ai-alternatives-for-model-training-4fb9</guid>
      <description>&lt;p&gt;If you mostly use Vertex AI to run custom training jobs, the strongest alternative on price is Nodus: a 24-hour run on 8 NVIDIA H100 GPUs costs $499.20 at the Nodus listed rate, against $2,424.18 as a Vertex AI custom training job. Nodus can also run single-node jobs inside your own Google Cloud project, so your GCP credits are spent before Nodus capacity. Modal, RunPod, Amazon SageMaker AI and Azure Machine Learning are the other honest options, and Vertex AI stays the better choice when its managed datasets, Model Registry and TPUs are part of how you ship.&lt;/p&gt;

&lt;p&gt;We build Nodus, so weigh our view accordingly. Every Google and competitor fact below links to the provider's own page, checked November 15, 2026, and the arithmetic is written out so you can redo it with your own region.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is Vertex AI called now?
&lt;/h2&gt;

&lt;p&gt;Google now documents Vertex AI as Gemini Enterprise Agent Platform. Its &lt;a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/overview" rel="noopener noreferrer"&gt;platform overview&lt;/a&gt; calls it "an evolution of Vertex AI", and the pricing page at the old Vertex AI address is now titled &lt;a href="https://cloud.google.com/vertex-ai/pricing" rel="noopener noreferrer"&gt;Gemini platform pricing&lt;/a&gt;. Custom training is called serverless training in the new &lt;a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/training/overview" rel="noopener noreferrer"&gt;training docs&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The product underneath is the same. You still submit a custom job, a hyperparameter tuning job or a training pipeline, and you still pay per machine-hour. We say Vertex AI below because that is what most people still search for.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does one training run cost on Vertex AI and its alternatives?
&lt;/h2&gt;

&lt;p&gt;We priced one job everywhere: 8 x H100 for 24 hours, which is 192 GPU-hours, at each provider's listed on-demand rate. This is GPU compute only. Storage, egress and other services are excluded on every row.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Platform&lt;/th&gt;
&lt;th&gt;Listed rate&lt;/th&gt;
&lt;th&gt;Per H100-hour&lt;/th&gt;
&lt;th&gt;8 x H100 for 24 hours&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Nodus&lt;/td&gt;
&lt;td&gt;$2.60 per GPU-hour&lt;/td&gt;
&lt;td&gt;$2.60&lt;/td&gt;
&lt;td&gt;$499.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Modal&lt;/td&gt;
&lt;td&gt;$0.001097 per GPU-second&lt;/td&gt;
&lt;td&gt;$3.95&lt;/td&gt;
&lt;td&gt;$758.25&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RunPod (Secure Cloud)&lt;/td&gt;
&lt;td&gt;$4.39 per GPU-hour, H100 SXM&lt;/td&gt;
&lt;td&gt;$4.39&lt;/td&gt;
&lt;td&gt;$842.88&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Amazon SageMaker AI&lt;/td&gt;
&lt;td&gt;$63.296 per hour, &lt;code&gt;ml.p5.48xlarge&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;$7.91&lt;/td&gt;
&lt;td&gt;$1,519.10&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Google Compute Engine&lt;/td&gt;
&lt;td&gt;$88.49 per hour, &lt;code&gt;a3-highgpu-8g&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;$11.06&lt;/td&gt;
&lt;td&gt;$2,123.76&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Azure Machine Learning&lt;/td&gt;
&lt;td&gt;$98.32 per hour, ND96isr H100 v5&lt;/td&gt;
&lt;td&gt;$12.29&lt;/td&gt;
&lt;td&gt;$2,359.68&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vertex AI custom training&lt;/td&gt;
&lt;td&gt;$101.007 per hour, &lt;code&gt;a3-highgpu-8g&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;$12.63&lt;/td&gt;
&lt;td&gt;$2,424.18&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Prices checked November 15, 2026: &lt;a href="https://www.nodus-compute.ai/pricing/" rel="noopener noreferrer"&gt;Nodus pricing&lt;/a&gt;, &lt;a href="https://modal.com/pricing" rel="noopener noreferrer"&gt;Modal pricing&lt;/a&gt;, &lt;a href="https://www.runpod.io/pricing" rel="noopener noreferrer"&gt;RunPod pricing&lt;/a&gt;, the &lt;a href="https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonSageMaker/current/us-east-1/index.csv" rel="noopener noreferrer"&gt;AWS SageMaker price list&lt;/a&gt; (US East, N. Virginia), &lt;a href="https://cloud.google.com/products/compute/pricing/accelerator-optimized" rel="noopener noreferrer"&gt;Compute Engine accelerator-optimized pricing&lt;/a&gt; (Iowa, us-central1), Azure's &lt;a href="https://prices.azure.com/api/retail/prices" rel="noopener noreferrer"&gt;retail prices API&lt;/a&gt; for East US with no extra &lt;a href="https://azure.microsoft.com/en-us/pricing/details/machine-learning/" rel="noopener noreferrer"&gt;Azure Machine Learning charge&lt;/a&gt;, and &lt;a href="https://cloud.google.com/vertex-ai/pricing" rel="noopener noreferrer"&gt;Gemini platform pricing&lt;/a&gt; for custom training (Iowa, us-central1). Instance prices are divided by 8 GPUs; Modal's per-second price is multiplied by 3,600.&lt;/p&gt;

&lt;p&gt;The Vertex row works out as 24 x $101.007352832 = $2,424.18. The Nodus row is 192 x $2.60 = $499.20, which is $1,924.98 or 79% less. Modal is 192 x $3.9492 = $758.25, and RunPod Secure Cloud is 192 x $4.39 = $842.88.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fddpeei0jyf6ne3vwhez2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fddpeei0jyf6ne3vwhez2.png" alt="Cost of one 24-hour run on 8 x H100 at listed on-demand prices on November 15, 2026: Nodus $499.20, Modal $758.25, RunPod $842.88, Amazon SageMaker AI $1,519.10, Google Compute Engine $2,123.76, Azure Machine Learning $2,359.68 and Vertex AI custom training $2,424.18." width="800" height="460"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The same 192 H100-hours cost $1,924.98 less on Nodus than as a Vertex AI custom training job.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;One row deserves a second look. The &lt;code&gt;a3-highgpu-8g&lt;/code&gt; machine is $101.007 per hour as a Vertex training job and $88.49 per hour as a plain Compute Engine VM in the same region. That is $12.52 more per hour, about 14%, for the managed training service. Over our 24 hours it adds $300.42.&lt;/p&gt;
&lt;h2&gt;
  
  
  Do Vertex AI discounts close the gap?
&lt;/h2&gt;

&lt;p&gt;They narrow it, but not by much at these rates. Gemini platform pricing lists a Gemini Enterprise Flexible Savings Plan for the same machine at $90.907 per hour for one year and $80.806 per hour for three years. Our 24-hour run then costs $2,181.76 or $1,939.34.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fztk33yub6you8dg93ry2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fztk33yub6you8dg93ry2.png" alt="Cost of the same 24-hour 8 x H100 run: Vertex AI on-demand $2,424.18, Vertex AI with a 1-year savings plan $2,181.76, Vertex AI with a 3-year savings plan $1,939.34, and Nodus $499.20." width="800" height="416"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Even a 3-year Vertex savings plan leaves the run $1,440.14 above the Nodus listed rate.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Spot capacity is the other lever. Vertex custom training can use Spot VMs, billed at Compute Engine Spot prices plus a Vertex management fee, which the training accelerator table on &lt;a href="https://cloud.google.com/vertex-ai/pricing" rel="noopener noreferrer"&gt;Gemini platform pricing&lt;/a&gt; lists at $1.47 per H100-hour. Compute Engine lists A3 High Spot at $52.70 per hour. The catch is what happens on preemption, which is the next question.&lt;/p&gt;
&lt;h2&gt;
  
  
  What happens to a Vertex AI training job when a machine goes away?
&lt;/h2&gt;

&lt;p&gt;It starts your code again from the beginning. Google's &lt;a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/training/code-requirements" rel="noopener noreferrer"&gt;training code requirements&lt;/a&gt; say Vertex VMs restart occasionally, for example for maintenance, and that a restarted job runs your code from its start. Google asks you to export progress to Cloud Storage at least every four hours and to reload it on startup. The &lt;code&gt;AIP_CHECKPOINT_DIR&lt;/code&gt; variable gives you a Cloud Storage path for that.&lt;/p&gt;

&lt;p&gt;On Spot, the &lt;a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/training/use-spot-vms" rel="noopener noreferrer"&gt;Spot VM docs&lt;/a&gt; say a preempted job fails with a &lt;code&gt;STOCKOUT&lt;/code&gt; error and Compute Engine tries to restart it up to six times. You are not billed while the job is queued or preempted.&lt;/p&gt;

&lt;p&gt;Nodus works on the same file contract with less code. You write model, optimizer and step to &lt;code&gt;/nodus/state&lt;/code&gt;. Nodus copies that directory while the job runs, and when capacity is reclaimed the next attempt starts with it restored, per the &lt;a href="https://www.nodus-compute.ai/docs/guides/checkpoints/" rel="noopener noreferrer"&gt;checkpoints guide&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Your script still has to load what it finds there. We covered the pattern step by step in &lt;a href="https://www.nodus-compute.ai/blog/resume-training-after-spot-preemption/" rel="noopener noreferrer"&gt;How do you make spot GPU training survive preemption?&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;
  
  
  What do you lose and keep when you leave Vertex AI?
&lt;/h2&gt;

&lt;p&gt;You keep your container and your training script. You lose the Vertex services wrapped around them. That second list decides whether a move pays off, so check it before you price anything.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Vertex AI piece&lt;/th&gt;
&lt;th&gt;On Nodus&lt;/th&gt;
&lt;th&gt;What changes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Custom job&lt;/td&gt;
&lt;td&gt;A Job, started with &lt;code&gt;nodus run&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Same image and command&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;AIP_CHECKPOINT_DIR&lt;/code&gt; (Cloud Storage)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;/nodus/state&lt;/code&gt;, in &lt;code&gt;NODUS_STATE_DIR&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Local path, saved and restored by Nodus&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;AIP_MODEL_DIR&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;--output NAME=/path&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Outputs instead of a Cloud Storage URI&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hyperparameter tuning job&lt;/td&gt;
&lt;td&gt;A Sweep&lt;/td&gt;
&lt;td&gt;Every combination you list, not a Bayesian search&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Training pipeline&lt;/td&gt;
&lt;td&gt;A Pipeline&lt;/td&gt;
&lt;td&gt;Stages share one spending cap&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Custom container in a registry&lt;/td&gt;
&lt;td&gt;The same image, via a registry Secret or &lt;code&gt;nodus image push&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;No rebuild&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Managed datasets, Model Registry&lt;/td&gt;
&lt;td&gt;No direct equivalent&lt;/td&gt;
&lt;td&gt;Keep them on Google Cloud&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TPUs&lt;/td&gt;
&lt;td&gt;Not offered&lt;/td&gt;
&lt;td&gt;Stay on Google Cloud&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Sources: Google's &lt;a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/training/overview" rel="noopener noreferrer"&gt;training overview&lt;/a&gt;, &lt;a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/training/hyperparameter-tuning-overview" rel="noopener noreferrer"&gt;hyperparameter tuning overview&lt;/a&gt; and &lt;a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/training/code-requirements" rel="noopener noreferrer"&gt;code requirements&lt;/a&gt;; Nodus &lt;a href="https://www.nodus-compute.ai/docs/reference/cli/nodus_run/" rel="noopener noreferrer"&gt;&lt;code&gt;nodus run&lt;/code&gt; reference&lt;/a&gt;, &lt;a href="https://www.nodus-compute.ai/docs/guides/sweeps/" rel="noopener noreferrer"&gt;Sweeps&lt;/a&gt;, &lt;a href="https://www.nodus-compute.ai/docs/guides/pipelines/" rel="noopener noreferrer"&gt;Pipelines&lt;/a&gt; and &lt;a href="https://www.nodus-compute.ai/docs/guides/images/" rel="noopener noreferrer"&gt;Images&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Two rows need care. Vertex hyperparameter tuning chooses trials with search algorithms, including Bayesian optimization. A Nodus Sweep runs the grid you write down and reports cost, time and throughput per cell, which suits small grids and GPU comparisons better than wide searches. And if your models go from a training pipeline into the Model Registry and then to a Vertex endpoint, leaving training means rebuilding that handoff yourself.&lt;/p&gt;
&lt;h2&gt;
  
  
  Can you leave Vertex AI and still spend your GCP credits?
&lt;/h2&gt;

&lt;p&gt;Yes, for single-node work. Nodus can connect a GCP project and run your Jobs and other single-node work in it first, so Google Cloud credits are used before Nodus capacity, per the &lt;a href="https://www.nodus-compute.ai/docs/guides/pools/cloud-accounts/" rel="noopener noreferrer"&gt;cloud accounts docs&lt;/a&gt;. This is a Beta feature. Multi-node runs and Sandboxes use Nodus capacity.&lt;/p&gt;

&lt;p&gt;The bill splits in two. Google bills the machine and disk at your project's Compute Engine rates, so no Vertex training premium applies. Nodus adds a Route fee of $0.02 per device-hour on GPU jobs. For our 192 GPU-hours, that is 192 x $0.02 = $3.84.&lt;/p&gt;

&lt;p&gt;Setup is an OAuth grant limited to read-only scopes, then a separate project-owner approval that creates a dedicated network and a service account limited to machines Nodus creates. Machines run in us-central1, us-east1, us-east4 or us-west1, subject to your project's GPU quota. When the project is full or out of credit, you choose whether work moves to Nodus capacity at Nodus rates or waits. After setup, the CLI step from the docs is one line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;nodus cloud connect gcp gcp-lab --credit-limit 500 --gcp-project lab-ml-prod
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you hold Google credits, this is usually the cheapest way out of Vertex training. Our &lt;a href="https://www.nodus-compute.ai/blog/startup-gpu-credits/" rel="noopener noreferrer"&gt;startup GPU credits guide&lt;/a&gt; covers how to get and stack them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which Vertex AI alternative should you pick?
&lt;/h2&gt;

&lt;p&gt;Start from the reason you are leaving. This is the rule we give teams who ask us.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Your release process depends on managed datasets, the Model Registry, Vertex endpoints or TPUs: stay on Vertex AI, and move only standalone training jobs if price matters.&lt;/li&gt;
&lt;li&gt;You hold GCP credits: connect the project to Nodus, spend the credits on single-node jobs first, and let overflow decide what happens after.&lt;/li&gt;
&lt;li&gt;You want the lowest listed H100 price and runs that restore their own checkpoints: Nodus.&lt;/li&gt;
&lt;li&gt;Your code is Python functions and you also serve models from the same place: &lt;a href="https://modal.com/pricing" rel="noopener noreferrer"&gt;Modal&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;You want a GPU pod you configure and manage yourself: &lt;a href="https://www.runpod.io/pricing" rel="noopener noreferrer"&gt;RunPod&lt;/a&gt;. Our &lt;a href="https://www.nodus-compute.ai/blog/nodus-vs-runpod/" rel="noopener noreferrer"&gt;Nodus vs RunPod&lt;/a&gt; post compares the two directly.&lt;/li&gt;
&lt;li&gt;Your company is committed to AWS or Azure spend: Amazon SageMaker AI or Azure Machine Learning. The &lt;a href="https://www.nodus-compute.ai/blog/sagemaker-alternatives-2026/" rel="noopener noreferrer"&gt;SageMaker alternatives&lt;/a&gt; post covers that side.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If your main question is raw Compute Engine GPUs rather than Vertex training, read &lt;a href="https://www.nodus-compute.ai/blog/nodus-vs-google-cloud-gpus/" rel="noopener noreferrer"&gt;Nodus vs Google Cloud GPUs&lt;/a&gt; instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where is Vertex AI the better choice?
&lt;/h2&gt;

&lt;p&gt;Vertex AI wins when the platform does real work for you. If a training pipeline reads a managed dataset, writes a model into the Model Registry and deploys it to an endpoint, that chain is what you are paying for. Rebuilding it on any alternative costs engineering time.&lt;/p&gt;

&lt;p&gt;It also wins for TPU work. Vertex training supports TPU VMs, and Nodus does not offer TPUs. Multi-node GPU training inside Google Cloud is another case: the &lt;a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/training/overview" rel="noopener noreferrer"&gt;training overview&lt;/a&gt; documents Reduction Server, an all-reduce algorithm for multi-node training on NVIDIA GPUs, while our GCP project routing covers single-node work only.&lt;/p&gt;

&lt;p&gt;Finally, money already committed changes the math. A Gemini Enterprise savings plan or a negotiated Google agreement can make Vertex the cheaper option for your company even when list prices say otherwise. Price the move against what you actually pay, not against the on-demand row.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you move a Vertex AI training job to Nodus?
&lt;/h2&gt;

&lt;p&gt;Change the checkpoint path, then run the same image with a cost cap. Vertex scripts read &lt;code&gt;AIP_CHECKPOINT_DIR&lt;/code&gt;. On Nodus, the &lt;a href="https://www.nodus-compute.ai/docs/guides/checkpoints/" rel="noopener noreferrer"&gt;checkpoints guide&lt;/a&gt; uses this line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;

&lt;span class="c1"&gt;# Vertex AI: os.environ["AIP_CHECKPOINT_DIR"], a Cloud Storage URI&lt;/span&gt;
&lt;span class="no"&gt;STATE_DIR&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"NODUS_STATE_DIR"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"/nodus/state"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then install the CLI, sign in, check the estimate with a dry run and submit, using flags from the &lt;a href="https://www.nodus-compute.ai/docs/reference/cli/nodus_run/" rel="noopener noreferrer"&gt;&lt;code&gt;nodus run&lt;/code&gt; reference&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install nodus-compute
nodus login
nodus run --gpu H100:8 --image nodus/pytorch --max-cost 550 --dry-run -- torchrun train.py
nodus run --gpu H100:8 --image nodus/pytorch --max-cost 550 -d -- torchrun train.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;--max-cost 550&lt;/code&gt; cap sits just above our $499.20 example, so a run that goes wrong is suspended at the cap instead of running on. If your data stays in Cloud Storage, store a Google credential as a &lt;a href="https://www.nodus-compute.ai/docs/guides/secrets/" rel="noopener noreferrer"&gt;Secret&lt;/a&gt; and read your &lt;code&gt;gs://&lt;/code&gt; paths from the script as you do today.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Is Gemini Enterprise Agent Platform a different product from Vertex AI?
&lt;/h3&gt;

&lt;p&gt;Google describes it as an evolution of Vertex AI, not a separate service. Custom training, hyperparameter tuning and training pipelines are documented under the new name, and their prices are on the page at the old Vertex AI pricing address.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does Vertex AI charge for time a training job spends waiting?
&lt;/h3&gt;

&lt;p&gt;Gemini platform pricing says training is charged in 30-second increments with no minimum usage duration. On Spot VMs, Google says you are not billed while the job is queued or preempted.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the cheapest Vertex AI alternative for H100 training?
&lt;/h3&gt;

&lt;p&gt;Of the seven platforms priced on November 15, 2026, Nodus had the lowest listed on-demand H100 rate at $2.60 per GPU-hour. If you have Google credits, running in your own project through Nodus means the credits pay Google for the machine, and the only Nodus charge is the $0.02 per device-hour Route fee.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I use Nodus and keep Vertex AI for serving?
&lt;/h3&gt;

&lt;p&gt;Yes. Have your script upload the trained weights to Cloud Storage, then import them into the Model Registry as you would any other artifact. Google's docs cover &lt;a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/training/exporting-model-artifacts" rel="noopener noreferrer"&gt;importing model artifacts&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.nodus-compute.ai/blog/vertex-ai-alternatives/" rel="noopener noreferrer"&gt;Nodus blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>googlecloud</category>
      <category>mlops</category>
      <category>gpu</category>
    </item>
  </channel>
</rss>
