DEV Community: Kavya

How to Monetize Idle GPUs in 2026: A Technical Guide for Data Center Operators

Kavya — Tue, 06 Oct 2026 12:42:54 +0000

If you operate NVIDIA GPU hardware and it is sitting at low utilization, you are paying to run a data center for an empty building.

Data centers average 12 to 18% GPU utilization. The rest of the time the hardware is powered on, cooling systems running, electricity meter ticking. No revenue. Meanwhile, AI developers cannot find compute. H100 rental rates climbed from $1.70/hr in late 2025 to $2.35/hr by March 2026. On-demand capacity is effectively sold out on most major neocloud platforms.

This post covers how the packet.ai provider marketplace works technically, what hardware qualifies, how the provisioning layer is set up, and what the economics actually look like.

How the Provider Marketplace Model Works

packet.ai sits between hardware owners (providers) and AI developers who need compute (buyers). Providers list their NVIDIA GPUs. The marketplace handles customer discovery, billing, VM provisioning, and support. Providers earn a share of the per-hour rental rate each time their hardware serves a workload.

The key technical piece is the hosted.ai provisioning agent, a lightweight process that runs on your hosts and exposes a minimal API for VM lifecycle management only. It handles:

VM creation and teardown on demand
GPU passthrough assignment per workload
Network isolation between tenants
Heartbeat and availability reporting to the marketplace scheduler

It does not transmit customer workload data back to packet.ai. Source-level documentation of the agent is provided to all vetted providers who request it.

You keep full root access, BMC access, and physical access to the hardware throughout. The agent does not elevate packet.ai's permissions on your systems beyond VM lifecycle calls.

Workload Isolation

Each customer workload runs in an isolated container with dedicated GPU passthrough. On the Dedicated tier, no multi-tenant sharing of a single GPU between customers occurs. On the Dynamic tier, scheduler-enforced isolation keeps each workload's memory and compute separated even when co-located on shared hardware.

For providers running sensitive adjacent workloads on the same cluster, the isolation model is worth reviewing in detail during onboarding. packet.ai provides the agent documentation before you commit.

Hardware Requirements

Accepted GPUs, in order of rate band:

GPU	VRAM	Generation
NVIDIA B200	180 GB HBM3e	Blackwell
NVIDIA H200	141 GB HBM3e	Hopper
NVIDIA H100	80 GB HBM3	Hopper
NVIDIA A100 80G	80 GB HBM2e	Ampere
NVIDIA RTX 6000 Pro	96 GB GDDR7	Blackwell
NVIDIA L40S	48 GB GDDR6	Ada Lovelace
NVIDIA RTX 5090	32 GB GDDR7	Blackwell
NVIDIA RTX 4090	24 GB GDDR6X	Ada Lovelace

Consumer-grade hardware and home-lab setups do not qualify. The infrastructure bar is:

Network: 1 Gbps or faster stable uplink, clean peering, low latency and jitter. Multi-node training and distributed inference are network-sensitive.
Hosting: Colocation or owned datacenter. Not residential.
Power: N+1 redundancy preferred, not required. BIOS configured for sustained 100% GPU load without thermal throttling.
Uptime: 99% or higher. Historical monitoring data speeds up the onboarding review.

What the Economics Look Like

The blended average across all SKUs on the packet.ai provider network is $2.84/GPU/hr, paid bi-weekly via direct deposit (ACH for US providers, SWIFT for international).

At 80% utilization, a single GPU generates approximately $1,640/month. A 24-GPU node at the live network average of 87% utilization comes to $12,450/month.

Rate cards by SKU and region are provided during onboarding. Blackwell-generation hardware earns the highest rate bands. Payouts include a per-node CSV breakdown for accounting.

What packet.ai Handles

The reason most hardware operators don't build their own cloud product is that the commercial stack is expensive to build before the first dollar arrives. packet.ai absorbs:

Customer acquisition: thousands of KYC-verified AI developers and businesses already on the platform. No ads or sales cycle required from the provider.
Billing and collections: invoicing, payment collection, currency conversion, bi-weekly payout.
Customer support: 24/7 tier-1 and tier-2 support handled by packet.ai. Providers are escalated only for hardware-level incidents (node down, power event, fan failure).
Customer vetting: KYC-verified businesses only. AUP enforcement filters crypto miners and abuse traffic.
VM provisioning: the hosted.ai agent handles the full VM lifecycle automatically.

Getting From Idle Hardware to First Payout

Three steps, typically under seven days end to end:

Step 1: Apply

Submit your infrastructure details at packet.ai/for-providers. The review covers GPU model and BIOS revision, network quality, power setup, and hosting environment. Two-business-day turnaround.

Step 2: Install the provisioning agent

The hosted.ai agent installs on your hosts. Minimal API surface, audited codebase, source available to approved providers. Does not touch customer workloads or transmit data outside VM lifecycle calls.

Step 3: Go live

Your capacity appears on the marketplace. Real-time dashboard shows GPU-by-GPU utilization, occupancy state, and earnings. CSV export and API access available for monitoring integrations. First payout on the next bi-weekly cycle.

No long-term contracts. No minimum GPU count. No penalty for withdrawing capacity.

Withdrawing Capacity

Month-to-month terms. Withdraw at any time with no financial penalty. If a customer workload is active on your hardware when you exit, 14 days of notice lets the platform migrate it cleanly. After that, full control returns to you.

This is practical for operators running mixed workloads: research clusters idle between training runs, colocation tenants with subletting rights, facilities with seasonal demand patterns.

Where Demand Is Concentrated

Fastest time to full occupancy after going live, by region:

US East (Virginia): AI startups, enterprise inference, research labs
EU Frankfurt and Amsterdam: Enterprise AI, GDPR-compliant inference
UK London: Financial services, sovereign AI
US West (California, Oregon): AI-native startups, fine-tuning workloads
EU Dublin and Paris: Sovereign AI, European tech sector

APAC and LATAM providers are onboarded selectively based on active demand and latency profile.

The Short Version

GPU lead times hit 12 months in mid-2026. Demand is structural, not cyclical. If your NVIDIA hardware is at low utilization, there are paying workloads queued up for it right now.

Application review: two business days. First revenue: typically under a week from approval.

Apply at packet.ai/for-providers

Originally published on the packet.ai blog. packet.ai is a GPU cloud platform delivering NVIDIA B200, H100, A100, and RTX hardware at prices typically 50% or more below market, powered by hosted.ai's intelligent GPU scheduling platform.

packet.ai Token Factory: A Managed LLM Inference API from the Team That Cut GPU Prices by 50%+

Kavya — Mon, 21 Sep 2026 07:33:56 +0000

packet.ai Token Factory: A Managed LLM Inference API from the Team That Cut GPU Prices by 50%+

packet.ai Token Factory is a managed LLM inference API. OpenAI-compatible change

your base_url, swap your API key, and your existing SDK calls run against open-weight
models with no GPU to manage, no serving stack to maintain, and per-token billing that
drops to zero when your app is not generating tokens.

Token Factory is launching soon. The waitlist is live at
packet.ai/token-factory.
Here's why the pricing will be different from what you're used to.

The GPU pricing track record

packet.ai sells NVIDIA GPUs at prices that make most
developers check the rate card twice.

The NVIDIA B200 — 192GB HBM3e, Blackwell — is $3.75/hr
on Dynamic and $6.99/hr on Dedicated. RunPod lists it at $5.89/hr. Lambda Labs at $6.99/hr.

The NVIDIA RTX 6000 Pro — 96GB GDDR7 — is $0.66/hr
on Dynamic. RunPod charges $2.09/hr for the same card. That is a 68% difference on
the same silicon.

Current live rates vs the competition, sourced from packet.ai/pricing:

GPU	packet.ai	RunPod	Lambda Labs	vs RunPod
NVIDIA B200 (192GB HBM3e)	$3.75/hr Dynamic	$5.89/hr	$6.99/hr	36% cheaper
NVIDIA RTX 6000 Pro (96GB GDDR7)	$0.66/hr Dynamic	$2.09/hr	—	68% cheaper
NVIDIA A100 80GB	$1.43/hr Dedicated	$1.49/hr	$1.99/hr	28% vs Lambda
NVIDIA RTX 4090 (24GB GDDR6X)	$0.39/hr Dedicated	$0.69/hr	—	43% cheaper

Prices in USD/GPU-hr. Published starting rates, September 2026.

This is what the scheduler makes possible: co-locating
workloads that stress different GPU dimensions so the silicon never contends. Same peak
performance. Same VRAM. Lower price. That is the structural advantage.

The same reasoning is now going into inference.

Why managed LLM inference pricing is broken

If you're building any LLM-powered product at real volume, you've hit one of three walls.

Proprietary APIs. OpenAI, Anthropic, Google. The developer experience is clean.
The per-token cost at scale is not. Output tokens on frontier models compound fast
every agent call, RAG retrieval, and background summarization job adds to the bill.
And you're paying frontier rates for tasks that open-weight models handle just as well.

Self-hosting on GPU compute. Cheaper at steady-state volume. Also a part-time job.
You pick a model, stand up vLLM or TGI, configure batching and quantization, write
autoscaling logic, build monitoring, and respond to the 2am CUDA OOM that takes your
endpoint down. Every hour spent there is an hour not spent on what you're building.

Managed open-weight inference. Fireworks AI, Together AI, Groq, Replicate these
exist. But most land closer to proprietary API rates than to what self-hosting actually
costs. You're paying a GPU margin you can't see, on models whose serving cost has
dropped significantly since 2024.

Token Factory goes into the gap: managed inference, open-weight models, on packet.ai
hardware, priced with the same logic as the GPU products.

What is packet.ai Token Factory?

Token Factory is packet.ai's managed LLM inference API, built on the same GPU
infrastructure that powers the packet.ai cloud.

It exposes the same HTTP API OpenAI uses: POST /v1/chat/completions, the same
request schema (model, messages, temperature, tools), the same response format.
Switching requires three config changes: base_url, api_key, and model name.
Nothing else.

# Before
from openai import OpenAI
client = OpenAI(api_key="sk-...")

# After: same SDK, same code, one line changed
from openai import OpenAI
client = OpenAI(
    base_url="<token-factory-endpoint>",  # published in dash.packet.ai/docs at launch
    api_key="your-packet-api-key"
)

LangChain's ChatOpenAI, LlamaIndex's OpenAI LLM class, any direct API call - all
accept base_url as a single parameter. Chains, agents, RAG pipelines, and tool-calling
loops carry over without structural changes to your application.

What Token Factory includes at launch:

OpenAI-compatible endpoint (POST /v1/chat/completions)
Streaming via Server-Sent Events
Function calling for tool-use and agentic workflows
Per-token billing with scale-to-zero
US and EU regions
No GPU to provision, no serving stack to maintain

Who Token Factory is for

Token Factory is not right for every workload. Here is where it fits clearly and
where it does not.

Good fit:

Solo developers who cannot justify renting a dedicated GPU for one application
Teams whose OpenAI bill is growing faster than revenue
ML engineers building agentic applications with high per-user call volume
Any team running batch eval, RAG, summarization, classification, or extraction at scale
Teams reducing single-provider dependency and wanting a clean fallback endpoint

Not the right fit:

Workloads that require extended frontier-model reasoning (o3, o4 class)
Fine-tuning runs those need direct GPU access (use a Dynamic POD)
On-premise compliance requirements
Teams still under 1,000 calls per day, a proprietary API is cheaper to start

For the detailed breakdown of each developer profile and when self-hosting is
the better call, read Who Is Token Factory For?

Token Factory vs self-hosting vs proprietary APIs

Factor	Token Factory	Self-host on GPU	Proprietary API
Billing model	Per token, no idle cost	Per GPU-hour (always on)	Per token (frontier rates)
Setup time	Minutes (URL + key swap)	Hours to days (vLLM config)	Minutes
Scale-to-zero	Yes, zero cost when idle	Manual (stop the pod)	Yes
Infrastructure to manage	None	GPU + serving stack	None
OpenAI SDK compatible	Yes	Yes, via vLLM	Yes (native)
Function calling	Yes	Depends on config	Yes
Streaming	Yes	Yes, via vLLM	Yes
Best for	Solo devs, growing startups	High-volume steady-state	Prototyping, frontier tasks

If you outgrow Token Factory and want direct GPU access to run your own stack,
packet.ai's Dynamic GPU tier starts at
$0.66/hr and spins up in under 5 minutes.

How Token Factory will be priced

We are not publishing rates before launch.

Here is what you need to know in the meantime: the team that decided the B200 should
cost $3.75/hr instead of $5.89/hr is building Token Factory. The same reasoning applies.
Open-weight inference has improved faster than the price of calling it has dropped.
There is no structural reason managed open-weight inference has to cost anywhere near
what closed-model APIs charge. We do not think it should.

If you have been watching our GPU pricing, you have a
reference point for the direction Token Factory pricing will take.

Frequently asked questions

What is packet.ai Token Factory?
Token Factory is packet.ai's managed LLM inference API. It serves open-weight models
through a drop-in OpenAI-compatible endpoint, same API format, same SDK, change only
base_url and API key. Per-token billing with scale-to-zero, function calling, and
streaming supported at launch. No GPU or serving stack to manage. Built on packet.ai's
GPU infrastructure across US and EU regions.

How does Token Factory compare to Fireworks AI, Together AI, and Groq?
All four are managed inference options for open-weight models with OpenAI-compatible
endpoints. Fireworks AI and Together AI offer larger model catalogs. Groq is optimized
for raw throughput speed via dedicated LPU hardware. Token Factory's differentiation
is pricing the same cost-structure advantage that makes packet.ai GPU rentals
36-68% below RunPod on the same silicon is the foundation for how Token Factory
will be priced.

Does Token Factory support function calling and streaming?
Yes to both. Streaming runs via Server-Sent Events. Function calling follows the same
tool-use schema as OpenAI's API. Agent loops and tool-use workflows built on the OpenAI
format carry over without changes to application logic.

When does Token Factory launch and how do I get access?
Token Factory is launching soon. Access opens in waves starting with the waitlist.
Join at packet.ai/token-factory we'll email
when your spot is ready.

Get in before the first wave

Token Factory opens access to waitlist members first.

If per-token cost is a real number in your budget, or you have been looking for a
managed inference option that does not charge like it is pre-DeepSeek pricing still,
this is worth 30 seconds.

Join the Token Factory waitlist

Related: Who Is Token Factory For?
the full breakdown on which developer profiles Token Factory is and is not built for,
including when running your own stack on a Dynamic GPU
is the right call instead.

Best GPU Cloud Providers for AI in 2026: Developer Comparison With Code

Kavya — Tue, 25 Aug 2026 11:38:15 +0000

I Tested Every Major GPU Cloud So You Don't Have To

Last quarter my team spent three weeks benchmarking GPU clouds for our LLM inference pipeline. We had a simple requirement: run Llama 3 70B in production, keep latency under 2 seconds, and don't spend a fortune doing it.

What followed was a crash course in how differently each provider approaches the same problem.

Here's everything we learned, with actual code.

Who This Is For

If you're an ML engineer, backend developer, or startup CTO trying to figure out where to run your AI workloads in 2026, this is the guide I wish existed when we started. We cover:

Real provisioning workflows, not just marketing claims
Code snippets that actually run
Cost estimates based on real usage
Honest takes on where each provider wins and loses

TL;DR: Quick Provider Summary

Provider	Dev Experience	GPU Availability	B200?	Best Use Case
AWS	Moderate friction	Good with reservation	Preview	Enterprise MLOps
Azure	Moderate friction	Good with reservation	Preview	Azure OpenAI integration
GCP	Moderate friction	Good (TPU excellent)	Announced	JAX/TF training
CoreWeave	Good for large clusters	Strong (reserved)	Yes	Lab-scale training
Lambda Labs	Good	Moderate (books out)	No	Research workloads
RunPod	Good	Variable	No	Budget experiments
packet.ai	Very good	Strong	Yes	AI-native GPU compute

What "AI Workload" Actually Means in Practice

Before we get into providers, let's be specific. "AI workload" means different things to different teams:

Pre-training / full training: Hundreds to thousands of GPU-hours, multi-node clusters, InfiniBand networking, checkpoint storage. If you're doing this, you already know your requirements.

Fine-tuning: Single-node or small multi-GPU, 1-100 GPU-hours, high VRAM GPUs (A100 80GB, H100). Iteration speed matters more than raw cluster size.

Inference serving: Low latency, high throughput, VRAM-efficient, OpenAI API compatibility, autoscaling.

Evaluation and benchmarking: Short burst compute, flexibility over raw performance.

Most developers are in the fine-tuning or inference bucket. That's where the real provider differences show up.

The Real Developer Workflow: Deploying vLLM for LLM Inference

Here's a concrete example. Deploying Llama 3 70B with vLLM for inference is the same across every provider once you have SSH. The difference is how fast you get there and what it costs.

Step 1: SSH into your instance

ssh ubuntu@your-instance-ip

Step 2: Install vLLM

pip install vllm

Step 3: Run inference in Python

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    tensor_parallel_size=2,
    dtype="bfloat16",
    max_model_len=8192,
)

sampling_params = SamplingParams(
    temperature=0.7,
    max_tokens=512,
)

outputs = llm.generate(
    ["What is the difference between A100 and B200?"],
    sampling_params
)
print(outputs[0].outputs[0].text)

Step 4: Serve as OpenAI-compatible API

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-70B-Instruct \
    --tensor-parallel-size 2 \
    --host 0.0.0.0 \
    --port 8000

Step 5: Hit the endpoint with OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://your-instance-ip:8000/v1",
    api_key="not-needed-for-local",
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    messages=[
        {"role": "user", "content": "Explain GPU memory bandwidth in simple terms"}
    ],
)
print(response.choices[0].message.content)

This workflow runs identically on every provider that gives SSH access to an NVIDIA GPU. The differences come down to provisioning speed, GPU options, and hourly cost.

Provider Breakdown: The Developer Perspective

AWS

Amazon Web Services gives you GPU instances through EC2. The key instance types for AI:

p4d.24xlarge -- 8x A100 40GB, NVLink, roughly $32/hr on-demand
p4de.24xlarge -- 8x A100 80GB, NVLink, roughly $40/hr on-demand
p5.48xlarge -- 8x H100 80GB, NVLink + EFA, roughly $98/hr on-demand

aws ec2 run-instances \
    --image-id ami-0abcdef1234567890 \
    --instance-type p4d.24xlarge \
    --key-name your-key-pair \
    --security-group-ids sg-your-group \
    --subnet-id subnet-your-subnet

SageMaker wraps this with managed training, experiment tracking, model registry, and deployment pipelines. Genuinely useful if you need that. The friction: on-demand availability for large GPU instances is unreliable without a reservation. NVIDIA B200 is currently preview-only.

Dev experience verdict: If you're already AWS-native, it works. For raw GPU access without managed services, the friction is higher than it needs to be.

Google Cloud Platform

Google Cloud GPU instances run on A2 (A100) and A3 (H100) machine families. TPU access is unique to GCP.

gcloud compute instances create gpu-instance \
    --zone=us-central1-a \
    --machine-type=a2-highgpu-1g \
    --accelerator=count=1,type=nvidia-tesla-a100 \
    --image-family=tf-latest-gpu \
    --image-project=deeplearning-platform-release \
    --boot-disk-size=100GB \
    --metadata="install-nvidia-driver=True"

TPU access requires a completely separate workflow and JAX/TF framework commitment. If your stack is CUDA-based, TPUs require a meaningful porting effort.

Dev experience verdict: Strong for Vertex AI and JAX-native workloads. CUDA-based GPU access has the same friction as other hyperscalers.

CoreWeave

CoreWeave is Kubernetes-native. Their workflow uses GPU workloads as Kubernetes pods, which is excellent if your ML team already uses Kubernetes.

apiVersion: v1
kind: Pod
metadata:
  name: vllm-inference
spec:
  containers:
  - name: vllm
    image: vllm/vllm-openai:latest
    resources:
      limits:
        nvidia.com/gpu: "2"
        memory: "160Gi"
    command: ["python", "-m", "vllm.entrypoints.openai.api_server"]
    args:
    - "--model"
    - "meta-llama/Meta-Llama-3-70B-Instruct"
    - "--tensor-parallel-size"
    - "2"
  restartPolicy: Never

Their H100 and H200 capacity is legitimate. InfiniBand networking for multi-node training is available. The catch: casual on-demand access is harder than with providers that offer instant provisioning. NVIDIA B200 is available through reservation.

Dev experience verdict: Excellent for Kubernetes-native teams running large clusters. Less accessible for quick iteration or solo developers.

Lambda Labs

Lambda Labs has a clean API and competitive pricing. Their instance availability can be tight for popular GPU types.

import requests

response = requests.post(
    "https://cloud.lambdalabs.com/api/v1/instance-operations/launch",
    auth=("your-api-key", ""),
    json={
        "region_name": "us-east-1",
        "instance_type_name": "gpu_1x_a100_sxm4",
        "ssh_key_names": ["your-ssh-key"],
        "quantity": 1,
    }
)
print(response.json())

Strong for research use cases. No inference-as-a-service layer. NVIDIA B200 not yet available.

Dev experience verdict: Good for straightforward GPU access. Availability is the main friction point for popular instance types.

RunPod

RunPod is fast to spin up and cheap for experimentation.

pip install runpod

runpod config

runpod pod create \
    --gpu-type "NVIDIA A100 80GB" \
    --image runpod/pytorch:latest \
    --ports "8000/http" \
    --volume-size 50

For serverless inference RunPod is particularly developer-friendly:

import runpod

runpod.api_key = "your-api-key"

endpoint = runpod.Endpoint("your-endpoint-id")

run_request = endpoint.run(
    {"input": {"prompt": "Explain transformers in two sentences"}}
)
print(run_request.output())

Dev experience verdict: Best for quick experiments and budget inference. Third-party hardware tier has variable reliability. Not the right choice for production workloads where uptime matters.

packet.ai

packet.ai is the provider that surprised me the most during our benchmarking. The pitch is simple: dedicated NVIDIA GPUs, hourly billing, no reservation commitment, SSH-ready in minutes.

After instance provisioning, access is immediate:

ssh ubuntu@your-packet-instance-ip

Environment setup is clean:

sudo apt update && sudo apt install -y nvidia-cuda-toolkit

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

pip install vllm

nvidia-smi

No driver drama. No 15-minute wait. The GPU is just there.

For inference using Token Factory, packet.ai's OpenAI-compatible inference API:

from openai import OpenAI

# Switching from OpenAI to packet.ai Token Factory
# is literally just changing base_url and api_key
client = OpenAI(
    base_url="https://api.packet.ai/v1",
    api_key="your-packet-api-key",
)

response = client.chat.completions.create(
    model="llama-3-70b-instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What makes NVIDIA B200 different from H100?"}
    ],
    max_tokens=512,
    temperature=0.7,
)

print(response.choices[0].message.content)

That's the whole migration from OpenAI to packet.ai for inference. Base URL change and a key swap.

For fine-tuning on packet.ai's A100 or RTX 6000 Pro:

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model
import torch

model_name = "meta-llama/Meta-Llama-3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

peft_config = LoraConfig(
    task_type="CAUSAL_LM",
    r=16,
    lora_alpha=32,
    lora_dropout=0.1,
    target_modules=["q_proj", "v_proj"],
)
model = get_peft_model(model, peft_config)

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    warmup_steps=100,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_steps=500,
)

On packet.ai's RTX 6000 Pro (48GB VRAM), this handles models up to 13B parameters comfortably. On A100 (80GB), 70B models with quantization are doable.

The NVIDIA B200 availability on packet.ai is worth calling out separately. Most providers have B200 on a waitlist or behind a reservation requirement. packet.ai has it on hourly billing with no commitment. If you want to benchmark Blackwell GPU performance for your specific model today, this is your most practical path.

Dev experience verdict: The fastest path from "I need a GPU" to "I have a running GPU." Token Factory removes the need to manage inference infrastructure entirely if you don't want to. Pricing is published and predictable.

Choosing the Right GPU for Your Task

VRAM Requirements by Task

Task	Model Size	Min VRAM	Recommended Instance
Inference only	7B	16GB	RTX 4090
Inference only	70B (4-bit)	48GB	RTX 6000 Pro or L40S
Fine-tuning (LoRA)	7B	24GB	RTX 6000 Pro
Fine-tuning (LoRA)	70B	80GB	A100 80GB
Full fine-tuning	7B	80GB+	A100 80GB
Training	70B+	Multi-GPU	A100 or B200 cluster

Real Cost Estimate: Fine-tuning Llama 3 8B

Training with LoRA for 3 epochs on a 100K sample dataset takes roughly 8-12 GPU-hours on an A100.

AWS SageMaker ml.p4d.24xlarge: roughly $4-5 for compute, plus SageMaker overhead and egress
Lambda Labs A100: roughly $8-12 total
packet.ai RTX 6000 Pro: lower than A100 alternatives, published on pricing page

The RTX 6000 Pro is worth considering for fine-tuning runs that fit in 48GB VRAM. It's less expensive per hour than A100 while covering most LoRA fine-tuning tasks.

When to Self-Host Inference vs Use an API

This is the question we spent the most time on.

Use Token Factory or similar API when:

You're in early stages and want zero infrastructure overhead
Request volume is unpredictable
You need OpenAI API compatibility without running servers

Self-host with vLLM on packet.ai when:

Request volume is predictable and high enough that per-token pricing adds up
You need custom model weights (fine-tuned models, private checkpoints)
Latency requirements need dedicated GPU access

The crossover point varies by model and request volume. For most production applications above 1M tokens per day, self-hosting on packet.ai's L40S or RTX 6000 Pro with vLLM becomes more cost-efficient than API pricing.

FAQ

Q: Can I use Hugging Face models directly on any of these providers?

Yes. Hugging Face Hub access requires only pip install transformers and a HF token. All providers that give SSH access work. For gated models like Llama 3 and Gemma, you need your token to have accepted the model license on the Hugging Face website first.

Q: What is the fastest way to get a GPU running for a quick experiment?

packet.ai and RunPod both provision instances quickly. packet.ai's dedicated instances are SSH-ready immediately after creation. For even faster experimentation without provisioning anything, packet.ai's Token Factory gives you inference via API with no GPU management required.

Q: Is there a free tier for GPU cloud?

Google Colab has free T4 GPU access with session limits. For real workloads, there's no meaningful free tier. Most providers have low minimum billing thresholds. packet.ai bills hourly, so you can run a quick experiment and pay only for the time actually used.

Q: Does NVIDIA B200 matter for inference workloads?

Yes. B200 delivers significantly higher memory bandwidth than H100, which directly impacts inference throughput for large models. For serving 70B+ parameter models at production scale, B200 reduces the GPU count needed for a given throughput target. packet.ai offers B200 on hourly billing without a reservation.

Q: Can I run multi-GPU inference with vLLM on packet.ai?

Yes. vLLM's tensor_parallel_size parameter distributes the model across GPUs. The example in this article shows a 2-GPU setup. The same approach scales to the GPU count of your instance on packet.ai.

Q: How do I reduce inference costs without switching providers?

Quantization is the biggest lever. Running a 70B model in 4-bit (GPTQ or AWQ) cuts VRAM requirements roughly in half, which means you can use a smaller GPU instance. vLLM supports both formats natively. On packet.ai, moving from an A100 to an RTX 6000 Pro for a quantized 70B model can significantly reduce hourly cost with minimal quality impact.

Q: What's the difference between packet.ai dedicated and dynamic instances?

Dedicated instances give you exclusive physical access to the GPU. Your VRAM, your compute, no other tenants. Dynamic instances share underlying hardware at lower cost. For production inference or training where consistent throughput matters, dedicated is the right choice. For experimentation or burst workloads, dynamic saves money. Both are available on hourly billing at packet.ai.

Final Take

If you're starting a new AI project in 2026 and don't have existing cloud commitments, the stack I'd reach for is:

packet.ai for GPU compute (training and fine-tuning)
packet.ai Token Factory for managed inference
vLLM for self-hosted inference when volume justifies it
AWS or Azure only if you need specific managed services that don't exist elsewhere

The hyperscalers are still the right answer for some teams. But for developers who just need reliable GPUs with predictable pricing and fast provisioning, the specialist providers have pulled far enough ahead that defaulting to AWS purely out of habit no longer makes sense.

packet.ai's pricing page is public. Run the numbers on your workload before committing to anything.

LLM Inference APIs in 2026: How to Choose Between OpenAI, Open Models, and Self-Hosted GPUs

Kavya — Thu, 13 Aug 2026 15:06:27 +0000

If you are building an AI application, choosing the model is only part of the problem.

Sooner or later, you also have to decide how you are going to run that model.

You can call a proprietary model through an API such as OpenAI. You can deploy an open model yourself on a GPU using tools such as vLLM. Or you can use a managed inference API that hosts open models for you.

All three work. The right choice depends on your traffic, model requirements, engineering resources, and how much you are willing to spend on inference.

For developers working with open models, managed inference is becoming an interesting middle ground. You get an API instead of a GPU to manage, while keeping access to models such as Llama, Qwen, DeepSeek, and Mistral.

That is the problem Token Factory from Packet.ai is built around.

What is an LLM inference API?

An LLM inference API gives your application access to a hosted language model through an API.

Your application sends a request. The provider runs the model on its infrastructure and returns the response.

You do not have to provision a GPU, download model weights, configure an inference server, or keep track of GPU capacity.

The basic setup looks like this:

Your application → LLM API → model → response

This sounds simple, but the infrastructure behind that API can be quite complicated.

Running an open model yourself can involve GPU selection, VRAM requirements, model loading, inference engines, autoscaling, monitoring, concurrency, and capacity planning.

A managed inference provider takes care of that layer.

What are the main ways to run an LLM?

There are three common approaches.

Option	When it makes sense	What you get	What you give up
Proprietary API	You want a simple way to use leading closed models	Easy setup and managed infrastructure	Higher costs at large token volumes
Managed open-model API	You want open models without running GPUs	Lower-cost inference with managed infrastructure	Your choices depend on the provider's model lineup
Self-hosted	You have high, predictable usage and an infrastructure team	Full control over models and serving	GPU costs and the work of running the stack

There is no universal winner here.

A startup processing a few million tokens a month has a very different infrastructure problem from a company processing billions.

That is why comparing providers only on model quality or price per million tokens can be misleading.

Why use an open-model inference API?

Open models have changed the economics of running AI applications.

Models such as Llama, Qwen, DeepSeek, and Mistral can be used for many production workloads without requiring developers to build their own model-serving infrastructure.

That makes it possible to separate two decisions:

Which model should run the workload?

and

Who should operate the infrastructure?

You can choose an open model while letting someone else handle the GPUs.

This is the main appeal of a managed open-model inference API.

Why does OpenAI compatibility matter?

This is one of the most practical things to look for when choosing an inference provider.

If your application already uses the OpenAI SDK, moving to another API can be annoying if you have to change your entire inference layer.

An OpenAI-compatible API reduces that work.

Token Factory uses an OpenAI-compatible API. Packet.ai says developers can change the base_url to api.packet.ai/v1 and replace the API key while keeping their application code unchanged.

For example:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.packet.ai/v1",
    api_key="YOUR_PACKET_API_KEY",
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {
            "role": "user",
            "content": "Explain retrieval-augmented generation."
        }
    ],
)

That matters when you want to test another provider without rebuilding the application around a new SDK.

It also makes the inference provider easier to change later.

What does Token Factory provide?

Token Factory is Packet.ai's managed LLM inference API for open models.

The current launch catalog has six models:

Llama 3.1 8B has a 128K context window and costs $0.06 per 1M tokens. It is best suited for classification, routing, and summarization tasks.

Mistral Small 3 supports a 32K context window and is priced at $0.18 per 1M tokens. It works well for tool calling and structured output generation.

Llama 3.3 70B also supports a 128K context window and is priced at $0.59 per 1M tokens. It is a strong general-purpose model for chat, RAG systems, and agent workflows.

Qwen2.5 72B offers a 128K context window at $0.62 per 1M tokens. It is particularly useful for multilingual applications and long-context reasoning tasks.

DeepSeek-V3 supports a 64K context window and costs $0.85 per 1M tokens. It is optimized for coding, debugging, and complex reasoning problems.

BGE-M3 has an 8K context window and is priced at $0.02 per 1M tokens. It is designed for embeddings and retrieval tasks in RAG pipelines.

These are indicative launch prices. Input and output tokens are metered separately at the same listed rate for each model.

The catalog is deliberately small rather than trying to list every open model available.

The idea is to cover common production workloads with a handful of models.

Which LLM should you use?

The cheapest model is not automatically the right model.

The better question is what the model actually needs to do.

Llama 3.1 8B for high-volume tasks

If you are doing classification, routing, summarization, or structured extraction, you may not need a large model.

Llama 3.1 8B is listed at $0.06 per million tokens on Token Factory and has a 128K context window.

For simple workloads running at high volume, a smaller model can make a noticeable difference to your inference bill.

Llama 3.3 70B for general applications

For chatbots, assistants, RAG applications, and more complex instruction-following tasks, Llama 3.3 70B is the general-purpose option in the catalog.

It is currently listed at $0.59 per million tokens with a 128K context window.

DeepSeek-V3 for coding and reasoning

DeepSeek-V3 is the higher-priced model in the current Token Factory catalog at $0.85 per million tokens.

It is aimed at coding, debugging, and more complex reasoning workloads.

BGE-M3 for RAG retrieval

RAG systems have two separate jobs.

First, retrieve the relevant information.

Then, generate the answer.

BGE-M3 can handle the embedding side of that workflow. Token Factory currently lists it at $0.02 per million tokens.

This is also a good example of why using one model for everything does not always make sense.

How much does LLM inference cost?

The answer depends heavily on the model and the number of tokens you process.

For example, using Token Factory's current listed rates, 10 million output tokens would cost approximately:

Model: Llama 3.1 8B
Price per 1M tokens: $0.06
Cost for 10M output tokens: $0.60

Model: Mistral Small 3
Price per 1M tokens: $0.18
Cost for 10M output tokens: $1.80

Model: Llama 3.3 70B
Price per 1M tokens: $0.59
Cost for 10M output tokens: $5.90

Model: DeepSeek-V3
Price per 1M tokens: $0.85
Cost for 10M output tokens: $8.50

These are output-token calculations only. Input tokens are charged separately at the same model rate.

Actual application costs will depend on how many input and output tokens your application generates.

That distinction matters.

A chatbot with long prompts and short answers has a different cost profile from an agent that sends large context windows and generates long responses.

How does managed inference compare with other providers?

Price comparisons are most useful when you compare the same model.

For Llama 3.3 70B, Packet.ai currently lists the following comparison:

Packet.ai Token Factory offers Llama 3.3 70B at $0.59 per 1M input tokens and $0.59 per 1M output tokens, with OpenAI compatibility enabled.

Together AI (Llama 3.3 70B Turbo) is priced at $1.04 per 1M input tokens and $1.04 per 1M output tokens, and it is OpenAI-compatible.

Fireworks AI (Llama 3.3 70B) is priced at $0.90 per 1M input tokens and $0.90 per 1M output tokens, and it is OpenAI-compatible.

Groq (Llama 3.3 70B) is priced at $0.59 per 1M input tokens and $0.79 per 1M output tokens, and it is OpenAI-compatible.

AWS Bedrock (Llama 3.3 70B) is priced at $0.72 per 1M input tokens and $0.72 per 1M output tokens, but it is not OpenAI-compatible.

Packet.ai says these rates were verified against the providers' official pricing pages in August 2026.

The point of a comparison like this is not simply to find the lowest number.

You also need to look at API compatibility, model availability, latency, throughput, scaling, features, and whether the provider fits your workload.

Managed inference vs self-hosting

Self-hosting an open model gives you control.

It also gives you more work.

A typical self-hosted setup may involve:

Choosing the right GPU
Provisioning the machine
Installing the inference stack
Downloading model weights
Managing VRAM
Configuring vLLM or another inference engine
Handling traffic spikes
Monitoring GPU utilization
Scaling capacity
Updating the model
Handling failures

If you already have an ML infrastructure team and your GPUs stay busy, this may be worth it.

If you are building an MVP or running an application with unpredictable traffic, it may not be.

This is where managed inference can make sense.

You pay for inference instead of spending engineering time running the infrastructure.

When is self-hosting cheaper?

There is no fixed answer.

It comes down to utilization.

If your GPU is busy around the clock, paying for a dedicated GPU can make sense.

If your application receives traffic for a few hours a day and sits idle the rest of the time, you are paying for capacity you are not using.

Token Factory takes the opposite approach. Packet.ai says the service scales to zero, so idle traffic costs nothing, with billing based on tokens rather than GPU hours.

So the useful comparison is not:

GPU price vs token price

It is:

What does my workload cost at its actual utilization?

That is the number worth calculating.

Model routing can reduce inference costs

There is another way to control LLM costs: stop sending every request to the same model.

Consider a customer-support application.

A simple request such as:

"Where can I download my invoice?"

probably does not need the same model as:

"Read these three contracts and explain the differences in their termination clauses."

You could route those requests differently.

For example:

Intent classification → Llama 3.1 8B

Document retrieval → BGE-M3

General response → Llama 3.3 70B

Complex reasoning → DeepSeek-V3

This kind of routing lets you spend more on the requests that actually need it.

It also makes your inference architecture easier to optimize as usage grows.

What should you look for in an LLM inference API?

Price is important, but it should not be the only thing you compare.

1. API compatibility

Can you use your existing OpenAI SDK?

2. Model selection

Does the provider offer the models your application actually needs?

3. Input and output pricing

Are both sides of the request clearly priced?

4. Scaling

What happens when traffic increases?

What happens when traffic drops?

5. Production features

Look for streaming, structured output, tool calling, rate limits, and other features your application depends on.

6. Data handling

If you are building an enterprise application, check data retention, training policies, data residency, and compliance support.

Token Factory currently lists US and EU data residency, DPA support, and says Packet.ai does not train on customer data.

7. Fine-tuning support

If you have your own model or LoRA adapter, check whether the provider can host it.

Token Factory says teams can bring their own LoRA adapter or full checkpoint and serve it through the same OpenAI-compatible endpoint.

Who should use a managed open-model API?

A managed open-model API is worth considering if you:

Are building an AI SaaS product
Are working on an MVP
Have bursty or unpredictable traffic
Want to use open models
Do not want to manage GPUs
Want usage-based billing
Already use the OpenAI SDK
Want to test different models
Have a small engineering team
Want to keep infrastructure work out of the product roadmap

Token Factory specifically positions itself for developers, indie builders, and smaller teams that want LLM inference without managing GPU infrastructure.

Who should probably self-host?

Self-hosting may be a better fit if you:

Run millions of tokens every day
Have consistently high GPU utilization
Need a model that a managed provider does not offer
Need control over vLLM configuration
Need specific GPU hardware
Need multi-GPU serving
Already have ML infrastructure expertise

In other words, managed inference is not a replacement for self-hosting in every situation.

The workload decides.

How do you switch from OpenAI to an open-model API?

If your application already uses the OpenAI SDK, the process can be straightforward when the new provider supports the same API format.

With Token Factory, for example, the main change is the API endpoint and API key:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.packet.ai/v1",
    api_key="YOUR_PACKET_API_KEY",
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {
            "role": "user",
            "content": "Summarize this document."
        }
    ],
)

Packet.ai says the existing application code can remain unchanged when switching to its OpenAI-compatible endpoint.

For a production migration, you should still test model quality, latency, token usage, error handling, rate limits, and output consistency before switching traffic.

What is the cheapest LLM inference API?

There is no single provider that is cheapest for every model and workload.

The useful comparison is model by model.

For example, Packet.ai currently lists Llama 3.3 70B at $0.59 per million tokens, compared with $1.04 from Together AI and $0.90 from Fireworks AI.

But price should always be checked against the provider's current pricing page because inference pricing changes frequently.

What is the best API for open-source LLMs?

There is no single best API for every application.

The right provider depends on the models you need, pricing, latency, throughput, API compatibility, infrastructure, and production features.

If you want managed access to open models without operating GPUs yourself, providers such as Token Factory, Together AI, Fireworks AI, Groq, and others are worth comparing.

Is managed inference better than self-hosting?

Not automatically.

Managed inference removes infrastructure work and can make sense for variable workloads.

Self-hosting gives you more control and can make economic sense when your GPUs remain highly utilized.

The best choice depends on your workload and engineering resources.

What is an OpenAI-compatible API?

An OpenAI-compatible API follows the same general API structure and request format used by OpenAI.

This can make it easier to move an existing application between inference providers because the application does not have to be rebuilt around a completely different API.

What is Token Factory?

Token Factory is Packet.ai's managed inference API for open models.

It currently offers six models, including Llama 3.1 8B, Llama 3.3 70B, Qwen2.5 72B, DeepSeek-V3, Mistral Small 3, and BGE-M3. It supports an OpenAI-compatible API, per-token billing, scale-to-zero, streaming, tool calling, JSON mode, and fine-tune hosting.

The bigger decision is not just the model

LLM infrastructure is becoming easier to access, but that does not mean the architecture decision has disappeared.

You still need to decide:

Which model is good enough for the task?

How many tokens will the application process?

How predictable is the traffic?

Do you need to manage GPUs yourself?

How important is API compatibility?

What will the workload actually cost at scale?

For some teams, a proprietary API will still be the easiest option.

For others, self-hosting will make sense.

And for teams that want open models without running the infrastructure themselves, managed inference sits somewhere in between.

That is the space Token Factory is targeting.

Try Token Factory

Token Factory gives developers access to open models through a managed, OpenAI-compatible API.

You can compare the models, pricing, supported features, and infrastructure on the official product page:

Explore Token Factory

You can also read the full technical breakdown, including the model catalog, pricing comparison, setup instructions, and self-hosted considerations:

Read the Token Factory LLM Inference API guide

If you are already using the OpenAI SDK, the simplest test is to run an existing workload against the Token Factory endpoint and compare the things that actually matter: cost, latency, throughput, and output quality.

Renting GPUs for AI? Start with VRAM, Not the GPU

Kavya — Wed, 05 Aug 2026 10:38:11 +0000

If you're deploying an LLM for the first time, you've probably searched for something like:

What's the best GPU for AI?

It's a common question, but it's usually the wrong place to start.

Before comparing A100s, H100s, or Blackwell GPUs, answer one question first:

How much VRAM does your workload actually need?

Choosing a GPU without first sizing VRAM often results in either out-of-memory errors or paying for memory you don't need. That's why infrastructure planning should start with the workload, not the hardware.

Why VRAM Comes First

For LLM inference, GPU memory is usually the first constraint.

Your VRAM budget isn't just the model itself. It also includes:

Model weights
KV cache
Framework overhead
Production headroom

Even two deployments running the same model can require very different amounts of memory depending on context length, concurrency, and inference settings.

That's why choosing a GPU before understanding memory requirements often leads to poor infrastructure decisions.

*Model Size Doesn't Tell the Whole Story
*
Many developers assume parameter count directly determines GPU requirements.

It doesn't.

The same model can require dramatically different amounts of VRAM depending on precision and quantization.

A model running in FP16 may require multiple high-memory GPUs, while the same model using 4-bit quantization can often fit on a single accelerator.

Understanding this difference is one of the easiest ways to avoid over provisioning infrastructure.

Don't Forget the KV Cache

A deployment that works perfectly during testing can run into memory limits once it reaches production.

One of the biggest reasons is the KV cache.

Every active request stores key-value tensors in GPU memory. As context length and concurrency increase, so does memory consumption.

Ignoring KV cache is one of the most common mistakes when sizing GPUs for production inference.

Mixture-of-Experts Models Still Need Careful Planning

Another misconception is that Mixture-of-Experts (MoE) models only require memory for the experts used during inference.

While only a subset of experts is activated for each token, the model's weights still need to be loaded into GPU memory when the model starts.

That means VRAM planning remains just as important for MoE deployments.

A Better Way to Choose a GPU

Instead of asking:

Which GPU should I rent?

Start with these questions:

Which model am I deploying?
What precision or quantization will I use?
How many concurrent requests do I expect?
What context length will I support?
How much VRAM will the complete workload require?

Once those answers are clear, selecting the right GPU becomes much more straightforward.

Final Thoughts

Choosing AI infrastructure isn't about renting the newest GPU.

It's about matching your workload to the right amount of memory, compute, and infrastructure.

Starting with VRAM helps avoid deployment failures, improves GPU utilization, and prevents unnecessary infrastructure costs.

Packet.ai's guide walks through VRAM sizing for today's leading open-source models, explains common sizing mistakes, and maps workloads to GPU configurations to help teams make better infrastructure decisions.

Read the full guide:
https://packet.ai/blog/rent-gpu-for-ai-vram-requirements-guide

Why memory bandwidth matters more than TFLOPS for LLM inference

Kavya — Mon, 03 Aug 2026 14:46:45 +0000

Your GPU's TFLOPS rating does not decide how fast it generates tokens. Memory bandwidth does. Here's the spec that actually separates A100-era inference from H200-era inference.

Key takeaways

HBM3e reaches up to 9.6 Gb/s per pin versus HBM2e's 3.2 Gb/s, tripling per-pin signalling speed with the same 1,024-bit interface width.
Total GPU memory bandwidth jumps from 2.0 TB/s on the A100 (HBM2e) to 4.8 TB/s on the H200 (HBM3e), a 140% increase, and 8.0 TB/s on the B200.
LLM token generation (decode phase) is memory-bound: tokens per second tracks memory bandwidth far more closely than TFLOPS.
H200 achieves 31,712 tokens/sec on Llama 2 70B with TensorRT-LLM, 45% faster than H100's 21,806 tokens/sec, driven entirely by the HBM3e memory upgrade.
HBM3e is 2.5x more power-efficient per watt than HBM2e, delivering higher bandwidth at lower voltage (1.1V versus 1.2V).

What is HBM and how does it work?

High Bandwidth Memory is a form of stacked DRAM co-packaged with the GPU die on a shared silicon interposer, connected through Through-Silicon Vias (TSVs). Unlike GDDR6 memory sitting on separate chips across a PCB trace, HBM stacks sit microns from the compute die. The interface is wider, latency is lower, and bandwidth per watt is substantially better.

For reference: GDDR6X on a consumer GPU achieves roughly 1.0 TB/s. HBM2e on the A100 doubles that at 2.0 TB/s. HBM3e on the H200 SXM more than doubles HBM2e again at 4.8 TB/s. Each generation achieves this by increasing per-pin signalling speed, not by widening the interface bus.

HBM3e vs HBM2e: the numbers

Specification	HBM2e	HBM3	HBM3e
Flagship GPU	A100 SXM	H100 SXM	H200 / B200 SXM
Total bandwidth	2.0 TB/s	3.35 TB/s	4.8 TB/s (H200) / 8.0 TB/s (B200)
Pin speed	~3.2 Gb/s	6.4 Gb/s	up to 9.6 Gb/s
Memory capacity	80 GB	80 GB	141 to 192 GB
Operating voltage	1.2V	1.1V	1.1V
Power efficiency vs HBM2e	1x	~2x	2.5x

Why LLM decode is memory-bound, not compute-bound

LLM inference runs in two phases. The prefill phase processes the full input prompt in one forward pass and is compute-bound. The decode phase generates one token per forward pass and is memory-bound: every token requires loading the complete model weight matrix and accumulated KV cache from HBM before a small amount of arithmetic work.

At batch size 1, arithmetic intensity falls as low as 1 to 2 FLOP per byte — far below the compute-to-bandwidth ratio of even the A100. Memory bandwidth is the first bottleneck for real-world serving workloads at batch sizes below roughly 64 to 128, which covers most production inference deployments.

This is why upgrading from an A100 (HBM2e, 2.0 TB/s) to an H200 (HBM3e, 4.8 TB/s) improves inference throughput by 2 to 3x on 70B parameter models.

Benchmark: H200 vs H100 on Llama 2 70B

GPU	HBM Type	Bandwidth	Llama 2 70B (tok/s)	vs H100
H200 SXM	HBM3e	4.8 TB/s	31,712	+45%
H100 SXM	HBM3	3.35 TB/s	21,806	baseline