DEV Community: Kavya

Best GPU Cloud Providers for AI in 2026: Developer Comparison With Code

Kavya — Tue, 25 Aug 2026 11:38:15 +0000

I Tested Every Major GPU Cloud So You Don't Have To

Last quarter my team spent three weeks benchmarking GPU clouds for our LLM inference pipeline. We had a simple requirement: run Llama 3 70B in production, keep latency under 2 seconds, and don't spend a fortune doing it.

What followed was a crash course in how differently each provider approaches the same problem.

Here's everything we learned, with actual code.

Who This Is For

If you're an ML engineer, backend developer, or startup CTO trying to figure out where to run your AI workloads in 2026, this is the guide I wish existed when we started. We cover:

Real provisioning workflows, not just marketing claims
Code snippets that actually run
Cost estimates based on real usage
Honest takes on where each provider wins and loses

TL;DR: Quick Provider Summary

Provider	Dev Experience	GPU Availability	B200?	Best Use Case
AWS	Moderate friction	Good with reservation	Preview	Enterprise MLOps
Azure	Moderate friction	Good with reservation	Preview	Azure OpenAI integration
GCP	Moderate friction	Good (TPU excellent)	Announced	JAX/TF training
CoreWeave	Good for large clusters	Strong (reserved)	Yes	Lab-scale training
Lambda Labs	Good	Moderate (books out)	No	Research workloads
RunPod	Good	Variable	No	Budget experiments
packet.ai	Very good	Strong	Yes	AI-native GPU compute

What "AI Workload" Actually Means in Practice

Before we get into providers, let's be specific. "AI workload" means different things to different teams:

Pre-training / full training: Hundreds to thousands of GPU-hours, multi-node clusters, InfiniBand networking, checkpoint storage. If you're doing this, you already know your requirements.

Fine-tuning: Single-node or small multi-GPU, 1-100 GPU-hours, high VRAM GPUs (A100 80GB, H100). Iteration speed matters more than raw cluster size.

Inference serving: Low latency, high throughput, VRAM-efficient, OpenAI API compatibility, autoscaling.

Evaluation and benchmarking: Short burst compute, flexibility over raw performance.

Most developers are in the fine-tuning or inference bucket. That's where the real provider differences show up.

The Real Developer Workflow: Deploying vLLM for LLM Inference

Here's a concrete example. Deploying Llama 3 70B with vLLM for inference is the same across every provider once you have SSH. The difference is how fast you get there and what it costs.

Step 1: SSH into your instance

ssh ubuntu@your-instance-ip

Step 2: Install vLLM

pip install vllm

Step 3: Run inference in Python

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    tensor_parallel_size=2,
    dtype="bfloat16",
    max_model_len=8192,
)

sampling_params = SamplingParams(
    temperature=0.7,
    max_tokens=512,
)

outputs = llm.generate(
    ["What is the difference between A100 and B200?"],
    sampling_params
)
print(outputs[0].outputs[0].text)

Step 4: Serve as OpenAI-compatible API

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-70B-Instruct \
    --tensor-parallel-size 2 \
    --host 0.0.0.0 \
    --port 8000

Step 5: Hit the endpoint with OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://your-instance-ip:8000/v1",
    api_key="not-needed-for-local",
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    messages=[
        {"role": "user", "content": "Explain GPU memory bandwidth in simple terms"}
    ],
)
print(response.choices[0].message.content)

This workflow runs identically on every provider that gives SSH access to an NVIDIA GPU. The differences come down to provisioning speed, GPU options, and hourly cost.

Provider Breakdown: The Developer Perspective

AWS

Amazon Web Services gives you GPU instances through EC2. The key instance types for AI:

p4d.24xlarge -- 8x A100 40GB, NVLink, roughly $32/hr on-demand
p4de.24xlarge -- 8x A100 80GB, NVLink, roughly $40/hr on-demand
p5.48xlarge -- 8x H100 80GB, NVLink + EFA, roughly $98/hr on-demand

aws ec2 run-instances \
    --image-id ami-0abcdef1234567890 \
    --instance-type p4d.24xlarge \
    --key-name your-key-pair \
    --security-group-ids sg-your-group \
    --subnet-id subnet-your-subnet

SageMaker wraps this with managed training, experiment tracking, model registry, and deployment pipelines. Genuinely useful if you need that. The friction: on-demand availability for large GPU instances is unreliable without a reservation. NVIDIA B200 is currently preview-only.

Dev experience verdict: If you're already AWS-native, it works. For raw GPU access without managed services, the friction is higher than it needs to be.

Google Cloud Platform

Google Cloud GPU instances run on A2 (A100) and A3 (H100) machine families. TPU access is unique to GCP.

gcloud compute instances create gpu-instance \
    --zone=us-central1-a \
    --machine-type=a2-highgpu-1g \
    --accelerator=count=1,type=nvidia-tesla-a100 \
    --image-family=tf-latest-gpu \
    --image-project=deeplearning-platform-release \
    --boot-disk-size=100GB \
    --metadata="install-nvidia-driver=True"

TPU access requires a completely separate workflow and JAX/TF framework commitment. If your stack is CUDA-based, TPUs require a meaningful porting effort.

Dev experience verdict: Strong for Vertex AI and JAX-native workloads. CUDA-based GPU access has the same friction as other hyperscalers.

CoreWeave

CoreWeave is Kubernetes-native. Their workflow uses GPU workloads as Kubernetes pods, which is excellent if your ML team already uses Kubernetes.

apiVersion: v1
kind: Pod
metadata:
  name: vllm-inference
spec:
  containers:
  - name: vllm
    image: vllm/vllm-openai:latest
    resources:
      limits:
        nvidia.com/gpu: "2"
        memory: "160Gi"
    command: ["python", "-m", "vllm.entrypoints.openai.api_server"]
    args:
    - "--model"
    - "meta-llama/Meta-Llama-3-70B-Instruct"
    - "--tensor-parallel-size"
    - "2"
  restartPolicy: Never

Their H100 and H200 capacity is legitimate. InfiniBand networking for multi-node training is available. The catch: casual on-demand access is harder than with providers that offer instant provisioning. NVIDIA B200 is available through reservation.

Dev experience verdict: Excellent for Kubernetes-native teams running large clusters. Less accessible for quick iteration or solo developers.

Lambda Labs

Lambda Labs has a clean API and competitive pricing. Their instance availability can be tight for popular GPU types.

import requests

response = requests.post(
    "https://cloud.lambdalabs.com/api/v1/instance-operations/launch",
    auth=("your-api-key", ""),
    json={
        "region_name": "us-east-1",
        "instance_type_name": "gpu_1x_a100_sxm4",
        "ssh_key_names": ["your-ssh-key"],
        "quantity": 1,
    }
)
print(response.json())

Strong for research use cases. No inference-as-a-service layer. NVIDIA B200 not yet available.

Dev experience verdict: Good for straightforward GPU access. Availability is the main friction point for popular instance types.

RunPod

RunPod is fast to spin up and cheap for experimentation.

pip install runpod

runpod config

runpod pod create \
    --gpu-type "NVIDIA A100 80GB" \
    --image runpod/pytorch:latest \
    --ports "8000/http" \
    --volume-size 50

For serverless inference RunPod is particularly developer-friendly:

import runpod

runpod.api_key = "your-api-key"

endpoint = runpod.Endpoint("your-endpoint-id")

run_request = endpoint.run(
    {"input": {"prompt": "Explain transformers in two sentences"}}
)
print(run_request.output())

Dev experience verdict: Best for quick experiments and budget inference. Third-party hardware tier has variable reliability. Not the right choice for production workloads where uptime matters.

packet.ai

packet.ai is the provider that surprised me the most during our benchmarking. The pitch is simple: dedicated NVIDIA GPUs, hourly billing, no reservation commitment, SSH-ready in minutes.

After instance provisioning, access is immediate:

ssh ubuntu@your-packet-instance-ip

Environment setup is clean:

sudo apt update && sudo apt install -y nvidia-cuda-toolkit

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

pip install vllm

nvidia-smi

No driver drama. No 15-minute wait. The GPU is just there.

For inference using Token Factory, packet.ai's OpenAI-compatible inference API:

from openai import OpenAI

# Switching from OpenAI to packet.ai Token Factory
# is literally just changing base_url and api_key
client = OpenAI(
    base_url="https://api.packet.ai/v1",
    api_key="your-packet-api-key",
)

response = client.chat.completions.create(
    model="llama-3-70b-instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What makes NVIDIA B200 different from H100?"}
    ],
    max_tokens=512,
    temperature=0.7,
)

print(response.choices[0].message.content)

That's the whole migration from OpenAI to packet.ai for inference. Base URL change and a key swap.

For fine-tuning on packet.ai's A100 or RTX 6000 Pro:

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model
import torch

model_name = "meta-llama/Meta-Llama-3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

peft_config = LoraConfig(
    task_type="CAUSAL_LM",
    r=16,
    lora_alpha=32,
    lora_dropout=0.1,
    target_modules=["q_proj", "v_proj"],
)
model = get_peft_model(model, peft_config)

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    warmup_steps=100,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_steps=500,
)

On packet.ai's RTX 6000 Pro (48GB VRAM), this handles models up to 13B parameters comfortably. On A100 (80GB), 70B models with quantization are doable.

The NVIDIA B200 availability on packet.ai is worth calling out separately. Most providers have B200 on a waitlist or behind a reservation requirement. packet.ai has it on hourly billing with no commitment. If you want to benchmark Blackwell GPU performance for your specific model today, this is your most practical path.

Dev experience verdict: The fastest path from "I need a GPU" to "I have a running GPU." Token Factory removes the need to manage inference infrastructure entirely if you don't want to. Pricing is published and predictable.

Choosing the Right GPU for Your Task

VRAM Requirements by Task

Task	Model Size	Min VRAM	Recommended Instance
Inference only	7B	16GB	RTX 4090
Inference only	70B (4-bit)	48GB	RTX 6000 Pro or L40S
Fine-tuning (LoRA)	7B	24GB	RTX 6000 Pro
Fine-tuning (LoRA)	70B	80GB	A100 80GB
Full fine-tuning	7B	80GB+	A100 80GB
Training	70B+	Multi-GPU	A100 or B200 cluster

Real Cost Estimate: Fine-tuning Llama 3 8B

Training with LoRA for 3 epochs on a 100K sample dataset takes roughly 8-12 GPU-hours on an A100.

AWS SageMaker ml.p4d.24xlarge: roughly $4-5 for compute, plus SageMaker overhead and egress
Lambda Labs A100: roughly $8-12 total
packet.ai RTX 6000 Pro: lower than A100 alternatives, published on pricing page

The RTX 6000 Pro is worth considering for fine-tuning runs that fit in 48GB VRAM. It's less expensive per hour than A100 while covering most LoRA fine-tuning tasks.

When to Self-Host Inference vs Use an API

This is the question we spent the most time on.

Use Token Factory or similar API when:

You're in early stages and want zero infrastructure overhead
Request volume is unpredictable
You need OpenAI API compatibility without running servers

Self-host with vLLM on packet.ai when:

Request volume is predictable and high enough that per-token pricing adds up
You need custom model weights (fine-tuned models, private checkpoints)
Latency requirements need dedicated GPU access

The crossover point varies by model and request volume. For most production applications above 1M tokens per day, self-hosting on packet.ai's L40S or RTX 6000 Pro with vLLM becomes more cost-efficient than API pricing.

FAQ

Q: Can I use Hugging Face models directly on any of these providers?

Yes. Hugging Face Hub access requires only pip install transformers and a HF token. All providers that give SSH access work. For gated models like Llama 3 and Gemma, you need your token to have accepted the model license on the Hugging Face website first.

Q: What is the fastest way to get a GPU running for a quick experiment?

packet.ai and RunPod both provision instances quickly. packet.ai's dedicated instances are SSH-ready immediately after creation. For even faster experimentation without provisioning anything, packet.ai's Token Factory gives you inference via API with no GPU management required.

Q: Is there a free tier for GPU cloud?

Google Colab has free T4 GPU access with session limits. For real workloads, there's no meaningful free tier. Most providers have low minimum billing thresholds. packet.ai bills hourly, so you can run a quick experiment and pay only for the time actually used.

Q: Does NVIDIA B200 matter for inference workloads?

Yes. B200 delivers significantly higher memory bandwidth than H100, which directly impacts inference throughput for large models. For serving 70B+ parameter models at production scale, B200 reduces the GPU count needed for a given throughput target. packet.ai offers B200 on hourly billing without a reservation.

Q: Can I run multi-GPU inference with vLLM on packet.ai?

Yes. vLLM's tensor_parallel_size parameter distributes the model across GPUs. The example in this article shows a 2-GPU setup. The same approach scales to the GPU count of your instance on packet.ai.

Q: How do I reduce inference costs without switching providers?

Quantization is the biggest lever. Running a 70B model in 4-bit (GPTQ or AWQ) cuts VRAM requirements roughly in half, which means you can use a smaller GPU instance. vLLM supports both formats natively. On packet.ai, moving from an A100 to an RTX 6000 Pro for a quantized 70B model can significantly reduce hourly cost with minimal quality impact.

Q: What's the difference between packet.ai dedicated and dynamic instances?

Dedicated instances give you exclusive physical access to the GPU. Your VRAM, your compute, no other tenants. Dynamic instances share underlying hardware at lower cost. For production inference or training where consistent throughput matters, dedicated is the right choice. For experimentation or burst workloads, dynamic saves money. Both are available on hourly billing at packet.ai.

Final Take

If you're starting a new AI project in 2026 and don't have existing cloud commitments, the stack I'd reach for is:

packet.ai for GPU compute (training and fine-tuning)
packet.ai Token Factory for managed inference
vLLM for self-hosted inference when volume justifies it
AWS or Azure only if you need specific managed services that don't exist elsewhere

The hyperscalers are still the right answer for some teams. But for developers who just need reliable GPUs with predictable pricing and fast provisioning, the specialist providers have pulled far enough ahead that defaulting to AWS purely out of habit no longer makes sense.

packet.ai's pricing page is public. Run the numbers on your workload before committing to anything.

LLM Inference APIs in 2026: How to Choose Between OpenAI, Open Models, and Self-Hosted GPUs

Kavya — Thu, 13 Aug 2026 15:06:27 +0000

If you are building an AI application, choosing the model is only part of the problem.

Sooner or later, you also have to decide how you are going to run that model.

You can call a proprietary model through an API such as OpenAI. You can deploy an open model yourself on a GPU using tools such as vLLM. Or you can use a managed inference API that hosts open models for you.

All three work. The right choice depends on your traffic, model requirements, engineering resources, and how much you are willing to spend on inference.

For developers working with open models, managed inference is becoming an interesting middle ground. You get an API instead of a GPU to manage, while keeping access to models such as Llama, Qwen, DeepSeek, and Mistral.

That is the problem Token Factory from Packet.ai is built around.

What is an LLM inference API?

An LLM inference API gives your application access to a hosted language model through an API.

Your application sends a request. The provider runs the model on its infrastructure and returns the response.

You do not have to provision a GPU, download model weights, configure an inference server, or keep track of GPU capacity.

The basic setup looks like this:

Your application → LLM API → model → response

This sounds simple, but the infrastructure behind that API can be quite complicated.

Running an open model yourself can involve GPU selection, VRAM requirements, model loading, inference engines, autoscaling, monitoring, concurrency, and capacity planning.

A managed inference provider takes care of that layer.

What are the main ways to run an LLM?

There are three common approaches.

Option	When it makes sense	What you get	What you give up
Proprietary API	You want a simple way to use leading closed models	Easy setup and managed infrastructure	Higher costs at large token volumes
Managed open-model API	You want open models without running GPUs	Lower-cost inference with managed infrastructure	Your choices depend on the provider's model lineup
Self-hosted	You have high, predictable usage and an infrastructure team	Full control over models and serving	GPU costs and the work of running the stack

There is no universal winner here.

A startup processing a few million tokens a month has a very different infrastructure problem from a company processing billions.

That is why comparing providers only on model quality or price per million tokens can be misleading.

Why use an open-model inference API?

Open models have changed the economics of running AI applications.

Models such as Llama, Qwen, DeepSeek, and Mistral can be used for many production workloads without requiring developers to build their own model-serving infrastructure.

That makes it possible to separate two decisions:

Which model should run the workload?

and

Who should operate the infrastructure?

You can choose an open model while letting someone else handle the GPUs.

This is the main appeal of a managed open-model inference API.

Why does OpenAI compatibility matter?

This is one of the most practical things to look for when choosing an inference provider.

If your application already uses the OpenAI SDK, moving to another API can be annoying if you have to change your entire inference layer.

An OpenAI-compatible API reduces that work.

Token Factory uses an OpenAI-compatible API. Packet.ai says developers can change the base_url to api.packet.ai/v1 and replace the API key while keeping their application code unchanged.

For example:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.packet.ai/v1",
    api_key="YOUR_PACKET_API_KEY",
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {
            "role": "user",
            "content": "Explain retrieval-augmented generation."
        }
    ],
)

That matters when you want to test another provider without rebuilding the application around a new SDK.

It also makes the inference provider easier to change later.

What does Token Factory provide?

Token Factory is Packet.ai's managed LLM inference API for open models.

The current launch catalog has six models:

Llama 3.1 8B has a 128K context window and costs $0.06 per 1M tokens. It is best suited for classification, routing, and summarization tasks.

Mistral Small 3 supports a 32K context window and is priced at $0.18 per 1M tokens. It works well for tool calling and structured output generation.

Llama 3.3 70B also supports a 128K context window and is priced at $0.59 per 1M tokens. It is a strong general-purpose model for chat, RAG systems, and agent workflows.

Qwen2.5 72B offers a 128K context window at $0.62 per 1M tokens. It is particularly useful for multilingual applications and long-context reasoning tasks.

DeepSeek-V3 supports a 64K context window and costs $0.85 per 1M tokens. It is optimized for coding, debugging, and complex reasoning problems.

BGE-M3 has an 8K context window and is priced at $0.02 per 1M tokens. It is designed for embeddings and retrieval tasks in RAG pipelines.

These are indicative launch prices. Input and output tokens are metered separately at the same listed rate for each model.

The catalog is deliberately small rather than trying to list every open model available.

The idea is to cover common production workloads with a handful of models.

Which LLM should you use?

The cheapest model is not automatically the right model.

The better question is what the model actually needs to do.

Llama 3.1 8B for high-volume tasks

If you are doing classification, routing, summarization, or structured extraction, you may not need a large model.

Llama 3.1 8B is listed at $0.06 per million tokens on Token Factory and has a 128K context window.

For simple workloads running at high volume, a smaller model can make a noticeable difference to your inference bill.

Llama 3.3 70B for general applications

For chatbots, assistants, RAG applications, and more complex instruction-following tasks, Llama 3.3 70B is the general-purpose option in the catalog.

It is currently listed at $0.59 per million tokens with a 128K context window.

DeepSeek-V3 for coding and reasoning

DeepSeek-V3 is the higher-priced model in the current Token Factory catalog at $0.85 per million tokens.

It is aimed at coding, debugging, and more complex reasoning workloads.

BGE-M3 for RAG retrieval

RAG systems have two separate jobs.

First, retrieve the relevant information.

Then, generate the answer.

BGE-M3 can handle the embedding side of that workflow. Token Factory currently lists it at $0.02 per million tokens.

This is also a good example of why using one model for everything does not always make sense.

How much does LLM inference cost?

The answer depends heavily on the model and the number of tokens you process.

For example, using Token Factory's current listed rates, 10 million output tokens would cost approximately:

Model: Llama 3.1 8B
Price per 1M tokens: $0.06
Cost for 10M output tokens: $0.60

Model: Mistral Small 3
Price per 1M tokens: $0.18
Cost for 10M output tokens: $1.80

Model: Llama 3.3 70B
Price per 1M tokens: $0.59
Cost for 10M output tokens: $5.90

Model: DeepSeek-V3
Price per 1M tokens: $0.85
Cost for 10M output tokens: $8.50

These are output-token calculations only. Input tokens are charged separately at the same model rate.

Actual application costs will depend on how many input and output tokens your application generates.

That distinction matters.

A chatbot with long prompts and short answers has a different cost profile from an agent that sends large context windows and generates long responses.

How does managed inference compare with other providers?

Price comparisons are most useful when you compare the same model.

For Llama 3.3 70B, Packet.ai currently lists the following comparison:

Packet.ai Token Factory offers Llama 3.3 70B at $0.59 per 1M input tokens and $0.59 per 1M output tokens, with OpenAI compatibility enabled.

Together AI (Llama 3.3 70B Turbo) is priced at $1.04 per 1M input tokens and $1.04 per 1M output tokens, and it is OpenAI-compatible.

Fireworks AI (Llama 3.3 70B) is priced at $0.90 per 1M input tokens and $0.90 per 1M output tokens, and it is OpenAI-compatible.

Groq (Llama 3.3 70B) is priced at $0.59 per 1M input tokens and $0.79 per 1M output tokens, and it is OpenAI-compatible.

AWS Bedrock (Llama 3.3 70B) is priced at $0.72 per 1M input tokens and $0.72 per 1M output tokens, but it is not OpenAI-compatible.

Packet.ai says these rates were verified against the providers' official pricing pages in August 2026.

The point of a comparison like this is not simply to find the lowest number.

You also need to look at API compatibility, model availability, latency, throughput, scaling, features, and whether the provider fits your workload.

Managed inference vs self-hosting

Self-hosting an open model gives you control.

It also gives you more work.

A typical self-hosted setup may involve:

Choosing the right GPU
Provisioning the machine
Installing the inference stack
Downloading model weights
Managing VRAM
Configuring vLLM or another inference engine
Handling traffic spikes
Monitoring GPU utilization
Scaling capacity
Updating the model
Handling failures

If you already have an ML infrastructure team and your GPUs stay busy, this may be worth it.

If you are building an MVP or running an application with unpredictable traffic, it may not be.

This is where managed inference can make sense.

You pay for inference instead of spending engineering time running the infrastructure.

When is self-hosting cheaper?

There is no fixed answer.

It comes down to utilization.

If your GPU is busy around the clock, paying for a dedicated GPU can make sense.

If your application receives traffic for a few hours a day and sits idle the rest of the time, you are paying for capacity you are not using.

Token Factory takes the opposite approach. Packet.ai says the service scales to zero, so idle traffic costs nothing, with billing based on tokens rather than GPU hours.

So the useful comparison is not:

GPU price vs token price

It is:

What does my workload cost at its actual utilization?

That is the number worth calculating.

Model routing can reduce inference costs

There is another way to control LLM costs: stop sending every request to the same model.

Consider a customer-support application.

A simple request such as:

"Where can I download my invoice?"

probably does not need the same model as:

"Read these three contracts and explain the differences in their termination clauses."

You could route those requests differently.

For example:

Intent classification → Llama 3.1 8B

Document retrieval → BGE-M3

General response → Llama 3.3 70B

Complex reasoning → DeepSeek-V3

This kind of routing lets you spend more on the requests that actually need it.

It also makes your inference architecture easier to optimize as usage grows.

What should you look for in an LLM inference API?

Price is important, but it should not be the only thing you compare.

1. API compatibility

Can you use your existing OpenAI SDK?

2. Model selection

Does the provider offer the models your application actually needs?

3. Input and output pricing

Are both sides of the request clearly priced?

4. Scaling

What happens when traffic increases?

What happens when traffic drops?

5. Production features

Look for streaming, structured output, tool calling, rate limits, and other features your application depends on.

6. Data handling

If you are building an enterprise application, check data retention, training policies, data residency, and compliance support.

Token Factory currently lists US and EU data residency, DPA support, and says Packet.ai does not train on customer data.

7. Fine-tuning support

If you have your own model or LoRA adapter, check whether the provider can host it.

Token Factory says teams can bring their own LoRA adapter or full checkpoint and serve it through the same OpenAI-compatible endpoint.

Who should use a managed open-model API?

A managed open-model API is worth considering if you:

Are building an AI SaaS product
Are working on an MVP
Have bursty or unpredictable traffic
Want to use open models
Do not want to manage GPUs
Want usage-based billing
Already use the OpenAI SDK
Want to test different models
Have a small engineering team
Want to keep infrastructure work out of the product roadmap

Token Factory specifically positions itself for developers, indie builders, and smaller teams that want LLM inference without managing GPU infrastructure.

Who should probably self-host?

Self-hosting may be a better fit if you:

Run millions of tokens every day
Have consistently high GPU utilization
Need a model that a managed provider does not offer
Need control over vLLM configuration
Need specific GPU hardware
Need multi-GPU serving
Already have ML infrastructure expertise

In other words, managed inference is not a replacement for self-hosting in every situation.

The workload decides.

How do you switch from OpenAI to an open-model API?

If your application already uses the OpenAI SDK, the process can be straightforward when the new provider supports the same API format.

With Token Factory, for example, the main change is the API endpoint and API key:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.packet.ai/v1",
    api_key="YOUR_PACKET_API_KEY",
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {
            "role": "user",
            "content": "Summarize this document."
        }
    ],
)

Packet.ai says the existing application code can remain unchanged when switching to its OpenAI-compatible endpoint.

For a production migration, you should still test model quality, latency, token usage, error handling, rate limits, and output consistency before switching traffic.

What is the cheapest LLM inference API?

There is no single provider that is cheapest for every model and workload.

The useful comparison is model by model.

For example, Packet.ai currently lists Llama 3.3 70B at $0.59 per million tokens, compared with $1.04 from Together AI and $0.90 from Fireworks AI.

But price should always be checked against the provider's current pricing page because inference pricing changes frequently.

What is the best API for open-source LLMs?

There is no single best API for every application.

The right provider depends on the models you need, pricing, latency, throughput, API compatibility, infrastructure, and production features.

If you want managed access to open models without operating GPUs yourself, providers such as Token Factory, Together AI, Fireworks AI, Groq, and others are worth comparing.

Is managed inference better than self-hosting?

Not automatically.

Managed inference removes infrastructure work and can make sense for variable workloads.

Self-hosting gives you more control and can make economic sense when your GPUs remain highly utilized.

The best choice depends on your workload and engineering resources.

What is an OpenAI-compatible API?

An OpenAI-compatible API follows the same general API structure and request format used by OpenAI.

This can make it easier to move an existing application between inference providers because the application does not have to be rebuilt around a completely different API.

What is Token Factory?

Token Factory is Packet.ai's managed inference API for open models.

It currently offers six models, including Llama 3.1 8B, Llama 3.3 70B, Qwen2.5 72B, DeepSeek-V3, Mistral Small 3, and BGE-M3. It supports an OpenAI-compatible API, per-token billing, scale-to-zero, streaming, tool calling, JSON mode, and fine-tune hosting.

The bigger decision is not just the model

LLM infrastructure is becoming easier to access, but that does not mean the architecture decision has disappeared.

You still need to decide:

Which model is good enough for the task?

How many tokens will the application process?

How predictable is the traffic?

Do you need to manage GPUs yourself?

How important is API compatibility?

What will the workload actually cost at scale?

For some teams, a proprietary API will still be the easiest option.

For others, self-hosting will make sense.

And for teams that want open models without running the infrastructure themselves, managed inference sits somewhere in between.

That is the space Token Factory is targeting.

Try Token Factory

Token Factory gives developers access to open models through a managed, OpenAI-compatible API.

You can compare the models, pricing, supported features, and infrastructure on the official product page:

Explore Token Factory

You can also read the full technical breakdown, including the model catalog, pricing comparison, setup instructions, and self-hosted considerations:

Read the Token Factory LLM Inference API guide

If you are already using the OpenAI SDK, the simplest test is to run an existing workload against the Token Factory endpoint and compare the things that actually matter: cost, latency, throughput, and output quality.

Renting GPUs for AI? Start with VRAM, Not the GPU

Kavya — Wed, 05 Aug 2026 10:38:11 +0000

If you're deploying an LLM for the first time, you've probably searched for something like:

What's the best GPU for AI?

It's a common question, but it's usually the wrong place to start.

Before comparing A100s, H100s, or Blackwell GPUs, answer one question first:

How much VRAM does your workload actually need?

Choosing a GPU without first sizing VRAM often results in either out-of-memory errors or paying for memory you don't need. That's why infrastructure planning should start with the workload, not the hardware.

Why VRAM Comes First

For LLM inference, GPU memory is usually the first constraint.

Your VRAM budget isn't just the model itself. It also includes:

Model weights
KV cache
Framework overhead
Production headroom

Even two deployments running the same model can require very different amounts of memory depending on context length, concurrency, and inference settings.

That's why choosing a GPU before understanding memory requirements often leads to poor infrastructure decisions.

*Model Size Doesn't Tell the Whole Story
*
Many developers assume parameter count directly determines GPU requirements.

It doesn't.

The same model can require dramatically different amounts of VRAM depending on precision and quantization.

A model running in FP16 may require multiple high-memory GPUs, while the same model using 4-bit quantization can often fit on a single accelerator.

Understanding this difference is one of the easiest ways to avoid over provisioning infrastructure.

Don't Forget the KV Cache

A deployment that works perfectly during testing can run into memory limits once it reaches production.

One of the biggest reasons is the KV cache.

Every active request stores key-value tensors in GPU memory. As context length and concurrency increase, so does memory consumption.

Ignoring KV cache is one of the most common mistakes when sizing GPUs for production inference.

Mixture-of-Experts Models Still Need Careful Planning

Another misconception is that Mixture-of-Experts (MoE) models only require memory for the experts used during inference.

While only a subset of experts is activated for each token, the model's weights still need to be loaded into GPU memory when the model starts.

That means VRAM planning remains just as important for MoE deployments.

A Better Way to Choose a GPU

Instead of asking:

Which GPU should I rent?

Start with these questions:

Which model am I deploying?
What precision or quantization will I use?
How many concurrent requests do I expect?
What context length will I support?
How much VRAM will the complete workload require?

Once those answers are clear, selecting the right GPU becomes much more straightforward.

Final Thoughts

Choosing AI infrastructure isn't about renting the newest GPU.

It's about matching your workload to the right amount of memory, compute, and infrastructure.

Starting with VRAM helps avoid deployment failures, improves GPU utilization, and prevents unnecessary infrastructure costs.

Packet.ai's guide walks through VRAM sizing for today's leading open-source models, explains common sizing mistakes, and maps workloads to GPU configurations to help teams make better infrastructure decisions.

Read the full guide:
https://packet.ai/blog/rent-gpu-for-ai-vram-requirements-guide

Why memory bandwidth matters more than TFLOPS for LLM inference

Kavya — Mon, 03 Aug 2026 14:46:45 +0000

Your GPU's TFLOPS rating does not decide how fast it generates tokens. Memory bandwidth does. Here's the spec that actually separates A100-era inference from H200-era inference.

Key takeaways

HBM3e reaches up to 9.6 Gb/s per pin versus HBM2e's 3.2 Gb/s, tripling per-pin signalling speed with the same 1,024-bit interface width.
Total GPU memory bandwidth jumps from 2.0 TB/s on the A100 (HBM2e) to 4.8 TB/s on the H200 (HBM3e), a 140% increase, and 8.0 TB/s on the B200.
LLM token generation (decode phase) is memory-bound: tokens per second tracks memory bandwidth far more closely than TFLOPS.
H200 achieves 31,712 tokens/sec on Llama 2 70B with TensorRT-LLM, 45% faster than H100's 21,806 tokens/sec, driven entirely by the HBM3e memory upgrade.
HBM3e is 2.5x more power-efficient per watt than HBM2e, delivering higher bandwidth at lower voltage (1.1V versus 1.2V).

What is HBM and how does it work?

High Bandwidth Memory is a form of stacked DRAM co-packaged with the GPU die on a shared silicon interposer, connected through Through-Silicon Vias (TSVs). Unlike GDDR6 memory sitting on separate chips across a PCB trace, HBM stacks sit microns from the compute die. The interface is wider, latency is lower, and bandwidth per watt is substantially better.

For reference: GDDR6X on a consumer GPU achieves roughly 1.0 TB/s. HBM2e on the A100 doubles that at 2.0 TB/s. HBM3e on the H200 SXM more than doubles HBM2e again at 4.8 TB/s. Each generation achieves this by increasing per-pin signalling speed, not by widening the interface bus.

HBM3e vs HBM2e: the numbers

Specification	HBM2e	HBM3	HBM3e
Flagship GPU	A100 SXM	H100 SXM	H200 / B200 SXM
Total bandwidth	2.0 TB/s	3.35 TB/s	4.8 TB/s (H200) / 8.0 TB/s (B200)
Pin speed	~3.2 Gb/s	6.4 Gb/s	up to 9.6 Gb/s
Memory capacity	80 GB	80 GB	141 to 192 GB
Operating voltage	1.2V	1.1V	1.1V
Power efficiency vs HBM2e	1x	~2x	2.5x

Why LLM decode is memory-bound, not compute-bound

LLM inference runs in two phases. The prefill phase processes the full input prompt in one forward pass and is compute-bound. The decode phase generates one token per forward pass and is memory-bound: every token requires loading the complete model weight matrix and accumulated KV cache from HBM before a small amount of arithmetic work.

At batch size 1, arithmetic intensity falls as low as 1 to 2 FLOP per byte — far below the compute-to-bandwidth ratio of even the A100. Memory bandwidth is the first bottleneck for real-world serving workloads at batch sizes below roughly 64 to 128, which covers most production inference deployments.

This is why upgrading from an A100 (HBM2e, 2.0 TB/s) to an H200 (HBM3e, 4.8 TB/s) improves inference throughput by 2 to 3x on 70B parameter models.

Benchmark: H200 vs H100 on Llama 2 70B

GPU	HBM Type	Bandwidth	Llama 2 70B (tok/s)	vs H100
H200 SXM	HBM3e	4.8 TB/s	31,712	+45%
H100 SXM	HBM3	3.35 TB/s	21,806	baseline