DEV Community: Kavya

LLM Inference APIs in 2026: How to Choose Between OpenAI, Open Models, and Self-Hosted GPUs

Kavya — Thu, 13 Aug 2026 15:06:27 +0000

If you are building an AI application, choosing the model is only part of the problem.

Sooner or later, you also have to decide how you are going to run that model.

You can call a proprietary model through an API such as OpenAI. You can deploy an open model yourself on a GPU using tools such as vLLM. Or you can use a managed inference API that hosts open models for you.

All three work. The right choice depends on your traffic, model requirements, engineering resources, and how much you are willing to spend on inference.

For developers working with open models, managed inference is becoming an interesting middle ground. You get an API instead of a GPU to manage, while keeping access to models such as Llama, Qwen, DeepSeek, and Mistral.

That is the problem Token Factory from Packet.ai is built around.

What is an LLM inference API?

An LLM inference API gives your application access to a hosted language model through an API.

Your application sends a request. The provider runs the model on its infrastructure and returns the response.

You do not have to provision a GPU, download model weights, configure an inference server, or keep track of GPU capacity.

The basic setup looks like this:

Your application → LLM API → model → response

This sounds simple, but the infrastructure behind that API can be quite complicated.

Running an open model yourself can involve GPU selection, VRAM requirements, model loading, inference engines, autoscaling, monitoring, concurrency, and capacity planning.

A managed inference provider takes care of that layer.

What are the main ways to run an LLM?

There are three common approaches.

Option	When it makes sense	What you get	What you give up
Proprietary API	You want a simple way to use leading closed models	Easy setup and managed infrastructure	Higher costs at large token volumes
Managed open-model API	You want open models without running GPUs	Lower-cost inference with managed infrastructure	Your choices depend on the provider's model lineup
Self-hosted	You have high, predictable usage and an infrastructure team	Full control over models and serving	GPU costs and the work of running the stack

There is no universal winner here.

A startup processing a few million tokens a month has a very different infrastructure problem from a company processing billions.

That is why comparing providers only on model quality or price per million tokens can be misleading.

Why use an open-model inference API?

Open models have changed the economics of running AI applications.

Models such as Llama, Qwen, DeepSeek, and Mistral can be used for many production workloads without requiring developers to build their own model-serving infrastructure.

That makes it possible to separate two decisions:

Which model should run the workload?

and

Who should operate the infrastructure?

You can choose an open model while letting someone else handle the GPUs.

This is the main appeal of a managed open-model inference API.

Why does OpenAI compatibility matter?

This is one of the most practical things to look for when choosing an inference provider.

If your application already uses the OpenAI SDK, moving to another API can be annoying if you have to change your entire inference layer.

An OpenAI-compatible API reduces that work.

Token Factory uses an OpenAI-compatible API. Packet.ai says developers can change the base_url to api.packet.ai/v1 and replace the API key while keeping their application code unchanged.

For example:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.packet.ai/v1",
    api_key="YOUR_PACKET_API_KEY",
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {
            "role": "user",
            "content": "Explain retrieval-augmented generation."
        }
    ],
)

That matters when you want to test another provider without rebuilding the application around a new SDK.

It also makes the inference provider easier to change later.

What does Token Factory provide?

Token Factory is Packet.ai's managed LLM inference API for open models.

The current launch catalog has six models:

Llama 3.1 8B has a 128K context window and costs $0.06 per 1M tokens. It is best suited for classification, routing, and summarization tasks.

Mistral Small 3 supports a 32K context window and is priced at $0.18 per 1M tokens. It works well for tool calling and structured output generation.

Llama 3.3 70B also supports a 128K context window and is priced at $0.59 per 1M tokens. It is a strong general-purpose model for chat, RAG systems, and agent workflows.

Qwen2.5 72B offers a 128K context window at $0.62 per 1M tokens. It is particularly useful for multilingual applications and long-context reasoning tasks.

DeepSeek-V3 supports a 64K context window and costs $0.85 per 1M tokens. It is optimized for coding, debugging, and complex reasoning problems.

BGE-M3 has an 8K context window and is priced at $0.02 per 1M tokens. It is designed for embeddings and retrieval tasks in RAG pipelines.

These are indicative launch prices. Input and output tokens are metered separately at the same listed rate for each model.

The catalog is deliberately small rather than trying to list every open model available.

The idea is to cover common production workloads with a handful of models.

Which LLM should you use?

The cheapest model is not automatically the right model.

The better question is what the model actually needs to do.

Llama 3.1 8B for high-volume tasks

If you are doing classification, routing, summarization, or structured extraction, you may not need a large model.

Llama 3.1 8B is listed at $0.06 per million tokens on Token Factory and has a 128K context window.

For simple workloads running at high volume, a smaller model can make a noticeable difference to your inference bill.

Llama 3.3 70B for general applications

For chatbots, assistants, RAG applications, and more complex instruction-following tasks, Llama 3.3 70B is the general-purpose option in the catalog.

It is currently listed at $0.59 per million tokens with a 128K context window.

DeepSeek-V3 for coding and reasoning

DeepSeek-V3 is the higher-priced model in the current Token Factory catalog at $0.85 per million tokens.

It is aimed at coding, debugging, and more complex reasoning workloads.

BGE-M3 for RAG retrieval

RAG systems have two separate jobs.

First, retrieve the relevant information.

Then, generate the answer.

BGE-M3 can handle the embedding side of that workflow. Token Factory currently lists it at $0.02 per million tokens.

This is also a good example of why using one model for everything does not always make sense.

How much does LLM inference cost?

The answer depends heavily on the model and the number of tokens you process.

For example, using Token Factory's current listed rates, 10 million output tokens would cost approximately:

Model: Llama 3.1 8B
Price per 1M tokens: $0.06
Cost for 10M output tokens: $0.60

Model: Mistral Small 3
Price per 1M tokens: $0.18
Cost for 10M output tokens: $1.80

Model: Llama 3.3 70B
Price per 1M tokens: $0.59
Cost for 10M output tokens: $5.90

Model: DeepSeek-V3
Price per 1M tokens: $0.85
Cost for 10M output tokens: $8.50

These are output-token calculations only. Input tokens are charged separately at the same model rate.

Actual application costs will depend on how many input and output tokens your application generates.

That distinction matters.

A chatbot with long prompts and short answers has a different cost profile from an agent that sends large context windows and generates long responses.

How does managed inference compare with other providers?

Price comparisons are most useful when you compare the same model.

For Llama 3.3 70B, Packet.ai currently lists the following comparison:

Packet.ai Token Factory offers Llama 3.3 70B at $0.59 per 1M input tokens and $0.59 per 1M output tokens, with OpenAI compatibility enabled.

Together AI (Llama 3.3 70B Turbo) is priced at $1.04 per 1M input tokens and $1.04 per 1M output tokens, and it is OpenAI-compatible.

Fireworks AI (Llama 3.3 70B) is priced at $0.90 per 1M input tokens and $0.90 per 1M output tokens, and it is OpenAI-compatible.

Groq (Llama 3.3 70B) is priced at $0.59 per 1M input tokens and $0.79 per 1M output tokens, and it is OpenAI-compatible.

AWS Bedrock (Llama 3.3 70B) is priced at $0.72 per 1M input tokens and $0.72 per 1M output tokens, but it is not OpenAI-compatible.

Packet.ai says these rates were verified against the providers' official pricing pages in August 2026.

The point of a comparison like this is not simply to find the lowest number.

You also need to look at API compatibility, model availability, latency, throughput, scaling, features, and whether the provider fits your workload.

Managed inference vs self-hosting

Self-hosting an open model gives you control.

It also gives you more work.

A typical self-hosted setup may involve:

Choosing the right GPU
Provisioning the machine
Installing the inference stack
Downloading model weights
Managing VRAM
Configuring vLLM or another inference engine
Handling traffic spikes
Monitoring GPU utilization
Scaling capacity
Updating the model
Handling failures

If you already have an ML infrastructure team and your GPUs stay busy, this may be worth it.

If you are building an MVP or running an application with unpredictable traffic, it may not be.

This is where managed inference can make sense.

You pay for inference instead of spending engineering time running the infrastructure.

When is self-hosting cheaper?

There is no fixed answer.

It comes down to utilization.

If your GPU is busy around the clock, paying for a dedicated GPU can make sense.

If your application receives traffic for a few hours a day and sits idle the rest of the time, you are paying for capacity you are not using.

Token Factory takes the opposite approach. Packet.ai says the service scales to zero, so idle traffic costs nothing, with billing based on tokens rather than GPU hours.

So the useful comparison is not:

GPU price vs token price

It is:

What does my workload cost at its actual utilization?

That is the number worth calculating.

Model routing can reduce inference costs

There is another way to control LLM costs: stop sending every request to the same model.

Consider a customer-support application.

A simple request such as:

"Where can I download my invoice?"

probably does not need the same model as:

"Read these three contracts and explain the differences in their termination clauses."

You could route those requests differently.

For example:

Intent classification → Llama 3.1 8B

Document retrieval → BGE-M3

General response → Llama 3.3 70B

Complex reasoning → DeepSeek-V3

This kind of routing lets you spend more on the requests that actually need it.

It also makes your inference architecture easier to optimize as usage grows.

What should you look for in an LLM inference API?

Price is important, but it should not be the only thing you compare.

1. API compatibility

Can you use your existing OpenAI SDK?

2. Model selection

Does the provider offer the models your application actually needs?

3. Input and output pricing

Are both sides of the request clearly priced?

4. Scaling

What happens when traffic increases?

What happens when traffic drops?

5. Production features

Look for streaming, structured output, tool calling, rate limits, and other features your application depends on.

6. Data handling

If you are building an enterprise application, check data retention, training policies, data residency, and compliance support.

Token Factory currently lists US and EU data residency, DPA support, and says Packet.ai does not train on customer data.

7. Fine-tuning support

If you have your own model or LoRA adapter, check whether the provider can host it.

Token Factory says teams can bring their own LoRA adapter or full checkpoint and serve it through the same OpenAI-compatible endpoint.

Who should use a managed open-model API?

A managed open-model API is worth considering if you:

Are building an AI SaaS product
Are working on an MVP
Have bursty or unpredictable traffic
Want to use open models
Do not want to manage GPUs
Want usage-based billing
Already use the OpenAI SDK
Want to test different models
Have a small engineering team
Want to keep infrastructure work out of the product roadmap

Token Factory specifically positions itself for developers, indie builders, and smaller teams that want LLM inference without managing GPU infrastructure.

Who should probably self-host?

Self-hosting may be a better fit if you:

Run millions of tokens every day
Have consistently high GPU utilization
Need a model that a managed provider does not offer
Need control over vLLM configuration
Need specific GPU hardware
Need multi-GPU serving
Already have ML infrastructure expertise

In other words, managed inference is not a replacement for self-hosting in every situation.

The workload decides.

How do you switch from OpenAI to an open-model API?

If your application already uses the OpenAI SDK, the process can be straightforward when the new provider supports the same API format.

With Token Factory, for example, the main change is the API endpoint and API key:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.packet.ai/v1",
    api_key="YOUR_PACKET_API_KEY",
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {
            "role": "user",
            "content": "Summarize this document."
        }
    ],
)

Packet.ai says the existing application code can remain unchanged when switching to its OpenAI-compatible endpoint.

For a production migration, you should still test model quality, latency, token usage, error handling, rate limits, and output consistency before switching traffic.

What is the cheapest LLM inference API?

There is no single provider that is cheapest for every model and workload.

The useful comparison is model by model.

For example, Packet.ai currently lists Llama 3.3 70B at $0.59 per million tokens, compared with $1.04 from Together AI and $0.90 from Fireworks AI.

But price should always be checked against the provider's current pricing page because inference pricing changes frequently.

What is the best API for open-source LLMs?

There is no single best API for every application.

The right provider depends on the models you need, pricing, latency, throughput, API compatibility, infrastructure, and production features.

If you want managed access to open models without operating GPUs yourself, providers such as Token Factory, Together AI, Fireworks AI, Groq, and others are worth comparing.

Is managed inference better than self-hosting?

Not automatically.

Managed inference removes infrastructure work and can make sense for variable workloads.

Self-hosting gives you more control and can make economic sense when your GPUs remain highly utilized.

The best choice depends on your workload and engineering resources.

What is an OpenAI-compatible API?

An OpenAI-compatible API follows the same general API structure and request format used by OpenAI.

This can make it easier to move an existing application between inference providers because the application does not have to be rebuilt around a completely different API.

What is Token Factory?

Token Factory is Packet.ai's managed inference API for open models.

It currently offers six models, including Llama 3.1 8B, Llama 3.3 70B, Qwen2.5 72B, DeepSeek-V3, Mistral Small 3, and BGE-M3. It supports an OpenAI-compatible API, per-token billing, scale-to-zero, streaming, tool calling, JSON mode, and fine-tune hosting.

The bigger decision is not just the model

LLM infrastructure is becoming easier to access, but that does not mean the architecture decision has disappeared.

You still need to decide:

Which model is good enough for the task?

How many tokens will the application process?

How predictable is the traffic?

Do you need to manage GPUs yourself?

How important is API compatibility?

What will the workload actually cost at scale?

For some teams, a proprietary API will still be the easiest option.

For others, self-hosting will make sense.

And for teams that want open models without running the infrastructure themselves, managed inference sits somewhere in between.

That is the space Token Factory is targeting.

Try Token Factory

Token Factory gives developers access to open models through a managed, OpenAI-compatible API.

You can compare the models, pricing, supported features, and infrastructure on the official product page:

Explore Token Factory

You can also read the full technical breakdown, including the model catalog, pricing comparison, setup instructions, and self-hosted considerations:

Read the Token Factory LLM Inference API guide

If you are already using the OpenAI SDK, the simplest test is to run an existing workload against the Token Factory endpoint and compare the things that actually matter: cost, latency, throughput, and output quality.

Renting GPUs for AI? Start with VRAM, Not the GPU

Kavya — Wed, 05 Aug 2026 10:38:11 +0000

If you're deploying an LLM for the first time, you've probably searched for something like:

What's the best GPU for AI?

It's a common question, but it's usually the wrong place to start.

Before comparing A100s, H100s, or Blackwell GPUs, answer one question first:

How much VRAM does your workload actually need?

Choosing a GPU without first sizing VRAM often results in either out-of-memory errors or paying for memory you don't need. That's why infrastructure planning should start with the workload, not the hardware.

Why VRAM Comes First

For LLM inference, GPU memory is usually the first constraint.

Your VRAM budget isn't just the model itself. It also includes:

Model weights
KV cache
Framework overhead
Production headroom

Even two deployments running the same model can require very different amounts of memory depending on context length, concurrency, and inference settings.

That's why choosing a GPU before understanding memory requirements often leads to poor infrastructure decisions.

*Model Size Doesn't Tell the Whole Story
*
Many developers assume parameter count directly determines GPU requirements.

It doesn't.

The same model can require dramatically different amounts of VRAM depending on precision and quantization.

A model running in FP16 may require multiple high-memory GPUs, while the same model using 4-bit quantization can often fit on a single accelerator.

Understanding this difference is one of the easiest ways to avoid over provisioning infrastructure.

Don't Forget the KV Cache

A deployment that works perfectly during testing can run into memory limits once it reaches production.

One of the biggest reasons is the KV cache.

Every active request stores key-value tensors in GPU memory. As context length and concurrency increase, so does memory consumption.

Ignoring KV cache is one of the most common mistakes when sizing GPUs for production inference.

Mixture-of-Experts Models Still Need Careful Planning

Another misconception is that Mixture-of-Experts (MoE) models only require memory for the experts used during inference.

While only a subset of experts is activated for each token, the model's weights still need to be loaded into GPU memory when the model starts.

That means VRAM planning remains just as important for MoE deployments.

A Better Way to Choose a GPU

Instead of asking:

Which GPU should I rent?

Start with these questions:

Which model am I deploying?
What precision or quantization will I use?
How many concurrent requests do I expect?
What context length will I support?
How much VRAM will the complete workload require?

Once those answers are clear, selecting the right GPU becomes much more straightforward.

Final Thoughts

Choosing AI infrastructure isn't about renting the newest GPU.

It's about matching your workload to the right amount of memory, compute, and infrastructure.

Starting with VRAM helps avoid deployment failures, improves GPU utilization, and prevents unnecessary infrastructure costs.

Packet.ai's guide walks through VRAM sizing for today's leading open-source models, explains common sizing mistakes, and maps workloads to GPU configurations to help teams make better infrastructure decisions.

Read the full guide:
https://packet.ai/blog/rent-gpu-for-ai-vram-requirements-guide

Why memory bandwidth matters more than TFLOPS for LLM inference

Kavya — Mon, 03 Aug 2026 14:46:45 +0000

Your GPU's TFLOPS rating does not decide how fast it generates tokens. Memory bandwidth does. Here's the spec that actually separates A100-era inference from H200-era inference.

Key takeaways

HBM3e reaches up to 9.6 Gb/s per pin versus HBM2e's 3.2 Gb/s, tripling per-pin signalling speed with the same 1,024-bit interface width.
Total GPU memory bandwidth jumps from 2.0 TB/s on the A100 (HBM2e) to 4.8 TB/s on the H200 (HBM3e), a 140% increase, and 8.0 TB/s on the B200.
LLM token generation (decode phase) is memory-bound: tokens per second tracks memory bandwidth far more closely than TFLOPS.
H200 achieves 31,712 tokens/sec on Llama 2 70B with TensorRT-LLM, 45% faster than H100's 21,806 tokens/sec, driven entirely by the HBM3e memory upgrade.
HBM3e is 2.5x more power-efficient per watt than HBM2e, delivering higher bandwidth at lower voltage (1.1V versus 1.2V).

What is HBM and how does it work?

High Bandwidth Memory is a form of stacked DRAM co-packaged with the GPU die on a shared silicon interposer, connected through Through-Silicon Vias (TSVs). Unlike GDDR6 memory sitting on separate chips across a PCB trace, HBM stacks sit microns from the compute die. The interface is wider, latency is lower, and bandwidth per watt is substantially better.

For reference: GDDR6X on a consumer GPU achieves roughly 1.0 TB/s. HBM2e on the A100 doubles that at 2.0 TB/s. HBM3e on the H200 SXM more than doubles HBM2e again at 4.8 TB/s. Each generation achieves this by increasing per-pin signalling speed, not by widening the interface bus.

HBM3e vs HBM2e: the numbers

Specification	HBM2e	HBM3	HBM3e
Flagship GPU	A100 SXM	H100 SXM	H200 / B200 SXM
Total bandwidth	2.0 TB/s	3.35 TB/s	4.8 TB/s (H200) / 8.0 TB/s (B200)
Pin speed	~3.2 Gb/s	6.4 Gb/s	up to 9.6 Gb/s
Memory capacity	80 GB	80 GB	141 to 192 GB
Operating voltage	1.2V	1.1V	1.1V
Power efficiency vs HBM2e	1x	~2x	2.5x

Why LLM decode is memory-bound, not compute-bound

LLM inference runs in two phases. The prefill phase processes the full input prompt in one forward pass and is compute-bound. The decode phase generates one token per forward pass and is memory-bound: every token requires loading the complete model weight matrix and accumulated KV cache from HBM before a small amount of arithmetic work.

At batch size 1, arithmetic intensity falls as low as 1 to 2 FLOP per byte — far below the compute-to-bandwidth ratio of even the A100. Memory bandwidth is the first bottleneck for real-world serving workloads at batch sizes below roughly 64 to 128, which covers most production inference deployments.

This is why upgrading from an A100 (HBM2e, 2.0 TB/s) to an H200 (HBM3e, 4.8 TB/s) improves inference throughput by 2 to 3x on 70B parameter models.

Benchmark: H200 vs H100 on Llama 2 70B

GPU	HBM Type	Bandwidth	Llama 2 70B (tok/s)	vs H100
H200 SXM	HBM3e	4.8 TB/s	31,712	+45%
H100 SXM	HBM3	3.35 TB/s	21,806	baseline