If you are building an AI application, choosing the model is only part of the problem.
Sooner or later, you also have to decide how you are going to run that model.
You can call a proprietary model through an API such as OpenAI. You can deploy an open model yourself on a GPU using tools such as vLLM. Or you can use a managed inference API that hosts open models for you.
All three work. The right choice depends on your traffic, model requirements, engineering resources, and how much you are willing to spend on inference.
For developers working with open models, managed inference is becoming an interesting middle ground. You get an API instead of a GPU to manage, while keeping access to models such as Llama, Qwen, DeepSeek, and Mistral.
That is the problem Token Factory from Packet.ai is built around.
What is an LLM inference API?
An LLM inference API gives your application access to a hosted language model through an API.
Your application sends a request. The provider runs the model on its infrastructure and returns the response.
You do not have to provision a GPU, download model weights, configure an inference server, or keep track of GPU capacity.
The basic setup looks like this:
Your application → LLM API → model → response
This sounds simple, but the infrastructure behind that API can be quite complicated.
Running an open model yourself can involve GPU selection, VRAM requirements, model loading, inference engines, autoscaling, monitoring, concurrency, and capacity planning.
A managed inference provider takes care of that layer.
What are the main ways to run an LLM?
There are three common approaches.
| Option | When it makes sense | What you get | What you give up |
|---|---|---|---|
| Proprietary API | You want a simple way to use leading closed models | Easy setup and managed infrastructure | Higher costs at large token volumes |
| Managed open-model API | You want open models without running GPUs | Lower-cost inference with managed infrastructure | Your choices depend on the provider's model lineup |
| Self-hosted | You have high, predictable usage and an infrastructure team | Full control over models and serving | GPU costs and the work of running the stack |
There is no universal winner here.
A startup processing a few million tokens a month has a very different infrastructure problem from a company processing billions.
That is why comparing providers only on model quality or price per million tokens can be misleading.
Why use an open-model inference API?
Open models have changed the economics of running AI applications.
Models such as Llama, Qwen, DeepSeek, and Mistral can be used for many production workloads without requiring developers to build their own model-serving infrastructure.
That makes it possible to separate two decisions:
Which model should run the workload?
and
Who should operate the infrastructure?
You can choose an open model while letting someone else handle the GPUs.
This is the main appeal of a managed open-model inference API.
Why does OpenAI compatibility matter?
This is one of the most practical things to look for when choosing an inference provider.
If your application already uses the OpenAI SDK, moving to another API can be annoying if you have to change your entire inference layer.
An OpenAI-compatible API reduces that work.
Token Factory uses an OpenAI-compatible API. Packet.ai says developers can change the base_url to api.packet.ai/v1 and replace the API key while keeping their application code unchanged.
For example:
from openai import OpenAI
client = OpenAI(
base_url="https://api.packet.ai/v1",
api_key="YOUR_PACKET_API_KEY",
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{
"role": "user",
"content": "Explain retrieval-augmented generation."
}
],
)
That matters when you want to test another provider without rebuilding the application around a new SDK.
It also makes the inference provider easier to change later.
What does Token Factory provide?
Token Factory is Packet.ai's managed LLM inference API for open models.
The current launch catalog has six models:
Llama 3.1 8B has a 128K context window and costs $0.06 per 1M tokens. It is best suited for classification, routing, and summarization tasks.
Mistral Small 3 supports a 32K context window and is priced at $0.18 per 1M tokens. It works well for tool calling and structured output generation.
Llama 3.3 70B also supports a 128K context window and is priced at $0.59 per 1M tokens. It is a strong general-purpose model for chat, RAG systems, and agent workflows.
Qwen2.5 72B offers a 128K context window at $0.62 per 1M tokens. It is particularly useful for multilingual applications and long-context reasoning tasks.
DeepSeek-V3 supports a 64K context window and costs $0.85 per 1M tokens. It is optimized for coding, debugging, and complex reasoning problems.
BGE-M3 has an 8K context window and is priced at $0.02 per 1M tokens. It is designed for embeddings and retrieval tasks in RAG pipelines.
These are indicative launch prices. Input and output tokens are metered separately at the same listed rate for each model.
The catalog is deliberately small rather than trying to list every open model available.
The idea is to cover common production workloads with a handful of models.
Which LLM should you use?
The cheapest model is not automatically the right model.
The better question is what the model actually needs to do.
Llama 3.1 8B for high-volume tasks
If you are doing classification, routing, summarization, or structured extraction, you may not need a large model.
Llama 3.1 8B is listed at $0.06 per million tokens on Token Factory and has a 128K context window.
For simple workloads running at high volume, a smaller model can make a noticeable difference to your inference bill.
Llama 3.3 70B for general applications
For chatbots, assistants, RAG applications, and more complex instruction-following tasks, Llama 3.3 70B is the general-purpose option in the catalog.
It is currently listed at $0.59 per million tokens with a 128K context window.
DeepSeek-V3 for coding and reasoning
DeepSeek-V3 is the higher-priced model in the current Token Factory catalog at $0.85 per million tokens.
It is aimed at coding, debugging, and more complex reasoning workloads.
BGE-M3 for RAG retrieval
RAG systems have two separate jobs.
First, retrieve the relevant information.
Then, generate the answer.
BGE-M3 can handle the embedding side of that workflow. Token Factory currently lists it at $0.02 per million tokens.
This is also a good example of why using one model for everything does not always make sense.
How much does LLM inference cost?
The answer depends heavily on the model and the number of tokens you process.
For example, using Token Factory's current listed rates, 10 million output tokens would cost approximately:
Model: Llama 3.1 8B
Price per 1M tokens: $0.06
Cost for 10M output tokens: $0.60
Model: Mistral Small 3
Price per 1M tokens: $0.18
Cost for 10M output tokens: $1.80
Model: Llama 3.3 70B
Price per 1M tokens: $0.59
Cost for 10M output tokens: $5.90
Model: DeepSeek-V3
Price per 1M tokens: $0.85
Cost for 10M output tokens: $8.50
These are output-token calculations only. Input tokens are charged separately at the same model rate.
Actual application costs will depend on how many input and output tokens your application generates.
That distinction matters.
A chatbot with long prompts and short answers has a different cost profile from an agent that sends large context windows and generates long responses.
How does managed inference compare with other providers?
Price comparisons are most useful when you compare the same model.
For Llama 3.3 70B, Packet.ai currently lists the following comparison:
Packet.ai Token Factory offers Llama 3.3 70B at $0.59 per 1M input tokens and $0.59 per 1M output tokens, with OpenAI compatibility enabled.
Together AI (Llama 3.3 70B Turbo) is priced at $1.04 per 1M input tokens and $1.04 per 1M output tokens, and it is OpenAI-compatible.
Fireworks AI (Llama 3.3 70B) is priced at $0.90 per 1M input tokens and $0.90 per 1M output tokens, and it is OpenAI-compatible.
Groq (Llama 3.3 70B) is priced at $0.59 per 1M input tokens and $0.79 per 1M output tokens, and it is OpenAI-compatible.
AWS Bedrock (Llama 3.3 70B) is priced at $0.72 per 1M input tokens and $0.72 per 1M output tokens, but it is not OpenAI-compatible.
Packet.ai says these rates were verified against the providers' official pricing pages in August 2026.
The point of a comparison like this is not simply to find the lowest number.
You also need to look at API compatibility, model availability, latency, throughput, scaling, features, and whether the provider fits your workload.
Managed inference vs self-hosting
Self-hosting an open model gives you control.
It also gives you more work.
A typical self-hosted setup may involve:
- Choosing the right GPU
- Provisioning the machine
- Installing the inference stack
- Downloading model weights
- Managing VRAM
- Configuring vLLM or another inference engine
- Handling traffic spikes
- Monitoring GPU utilization
- Scaling capacity
- Updating the model
- Handling failures
If you already have an ML infrastructure team and your GPUs stay busy, this may be worth it.
If you are building an MVP or running an application with unpredictable traffic, it may not be.
This is where managed inference can make sense.
You pay for inference instead of spending engineering time running the infrastructure.
When is self-hosting cheaper?
There is no fixed answer.
It comes down to utilization.
If your GPU is busy around the clock, paying for a dedicated GPU can make sense.
If your application receives traffic for a few hours a day and sits idle the rest of the time, you are paying for capacity you are not using.
Token Factory takes the opposite approach. Packet.ai says the service scales to zero, so idle traffic costs nothing, with billing based on tokens rather than GPU hours.
So the useful comparison is not:
GPU price vs token price
It is:
What does my workload cost at its actual utilization?
That is the number worth calculating.
Model routing can reduce inference costs
There is another way to control LLM costs: stop sending every request to the same model.
Consider a customer-support application.
A simple request such as:
"Where can I download my invoice?"
probably does not need the same model as:
"Read these three contracts and explain the differences in their termination clauses."
You could route those requests differently.
For example:
Intent classification → Llama 3.1 8B
Document retrieval → BGE-M3
General response → Llama 3.3 70B
Complex reasoning → DeepSeek-V3
This kind of routing lets you spend more on the requests that actually need it.
It also makes your inference architecture easier to optimize as usage grows.
What should you look for in an LLM inference API?
Price is important, but it should not be the only thing you compare.
1. API compatibility
Can you use your existing OpenAI SDK?
2. Model selection
Does the provider offer the models your application actually needs?
3. Input and output pricing
Are both sides of the request clearly priced?
4. Scaling
What happens when traffic increases?
What happens when traffic drops?
5. Production features
Look for streaming, structured output, tool calling, rate limits, and other features your application depends on.
6. Data handling
If you are building an enterprise application, check data retention, training policies, data residency, and compliance support.
Token Factory currently lists US and EU data residency, DPA support, and says Packet.ai does not train on customer data.
7. Fine-tuning support
If you have your own model or LoRA adapter, check whether the provider can host it.
Token Factory says teams can bring their own LoRA adapter or full checkpoint and serve it through the same OpenAI-compatible endpoint.
Who should use a managed open-model API?
A managed open-model API is worth considering if you:
- Are building an AI SaaS product
- Are working on an MVP
- Have bursty or unpredictable traffic
- Want to use open models
- Do not want to manage GPUs
- Want usage-based billing
- Already use the OpenAI SDK
- Want to test different models
- Have a small engineering team
- Want to keep infrastructure work out of the product roadmap
Token Factory specifically positions itself for developers, indie builders, and smaller teams that want LLM inference without managing GPU infrastructure.
Who should probably self-host?
Self-hosting may be a better fit if you:
- Run millions of tokens every day
- Have consistently high GPU utilization
- Need a model that a managed provider does not offer
- Need control over vLLM configuration
- Need specific GPU hardware
- Need multi-GPU serving
- Already have ML infrastructure expertise
In other words, managed inference is not a replacement for self-hosting in every situation.
The workload decides.
How do you switch from OpenAI to an open-model API?
If your application already uses the OpenAI SDK, the process can be straightforward when the new provider supports the same API format.
With Token Factory, for example, the main change is the API endpoint and API key:
from openai import OpenAI
client = OpenAI(
base_url="https://api.packet.ai/v1",
api_key="YOUR_PACKET_API_KEY",
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{
"role": "user",
"content": "Summarize this document."
}
],
)
Packet.ai says the existing application code can remain unchanged when switching to its OpenAI-compatible endpoint.
For a production migration, you should still test model quality, latency, token usage, error handling, rate limits, and output consistency before switching traffic.
What is the cheapest LLM inference API?
There is no single provider that is cheapest for every model and workload.
The useful comparison is model by model.
For example, Packet.ai currently lists Llama 3.3 70B at $0.59 per million tokens, compared with $1.04 from Together AI and $0.90 from Fireworks AI.
But price should always be checked against the provider's current pricing page because inference pricing changes frequently.
What is the best API for open-source LLMs?
There is no single best API for every application.
The right provider depends on the models you need, pricing, latency, throughput, API compatibility, infrastructure, and production features.
If you want managed access to open models without operating GPUs yourself, providers such as Token Factory, Together AI, Fireworks AI, Groq, and others are worth comparing.
Is managed inference better than self-hosting?
Not automatically.
Managed inference removes infrastructure work and can make sense for variable workloads.
Self-hosting gives you more control and can make economic sense when your GPUs remain highly utilized.
The best choice depends on your workload and engineering resources.
What is an OpenAI-compatible API?
An OpenAI-compatible API follows the same general API structure and request format used by OpenAI.
This can make it easier to move an existing application between inference providers because the application does not have to be rebuilt around a completely different API.
What is Token Factory?
Token Factory is Packet.ai's managed inference API for open models.
It currently offers six models, including Llama 3.1 8B, Llama 3.3 70B, Qwen2.5 72B, DeepSeek-V3, Mistral Small 3, and BGE-M3. It supports an OpenAI-compatible API, per-token billing, scale-to-zero, streaming, tool calling, JSON mode, and fine-tune hosting.
The bigger decision is not just the model
LLM infrastructure is becoming easier to access, but that does not mean the architecture decision has disappeared.
You still need to decide:
Which model is good enough for the task?
How many tokens will the application process?
How predictable is the traffic?
Do you need to manage GPUs yourself?
How important is API compatibility?
What will the workload actually cost at scale?
For some teams, a proprietary API will still be the easiest option.
For others, self-hosting will make sense.
And for teams that want open models without running the infrastructure themselves, managed inference sits somewhere in between.
That is the space Token Factory is targeting.
Try Token Factory
Token Factory gives developers access to open models through a managed, OpenAI-compatible API.
You can compare the models, pricing, supported features, and infrastructure on the official product page:
You can also read the full technical breakdown, including the model catalog, pricing comparison, setup instructions, and self-hosted considerations:
Read the Token Factory LLM Inference API guide
If you are already using the OpenAI SDK, the simplest test is to run an existing workload against the Token Factory endpoint and compare the things that actually matter: cost, latency, throughput, and output quality.
Top comments (0)