Right now, as of this writing, Together AI's on-demand H100 rate is running a promotional $3.99/hour instead of the standard $5.49/hour — and that promotion is explicitly time-boxed to end September 30, 2026. If you're pricing out a dedicated deployment this month using that number, it's worth building your actual budget around the post-promo rate, not the one currently on the page, since a plan built on a temporary rate understates real cost by close to 40% once it lapses. That kind of detail — current, dated, and easy to miss if you're reading a guide from even a few weeks earlier — is a decent illustration of why "what is Together AI" is worth revisiting rather than assuming an older mental model still applies.
The distinction that actually matters: host, not router
I've written about several routing gateways in this space — OpenRouter, LiteLLM, RouteAI — and it's easy to mentally file Together AI into the same category, since the API shape and the pitch ("access many models through one key") sound similar on the surface. They're not the same kind of product. A router forwards your request to whichever upstream provider — OpenAI, Anthropic, Moonshot, DeepSeek, whoever — actually runs the model. Together AI runs the inference itself, on its own infrastructure, for open and open-adjacent model families: Llama, Qwen, Mixtral, DeepSeek, MiniMax, and others. There's no upstream provider being resold here — Together is the compute.
That distinction has a real practical consequence: you can't get GPT, Claude, or Gemini through Together AI, because those are closed models Together doesn't host. If your actual need is "one key for every model family, proprietary and open alike," Together AI structurally can't be that on its own — that's the job a router does. If your need is specifically open-weight model inference with real infrastructure behind it — fine-tuning, dedicated capacity, GPU rental — Together is a direct provider built for exactly that, in a way a router (which doesn't own compute) fundamentally isn't.
Four different ways you can actually get billed
This is where a lot of confusion comes from, because "Together AI pricing" isn't one number — it's four different billing surfaces depending on which product you're using:
Serverless inference is pay-per-token, no infrastructure to manage, and the default starting point for most people. Rates vary enormously by model — roughly $0.05 to $7 per million tokens across the catalog is a reasonable current range, with most commonly used models clustering between $0.27 and $3.00. There's also at least one genuinely free community model currently on the serverless catalog, priced at $0 per million tokens — worth checking if a smaller open model fits your task before assuming you need a paid one.
Dedicated endpoints bill per GPU-hour for reserved, isolated capacity — an H100 running around $6.49/hour standard. This is the right call once traffic is sustained and predictable enough that reserved throughput beats per-token variability.
Rented GPU clusters are a separate per-GPU-hour product for teams that want raw compute rather than a managed endpoint — currently around $5.49/hour on-demand, $3.99/hour reserved (the promotional rate mentioned above, worth rechecking after it expires).
Fine-tuning bills per million training tokens — roughly $8-12/M depending on method — plus a separate ongoing hosting bill for the resulting model once trained. Teams sometimes budget the one-time training charge and get surprised by the recurring hosting line item that follows it.
None of these are hidden costs exactly — they're documented — but treating "Together AI pricing" as a single number instead of four distinct products is the fastest way to underbudget.
The discount levers actually worth using
A few mechanisms meaningfully change your real bill versus the sticker rate. The Batch API gives up to 50% off most chat models for asynchronous workloads — genuinely useful if part of your traffic doesn't need real-time responses. Cached input pricing, on select chat models, runs 5-10x cheaper than standard input when a prompt prefix is reused across calls — the same cache-discount pattern worth checking on any provider if your workload resends a large static context. Neither of these show up if you're only looking at the headline per-token rate for a model.
The catalog moves fast enough that old guides go stale
Worth flagging the same way it's worth flagging for any actively developed platform: Together's model catalog turns over quickly enough that pricing guides even a few months old sometimes reference models no longer listed. Mixtral 8x7B and the original Llama 3 70B — models a lot of older Together AI guides anchor their examples on — aren't on the current serverless catalog as of this writing. Pin model identifiers deliberately in production code, and check the live catalog before assuming a model string from a guide (including this one) still resolves to something real.
How it actually compares, briefly
Together and Fireworks are usually described as the two main "neutral" open-model hosts, pricing within a few cents of each other on most shared models — Together tends to carry a broader catalog of smaller and specialty models, Fireworks leans harder into inference speed and function-calling optimization. Groq, running on its own custom hardware, is commonly cited as 5-10x faster on the same Llama-family models when raw latency is the deciding factor, at the cost of a narrower catalog. None of these differences matter much until you've actually defined what you're optimizing for — catalog breadth, latency, or price — since each provider wins on a different axis.
A reasonable way to actually use it
The advice that held up best across everything I read while researching this: treat Together AI as a model platform to route across, not a single-model commitment. Start with a serverless shortlist, send each workload to the cheapest model that actually clears your quality bar for that task, and measure cost per accepted result rather than cost per token in isolation — a cheap model that needs three retries to get a usable answer isn't actually cheap. Only move to dedicated endpoints or fine-tuning once real traffic volume demonstrates the added commitment pays for itself; committing early on assumed volume is the more common way teams overspend here, not the sticker prices themselves.
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["TOGETHER_API_KEY"],
base_url="https://api.together.xyz/v1",
)
def try_cheap_then_escalate(prompt, cheap_model, strong_model, quality_check):
"""Route to the cheaper model first; only pay for the stronger one
when the cheap model's output actually fails your quality bar."""
cheap_result = client.chat.completions.create(
model=cheap_model,
messages=[{"role": "user", "content": prompt}],
).choices[0].message.content
if quality_check(cheap_result):
return cheap_result, cheap_model
strong_result = client.chat.completions.create(
model=strong_model,
messages=[{"role": "user", "content": prompt}],
).choices[0].message.content
return strong_result, strong_model
# quality_check is task-specific — a length floor, a keyword check,
# a cheap classifier call, whatever "did this actually work" means for your task
result, model_used = try_cheap_then_escalate(
prompt="Summarize this support ticket in one sentence: ...",
cheap_model="meta-llama/Llama-3.1-8B-Instruct",
strong_model="meta-llama/Llama-3.3-70B-Instruct",
quality_check=lambda text: len(text.strip()) > 20,
)
print(f"Used {model_used}: {result}")
That's a deliberately simple version of the "route to cheapest-that-passes" principle — a real implementation would want a more meaningful quality check than a length floor, but the shape (try cheap, escalate only on failure, measure which path actually got used) is the useful pattern regardless of how sophisticated the check gets.
Where a router still fits
If what you actually want is Together's open-model catalog and closed-frontier models like Claude or GPT behind the same key, that's exactly the gap a router fills — including gateways that list Together-hosted models alongside proprietary ones. RouteAI, for instance, provides access to a mix of open and other model families through one OpenAI-compatible key, which is a different trade-off than going directly to Together: less direct control over Together-specific features like dedicated endpoints or fine-tuning, in exchange for not needing a separate account and billing relationship just for the open-model slice of your stack.
TL;DR: Together AI is a direct inference host for open and open-adjacent models — not a routing gateway — which means it can't provide access to closed models like GPT or Claude, but does offer real infrastructure (fine-tuning, dedicated GPUs, cluster rental) a router fundamentally can't. Pricing spans four separate billing surfaces, the catalog changes fast enough to make old guides stale, and the most reliable way to use it well is routing tasks to the cheapest model that actually passes your quality bar rather than committing to one model upfront.
Website: https://www.fastrouteai.com
Top comments (0)