Daniel Kim

Posted on Aug 11

ngrok's AI Gateway Skips the LLM Router Wars and Wins on the One Thing It Already Owned: The Tunnel

#ai #devtools #llm #networking

Every backend team that calls more than one model provider eventually writes the same file. It's usually called something like llmClient.ts or model_router.py, and it does the same five things every time: pick a provider, catch the timeout, retry against a fallback, keep the API key out of the client bundle, and log enough to answer "which app burned through our OpenAI budget this month." Nobody budgets time for this file. It gets written in an afternoon, in anger, right after the first outage.

On August 5, 2026, ngrok — the company that has spent over a decade being the fastest way to expose a local port to the internet — launched on Product Hunt with a hosted product that replaces that file. It's called ngrok AI Gateway, and it landed at #4 on Product Hunt that day. The pitch is unremarkable on its face: one URL, one key, route to any provider. That description alone puts it in a crowded field, competing against OpenRouter, LiteLLM, Portkey, Cloudflare AI Gateway, and Helicone, all of which already do "one endpoint, many providers" reasonably well.

The part worth paying attention to isn't the routing. It's what ngrok does with the model running on the machine under your desk — and the fact that this is ngrok's second attempt at solving this problem, not its first.

This isn't actually a new product

Most Product Hunt coverage will treat ngrok AI Gateway as a fresh launch. It isn't. According to ngrok's own engineering blog, the AI Gateway was first introduced in December 2025, built on top of ngrok's existing platform primitives: endpoints, traffic policy, the whole ingress stack that ngrok already sold. Early users liked the idea — centralized routing across hosted and self-hosted models — but getting there meant learning parts of the ngrok platform that had nothing to do with making an inference call. Endpoints and traffic policy are powerful concepts if you're building a tunnel. They're friction if you just want to swap a baseURL.

So in July 2026, ngrok rebuilt the front door. app.ngrok.ai shipped as a dedicated dashboard, API, and gateway URL, purpose-built for people who never plan to touch the rest of the ngrok product. The August 5 Product Hunt launch is that rebuild's public debut, not a v1. That matters for how you should read the reviews and the "what's missing" complaints below: this is a product with eight months of real usage behind a UX that is, by the company's own account, only a few weeks old.

It also tells you something about ngrok's motivation that a straight "new AI gateway, who dis" narrative wouldn't. This isn't a startup chasing the LLM infrastructure gold rush with a greenfield idea. It's an infrastructure company noticing that its core competency — making things reachable over a network without opening a port — happens to be exactly the unsolved half of the LLM gateway problem.

What it actually does

Strip away the AI framing and ngrok AI Gateway is a reverse proxy with credential management, routing, and observability bolted on, which is precisely what every gateway in this category is. You point your existing OpenAI, Anthropic, or Vercel AI SDK at https://gateway.ngrok.ai instead of the provider's own endpoint, swap the API key for an ngrok access key, and the gateway takes it from there:

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.ngrok.ai",
  apiKey: process.env.NGROK_AI_ACCESS_KEY,
});

const completion = await client.chat.completions.create({
  // Route to a self-hosted model first
  model: "unsloth/Qwen3.5-1228-A10B-GGUF",
  // Fallback to public providers
  models: ["gpt-5.4", "claude-opus-4-6"],
  messages: [
    { role: "system", content: "Talk like a pirate." },
    { role: "user", content: "Are semicolons optional in JavaScript?" },
  ],
  stream: true,
});

That's the entire integration. No new client library, no wrapper SDK, no rewritten call sites — which is worth noting because it's also true of most competitors. The differentiation shows up in four places: how credentials are split, how local models get reached, how failover is expressed, and how the whole thing is priced.

Credentials never touch the client

Your application authenticates to the gateway with an ngrok access key (format ng-xxxxx-g1-xxxxx), and the actual upstream provider key — your OpenAI or Anthropic secret — stays server-side, entirely invisible to the calling application. You can mint a separate access key per app or per developer and revoke one without touching the others. Keys display exactly once at creation; lose it and you delete-and-recreate rather than retrieve it. It's a small design choice, but it directly targets a specific, recurring failure mode: provider keys leaking into mobile app bundles or client-side JS because someone wired the SDK up before thinking about where the secret lives.

Reaching a model that was never meant to be public

This is the part that doesn't show up in a feature-comparison table, and it's the actual argument for why ngrok is a credible entrant here rather than a me-too. Every other gateway in this space treats "route to a self-hosted model" as: you already exposed it somehow, now point us at the URL. LiteLLM, Portkey, and the rest assume the network problem is solved before the gateway gets involved.

ngrok's entire original product is the network problem. With ngrok AI Gateway, you register a local model — Ollama or vLLM running on a colo box, a Mac mini, or, per ngrok's own marketing copy, "the gaming GPUs you stopped gaming on" — and it becomes reachable through the gateway over ngrok's private connectivity, without opening an inbound port or exposing a public IP. For any team that has tried to put a local model into a real request path, this is the step that normally turns into its own mini-project: a reverse SSH tunnel, a Tailscale exit node, or a public endpoint you then have to lock down with an API gateway you build yourself. ngrok collapses that into "same door as everything else."

Failover and retries as infrastructure, not application code

If a configured provider or key fails, the gateway tries the next one in the list automatically. Failed requests retry without your code handling the error. This is the un-glamorous part of the "model_router.py" problem — the part every team eventually implements badly, with an ad hoc retry loop that doesn't account for rate limits correctly, or a fallback list that's hardcoded and never updated. Putting it in the routing layer, where timeouts and backoff actually belong, is the correct architectural call, and it's one every serious competitor also makes — this isn't a differentiator so much as table stakes done properly.

No provider accounts required to start

You can call OpenAI and Anthropic through the gateway without holding accounts with either. ngrok credits cover the gateway's processing fee and the upstream model cost as a single bundled charge. For a prototype, or a team that doesn't want three vendor relationships and three separate invoices before shipping anything, that's real friction removed — with a caveat covered below.

Four concrete situations this actually solves

Abstractions like "unified gateway" are easy to nod along to and hard to picture. Here's what the four headline features above look like as actual production scenarios:

A mobile app that can't hold a provider key. Today, most teams solve "the OpenAI key can't live in the app bundle" by standing up their own thin proxy — a Cloudflare Worker or a Lambda that does nothing but forward requests and inject a secret. That proxy has no failover, no per-user rate limiting, and becomes its own on-call surface the day it goes down. Swapping it for the gateway's access-key model removes a service you'd otherwise own and staff.

A cost-sensitive pipeline that wants to try local inference without betting the farm. A team running document classification or embeddings at volume might want to shift 80% of that traffic to a self-hosted Llama or Qwen model on owned hardware, while keeping a frontier model as the fallback for anything the local model flags as low-confidence. Today that split lives in application code, with a manually maintained "if local model fails, call OpenAI" branch. Expressed as a models fallback array against one endpoint, it's a config change instead of a code change — and the local half of that split is the part every other gateway makes you solve yourself first.

Compliance-adjacent teams that need to know who spent what. Scoped access keys per application or per developer mean a security review can answer "which surface could have called Anthropic with a compromised credential" without grepping application code for embedded secrets. It's a coarser answer than Portkey's role-based access control, but it's a real one, and it's available on day one rather than an enterprise add-on.

Prototypes that shouldn't require three vendor sign-ups before a demo. The no-provider-accounts-required path exists specifically for this: a hackathon team or a solo builder who wants to compare GPT-5.4 and Claude Opus 4.6 output on the same prompt without creating two billing relationships first. This is the same value proposition OpenRouter built a business on — ngrok is entering that specific wedge too, just bundled alongside the local-model story rather than as the whole pitch.

The pricing model is a genuine bet against the market

Here's where ngrok AI Gateway diverges most sharply from the field, and it's a pricing bet, not a features one. The gateway charges a flat $0.05 per million tokens (combined input and output) on every successful request when you bring your own provider keys. That's it. No percentage of spend.

Compare that to how the rest of the category prices itself:

OpenRouter passes through provider pricing at 0% markup, then charges a 5.5% platform fee on pay-as-you-go credit purchases (5% if you bring your own key, with the first million monthly requests waived), minimum $0.80 per purchase.
Cloudflare AI Gateway is free for its core routing, caching, and analytics, but applies a 5% fee if you route spend through its Unified Billing.
Portkey and Helicone don't touch your token spend at all — they bill on log volume instead, starting around $49/month and $79/month respectively once you outgrow the free tier's short retention window.
LiteLLM is MIT-licensed and free to self-host, but "free" means you're now running Postgres, Redis, and a proxy fleet yourself — commonly $300–$1,500/month in infrastructure plus whoever's on call when it falls over.

A flat per-token fee is a genuinely different shape of cost curve than a percentage-of-spend fee. At $6,000/month in LLM spend, a 5.5% platform fee is already $330/month; ngrok's $0.05/million-token charge stays flat relative to token volume, not dollar volume, and gets proportionally cheaper as your average request grows more expensive (longer context, pricier frontier models) since the fee doesn't scale with the price of the model, only its token count. If you're running high volumes of large-context or reasoning-heavy calls through a frontier model, ngrok's fee is close to a rounding error next to Portkey's or Cloudflare's percentage-based math. If you're doing lots of small, cheap completions, a log-based or self-hosted approach may still win.

Underneath the per-token fee sits ngrok's ordinary subscription: Free ($0, includes $5 one-time usage, 3 endpoints, 20,000 requests/month), Hobbyist (~$8–10/month), Pay-as-you-go ($20/month, unlimited endpoints), and custom Enterprise with SSO/SAML, SCIM, and SLAs. If you already pay ngrok for tunneling or ingress, the AI Gateway rides on billing and team management you've already set up — a real, if narrow, advantage for existing ngrok customers.

The one pricing detail that's genuinely opaque

There's a catch worth flagging because it's easy to miss on the pricing page. When you bring your own provider keys, the math is transparent: ngrok takes its flat $0.05/million-token cut, and your provider bills you directly and separately for the model. But when you use ngrok's built-in inference — the no-provider-accounts-required path — credits cover the processing fee and the model cost as one bundled rate, and the documentation doesn't itemize what portion is which. That's a fine tradeoff for a prototype. It's a bad one for a team trying to build a real cost model, because you can't audit what you're actually paying for tokens versus routing. Treat the built-in inference path as a fast start, not a destination, and plan the migration to your own keys before you need the itemized bill, not after.

What the launch page doesn't tell you

Independent review coverage from launch day — and ngrok's own documentation — surfaces three gaps that are worth knowing before you route production traffic through this:

No semantic caching. Several competing gateways, including LiteLLM (via Redis) and Portkey (hosted), will return a stored response for a repeated or near-identical prompt, cutting both latency and spend on workloads with repetitive queries. ngrok AI Gateway doesn't do this yet. If your traffic has that shape — support bots answering the same handful of questions, RAG pipelines re-querying similar context — the savings you're leaving on the table can dwarf the $0.05/million-token routing fee entirely.

No per-key rate limiting. You can scope an access key to specific providers and models, but you can't yet cap how much a given key can spend before it needs a manual credit top-up. The first time a retry loop or a misbehaving client ships to production and starts hammering the gateway, this is the control you'll wish you had.

Thin observability at launch. The documentation covers authentication, credits, routing, and failover well. It does not yet cover, in detail, per-model latency breakdowns, prompt/response logging with redaction controls, or cost attribution by application or team as a built-in report — you can infer per-app cost from per-app access keys, but you're reconstructing it yourself rather than reading it off a dashboard. For a category where "observability" is often the actual reason teams adopt a gateway in the first place, this is the most consequential gap.

None of these are unusual for a UX that's a few weeks old sitting on top of an eight-month-old product. But if your reason for wanting a gateway is governance and cost control rather than routing and credential hygiene, check the current state of the docs before assuming these arrived by the time you read this.

Where it sits against the rest of the field

The LLM gateway category has sorted itself into fairly clean positions, and it's worth being precise about where ngrok actually competes rather than treating every entrant as interchangeable:

OpenRouter wins on breadth — 400+ models, zero infrastructure, hosted-only. Its 5.5% platform fee is the cost of not thinking about infrastructure at all.
LiteLLM wins on control — MIT-licensed, self-hosted, no percentage fee ever, at the cost of running Postgres, Redis, and the proxy yourself.
Portkey wins on enterprise governance — 50+ built-in guardrails, prompt versioning, evaluation tooling, SOC 2/HIPAA on paid tiers, billed on log volume rather than spend.
Cloudflare AI Gateway wins on "you're already there" — free core routing, caching, and analytics if you're on Workers already.
Helicone pairs an Apache 2.0 Rust gateway with hosted observability, positioned similarly to Portkey but leaner.

None of them make reaching a locally-hosted model a first-class, zero-config path — because none of them are, at their core, networking companies. That's the lane ngrok is actually running in, and it's a narrower, more defensible one than "yet another LLM router." If your workload genuinely never touches a model you don't control the network path to, ngrok's advantage evaporates and you're just comparing flat-fee pricing against percentage fees and log-based subscriptions, which is a much closer contest — probably a win for LiteLLM if you have the operational appetite, or Cloudflare if you're already on that edge.

Where ngrok pulls ahead is the hybrid case that's becoming increasingly common: bulk, low-stakes work routed to a self-hosted model on hardware you already own, with the hard cases falling back to a frontier model through the same endpoint, the same access-key scoping, and the same failover logic. That's a specific, growing shape of workload — cost-conscious teams offloading what they can to local inference while keeping GPT-5.4 or Claude Opus 4.6 as the safety net — and right now it's a shape none of the five competitors above serve as directly.

The catch nobody's asking about yet

The Coding4Food writeup of the launch flagged the two questions worth taking seriously before adopting this in production: what's the latency penalty of adding another network hop, and what happens to your production system if ngrok itself has downtime. Every hosted gateway carries this risk — OpenRouter and Portkey both publish status pages for exactly this reason — but it's sharper here because ngrok's core business model has, for over a decade, been "be the thing standing between your service and the internet." That's a lot of trust to extend to a product that's a few weeks past its UX rebuild, on a company whose reliability track record is otherwise excellent but whose AI Gateway specifically doesn't have years of production incidents (or the absence of them) to point to yet. Benchmark the added latency against your actual request path before committing, and don't take "it's ngrok, they're good at networking" as a substitute for checking your own numbers.

There's also a lock-in question that's easy to underweight: unlike LiteLLM and Portkey, ngrok AI Gateway has no self-hosted option. If you need data sovereignty — prompts and responses that provably never leave infrastructure you control — this product is disqualified by design, full stop, regardless of how good the pricing or the local-model story is.

Who should actually adopt this

Try it now if you're already calling two or more model providers from a backend and have started writing your own routing/retry/key-management glue — that code is a liability, and this replaces it cleanly, especially if you're already an ngrok ingress customer with billing and team management already set up. It's also a strong fit if you're specifically trying to route bulk work to hardware you own without exposing it publicly; nobody else makes that as painless.

Wait if cost control and governance are your actual motivation for wanting a gateway — the per-key rate limiting and detailed cost-attribution dashboards aren't there yet, and you'll be reconstructing that visibility yourself in the meantime.

Ignore it if you call exactly one provider (a gateway in front of a single upstream is a hop, a dependency, and a fee for very little), if semantic caching would meaningfully cut your bill (it isn't built yet), or if self-hosting is a hard requirement rather than a preference — go to LiteLLM or Portkey's open-source gateway instead.

The most interesting fact about this launch isn't the Product Hunt ranking. It's that a networking company looked at the increasingly crowded LLM gateway market and, instead of trying to out-feature Portkey or out-price OpenRouter, shipped the one integration nobody else could ship as naturally: making a model on a machine you own behave, from your application's point of view, exactly like a model OpenAI is hosting. Whether that's enough to matter against five entrenched competitors depends entirely on how many teams actually have a GPU worth routing to — and that number is growing faster than the gateway market is used to.

If you're running any inference locally right now — Ollama, vLLM, a fine-tuned model on your own hardware — how are you currently getting it into a real request path safely, and would you trade that setup for a hosted gateway you don't control?

Sources:

DEV Community