DEV Community: Aaishika S Bhattacharya

What are AI gateways, and do you even need them?

Aaishika S Bhattacharya — Fri, 31 Oct 2025 17:37:29 +0000

The AI ecosystem is evolving so quickly that even those of us building in it are still figuring out where we fit. Between the scramble to deploy ✨something AI✨ and the chaos of managing multiple model APIs and keys, a new player called "AI gateway" has entered the chat.

If the term makes you think of API gateways, you're not wrong, except this one comes with brains and boundaries. It doesn't just route traffic; it watches what goes in, what comes out, and keeps it all in check. Let's take a deeper look.

What is an AI gateway?

An AI gateway is essentially a control tower for your AI models, a middleware layer that sits between your applications and the AI services they rely on. Every prompt you send and every model you use, from OpenAI to Anthropic to your in-house Ollama setup, must pass by this control tower.

In many ways, AI gateways play a role similar to what ngrok does for production API workloads. ngrok creates a secure tunnel between your upstream services and the public internet, giving you a controlled and observable interface between any environment and the public internet.

AI gateways do the same, but for model interactions. They act as the secure bridge between your internal systems and the unpredictable and probabilistic landscape of external AI APIs, enforcing governance, logging, and routing rules every step of the way.

AI gateways secure and manage all AI interactions under one roof so your engineers aren't busy juggling or rotating a dozen API keys, your legal team isn't scratching their heads about PII leaks, and you can rest assured that the CTO wouldn't come knocking, enquiring about the extra $$ that showed up on the bill.

In a nutshell, if ngrok is the gateway to your web traffic, an AI gateway is the gateway to your LLM traffic.

But why are AI gateways suddenly everywhere?

The AI gold rush has led to a very modern problem: too many shovels (models), too little gold (control).

Organizations, no matter their scale, are juggling OpenAI, Anthropic, and open-source models at once, each with its own rate limits, authentication quirks, and billing nightmares. API keys expire, get revoked, or worse—leaked. Rate limits spike without warning, and failovers become a weekend project no one signed up for. Suddenly, what started as a simple prototype now requires a mini-orchestra of secrets management, retry logic, and routing scripts just to keep the lights on.

This is where AI gateways quietly slipped in, promising reliability, observability, and most importantly, sanity. They automate the thankless parts of scaling AI: rotating keys before they break production, failing over to a backup model when one provider struggles, and even dynamically switching models based on latency, cost, or accuracy.

If you've ever used ngrok's Endpoint Pools, the idea will feel familiar. A pool of secure, intelligent endpoints sitting behind a single entry point, distributing requests for reliability and performance. The only difference is that in this world, the "endpoints" aren't origin servers and upstream services, but LLMs.

How do they actually work?

At a high level, an AI gateway sits right between your app and the AI models you call. Every request passes through this layer before reaching the model. The gateway then handles tasks like deciding which model to send it to, checking for security leaks and discrepancies, logging requests and responses, among others, depending on the architecture of your app.

For example, here's your app without a gateway:

And, here’s your app with one:

If you are not inclined to subscribe to multiple third-party services or build complex features, like every 10x engineer, you too would prefer an AI gateway—Simply because it’s actually more cost- and resource-effective to buy than build all the features above.

Do you, a developer, actually need an AI gateway?

If you're an enterprise running dozens of AI workloads across teams and vendors; yes, you need an AI gateway yesterday.

If you're a startup running a few calls to GPT-5 for your chatbot: nope.

So the question actually isn't whether an AI gateway is good, but whether your complexity warrants it. Think of it like Kubernetes---absolutely fantastic for orchestration, but overkill for a personal blog.

To be very honest, for many developers, the gateway is another layer that needs to be set up. If your app talks to a single model and doesn't need elaborate governance or cost-tracking, your SDK already has you covered. But if your use case involves:

Switching between providers dynamically and gracefully
Auditing, anonymizing or filtering user prompts
Controlling cost and usage automatically

Then an AI gateway can be a lifesaver and not a luxury. Back to the previous section: If you're dealing with multiples of LLMs and/or keys, you probably would benefit from AI gateways.

The future of AI gateways

AI gateways are the foundation of AI infrastructure maturity. In a few years, they might evolve into AI mesh networks, balancing workloads between providers the way CDNs do for content today or how ESP8266-based devices communicate amongst themselves (makes my inner IoT enthusiast happy).

My take on the current AI Landscape, based on the modern AI infrastructure layers, is divided into four distinct layers, and here’s where some of my favorite AI tools fit:

Be on the lookout for our involvement with them wink wink.

Be part of what's next

Speaking of the future: ngrok.ai is here and you can sign up and request early access right now. We are building the next generation of networking infrastructure rather fast, so watch this place and our social spaces [X (formerly Twitter), LinkedIn, Bluesky, YouTube] for more AI-centric announcements from your favorite networking platform!

The Ultimate Guide to ngrok

Aaishika S Bhattacharya — Wed, 22 Oct 2025 00:00:00 +0000

Getting started with ngrok is almost suspiciously easy. Which is why the question we hear most often isn’t "where do I start?" but rather “what else can I do with ngrok?” For over a decade, we’ve been serving millions of developers and, through them, millions of users.

Sure, our documentation has all the answers, every CLI flag, every Traffic Policy configuration neatly laid out, but what about the developers who are always on the lookout for a TL;DR? What if you too are not looking for a full solution or a specific use case for your next project, but just want to know… what else you can do with ngrok?

Presenting ngrok's new cheatsheet that walks you through some of our most interesting offerings, designed to scratch that itch of not just serving, but also securing endpoints in as little as two steps. Read on, or download the PDF format, or a crisp two-pager printable.

Installation

Sign up for a free account: https://ngrok.com/signup
Keep your authtoken handy: https://dashboard.ngrok.com/get-started/your-authtoken

macOS

# Install via Homebrew
brew install ngrok

# Add your authtoken
ngrok config add-authtoken <token>

# Start an endpoint
ngrok http 80

Linux

# Install via Apt
curl -sSL https://ngrok-agent.s3.amazonaws.com/ngrok.asc \
  | sudo tee /etc/apt/trusted.gpg.d/ngrok.asc >/dev/null \
  && echo "deb https://ngrok-agent.s3.amazonaws.com bookworm main" \
  | sudo tee /etc/apt/sources.list.d/ngrok.list \
  && sudo apt update \
  && sudo apt install ngrok
# OR
# Install via Snap
snap install ngrok

# Add your authtoken
ngrok config add-authtoken <token>

# Start an endpoint
ngrok http 80

Windows

# Install via WinGet
winget install ngrok -s msstore
# OR
# Install via Scoop
scoop install ngrok

# Add your authtoken
ngrok config add-authtoken <token>

# Start an endpoint
ngrok http 80

Kubernetes

# Add ngrok Kubernetes Operator to Helm
helm repo add ngrok https://charts.ngrok.com

# Add ngrok API key and authtoken
export NGROK_AUTHTOKEN=YOUR_NGROK_AUTHTOKEN
export NGROK_API_KEY=YOUR_NGROK_API_KEY

helm install ngrok-operator ngrok/ngrok-operator \
  --namespace ngrok-operator \
  --create-namespace \
  --set credentials.apiKey=$NGROK_API_KEY \
  --set credentials.authtoken=$NGROK_AUTHTOKEN

Docker

# Install via Docker
docker pull ngrok/ngrok

# Run ngrok via Docker
docker run --net=host -it -e NGROK_AUTHTOKEN=xyz ngrok/ngrok:latest http 80

SDKs

# Node.js: https://ngrok.com/downloads/node-js
npm install @ngrok/ngrok

# Go: https://ngrok.com/downloads/go
go get golang.ngrok.com/ngrok/v2

# Python: https://ngrok.com/downloads/python
python3 -m pip install ngrok

# Rust: https://ngrok.com/downloads/rust
# Install ngrok-rust package and the required dependencies
cargo add ngrok -F axum && cargo add axum && cargo add tokio -F rt-multi-thread -F macros

Expose different kinds of servers

API service

# Example: API service on localhost:8080
ngrok http 8080

Web app

# Example: On localhost:3000
ngrok http 3000

SSH server

# Example: On Port 22
ngrok tcp 22

Postgres server

ngrok tcp 5432

Any service or server on a different machine

ngrok http http://192.168.1.50:8080

Troubleshoot

ngrok diagnose

# To test IPv6 connectivity
ngrok diagnose --ipv6 true

# To test connectivity between the ngrok agent and all ngrok points of presence
ngrok diagnose --region all

# For a verbose report
ngrok diagnose -w out.txt #OR
ngrok diagnose --write-report out.txt

Add Authentication with Traffic Policy

Create a Traffic Policy file `policy.yaml`

nano policy.yaml

Add the OAuth Action with Google

List of Providers: https://ngrok.com/docs/traffic-policy/actions/oauth/#supported-providers

# policy.yaml
on_http_request:
  - actions:
      - type: oauth
        config:
          provider: google # OAuth available with Amazon, Facebook, GitHub, GitLab, Google, LinkedIn, Microsoft, Twitch

Run your endpoint with the Traffic Policy file

ngrok http 8080 --traffic-policy-file=policy.yaml

Restrict OAuth to specific emails

# policy.yaml
on_http_request:
  - actions:
      - type: oauth
        config:
          provider: google
  - expressions:
      - "!(actions.ngrok.oauth.identity.email in ['alice@example.com','bob@example.com'])"
    actions:
      - type: deny

Restrict OAuth to specific domains

# policy.yaml
on_http_request:
  - actions:
      - type: oauth
        config:
          provider: google
  - expressions:
      - "!(actions.ngrok.oauth.identity.email.endsWith('@example.com'))"
    actions:
      - type: deny

Verify your webhooks

Add the `verify-webhook` action for Slack

# policy.yaml
on_http_request:
  - actions:
      - type: verify-webhook
        config:
          provider: slack
          secret: $SLACK_TOKEN

CLI Alternative

ngrok http 3000 \
  --verify-webhook=slack \
  --verify-webhook-secret=$SLACK_TOKEN

Replace `provider` for any supported provider

List of Supported Providers: https://ngrok.com/docs/traffic-policy/actions/verify-webhook/

# policy.yaml
on_http_request:
  - actions:
      - type: verify-webhook
        config:
          provider: $PROVIDER # Example: GitHub
          secret: $PROVIDER_TOKEN

Do even more with internal endpoints

Create a Cloud Endpoint

# Create a private, internal agent endpoint only reachable via forward-internal
ngrok http 8080 --binding=internal --url https://api.internal

Example Traffic Policy File

# policy.yaml
# Forward to an internal endpoint from a public endpoint
on_http_request:
  - actions:
      - type: forward-internal
        config:
          url: https://api.internal

Start Public Endpoint with `forward-internal` action

ngrok http 8080 --url forward-internal-example.ngrok.app --traffic-policy-file policy.yml

Manage traffic in other ways

Add path-based routing

# policy.yaml
# Route /api/* to api.internal, /app/* to app.internal
on_http_request:
  - expressions:
      - "req.path.startsWith('/api/')"
    actions:
      - type: forward-internal
        config:
          url: https://api.internal
  - expressions:
      - "req.path.startsWith('/app/')"
    actions:
      - type: forward-internal
        config:
          url: https://app.internal

Route traffic by anything

# policy.yaml
# Host-based and header-based dynamic forwarding to internal endpoints via forward-internal action
on_http_request:
  - expressions:
      - "req.host == 'api.example.com'"
    actions:
      - type: forward-internal
        config: { url: https://api.internal }
  - expressions:
      - "getReqHeader('X-Tenant') != ''"
    actions:
      - type: forward-internal
        config: { url: https://tenant.internal }

Multiplex to Internal Services from a Single Domain

# policy.yaml
on_http_request:
  - actions:
      - type: forward-internal
        config:
          url: https://${req.host.split(".$NGROK_DOMAIN")[0]}.internal

Add rate limiting

# policy.yaml
# 10 requests per 60s window per client IP -> 429 on limit
on_http_request:
  - actions:
      - type: rate-limit
        config:
          name: per-ip-60s
          algorithm: sliding_window
          capacity: 10
          rate: "60s"
          bucket_key:
            - conn.client_ip

Block search and AI bots

# policy.yaml
# Send a robots.txt denying crawlers
on_http_request:
  - expressions:
      - "req.path == '/robots.txt'"
    actions:
      - type: custom-response
        config:
          status_code: 200
          headers:
            Content-Type: "text/plain"
          body: |
            User-agent: *
            Disallow: /

# policy.yaml
# Deny common bot/AI user agents
on_http_request:
  - expressions:
      - "req.user_agent.raw.matches('(?i)(gptbot|chatgpt-user|ccbot|bingbot|googlebot)')"
    actions:
      - type: deny

# policy.yaml
# Also add X-Robots-Tag to all responses
on_http_response:
  - actions:
      - type: add-headers
        config:
          headers:
            X-Robots-Tag: "noindex, nofollow, noai, noimageai"

Add headers

Via CLI

# Common security headers
ngrok http 8080 \
  --request-header-add "X-Frame-Options: DENY" \
  --response-header-add "Referrer-Policy: no-referrer"

Via Traffic Policy

# policy.yaml
# Add headers on request/response
on_http_request:
  - actions:
      - type: add-headers
        config:
          headers:
            X-Frame-Options: "DENY"
on_http_response:
  - actions:
      - type: add-headers
        config:
          headers:
            Referrer-Policy: "no-referrer"

Restrict access by IPs

# Allow only 203.0.113.0/24; deny others
ngrok http 8080 --cidr-allow 203.0.113.0/24

# Or explicitly deny CIDRs
ngrok http 8080 --cidr-deny 0.0.0.0/0

Block all the potentially bad things

# policy.yaml
# Apply OWASP Core Rule Set on requests/responses
on_http_request:
  - actions:
      - type: owasp-crs-request
on_http_response:
  - actions:
      - type: owasp-crs-response

CLI Flags

`url`

# Choose a URL instead of random assignment
ngrok http 8080 --url https://baz.ngrok.dev

`traffic-policy-file`

# Manipulate traffic to your endpoint with a traffic policy file
ngrok http 8080 --url https://baz.ngrok.dev --traffic-policy-file policy.yaml

`traffic-policy-url`

# Manipulate traffic to your endpoint with a traffic policy URL
ngrok http 8080 --url https://baz.ngrok.dev policy --traffic-policy-url https://example.com/policy.yml

`pooling-enabled`

# Load Balance (different ports)
ngrok http 8080 --url https://api.example.com --pooling-enabled
ngrok http 8081 --url https://api.example.com --pooling-enabled

What else can I do with ngrok?

Use ngrok Kubernetes Operator: https://ngrok.com/docs/k8s/
Get started with Traffic Observability: https://ngrok.com/docs/obs/
Look into Identity and Access Management: https://ngrok.com/docs/iam/
Read ngrok's Security Best Practices: https://ngrok.com/docs/guides/security-dev-productivity/
Check out Gateway Examples Gallery: https://ngrok.com/docs/universal-gateway/examples/

Ending Notes

Get started with ngrok absolutely free of charge, sign up today!