DEV Community

Cover image for Deploying Portkey - Open-Source AI Gateway for LLM Routing
Vultr profile image Sanskriti Harmukh
Sanskriti Harmukh for Vultr

Posted on with Aashish Chaurasiya Originally published at docs.vultr.com

Deploying Portkey - Open-Source AI Gateway for LLM Routing

#ai #llm #docker #devops

Portkey is an open-source AI gateway that unifies access to 250+ Large Language Models (LLMs) across dozens of providers behind a single, OpenAI-compatible API. Instead of integrating with each provider's SDK, applications send requests through Portkey, which handles routing, load balancing, and automatic fallback to keep AI-powered systems reliable. This guide deploys Portkey on a Linux server using Docker Compose and secures it with Traefik as a reverse proxy for TLS termination, protecting access with Basic Authentication for the web console and an IP allowlist for the API endpoint. By the end, you'll have a working AI gateway routing requests to your LLM provider over HTTPS.

Set Up the Directory Structure and Environment Variables

Portkey's gateway container is stateless and does not require persistent storage of its own. The project directory contains the Docker Compose configuration and environment variables used to configure and run the gateway.

1. Create the project directory:

$ mkdir ~/portkey
Enter fullscreen mode Exit fullscreen mode

2. Navigate to the project directory:

$ cd ~/portkey
Enter fullscreen mode Exit fullscreen mode

3. Find your server's public IP address, which is used later to restrict API access:

$ curl -4 ifconfig.me
Enter fullscreen mode Exit fullscreen mode

Copy this value for use when configuring the .env file.

4. Create an environment file to store configuration variables:

$ nano .env
Enter fullscreen mode Exit fullscreen mode

5. Add the following configuration:

DOMAIN=portkey.example.com
LETSENCRYPT_EMAIL=admin@example.com
SERVER_IP=YOUR_SERVER_IP/32
Enter fullscreen mode Exit fullscreen mode
  • Replace portkey.example.com with your domain name.
  • Replace admin@example.com with your email address.
  • Replace YOUR_SERVER_IP with your server's public IP address obtained in the earlier step.

Save and close the file.

6. Install the htpasswd utility, which is used to generate credentials for the web console:

$ sudo apt install -y apache2-utils
Enter fullscreen mode Exit fullscreen mode

7. Generate a username and password for accessing the Portkey console. Replace portkeyadmin with your preferred username:

$ htpasswd -c ~/portkey/.htpasswd portkeyadmin
Enter fullscreen mode Exit fullscreen mode

Enter and confirm a password when prompted. This creates a .htpasswd file containing the hashed credentials.

Deploy Portkey with Docker Compose

This section deploys Portkey and Traefik together, with Traefik handling HTTPS certificate issuance through Let's Encrypt. The self-hosted Portkey gateway has no built-in authentication of its own, so this guide adds Basic Authentication and an IP allowlist at the Traefik level to restrict access to the console and the API.

1. Create the Docker Compose manifest file:

$ nano docker-compose.yml
Enter fullscreen mode Exit fullscreen mode

2. Add the following configuration:

services:
  traefik:
    image: traefik:v3.7.11
    container_name: traefik
    restart: unless-stopped
    environment:
      DOCKER_API_VERSION: "1.44"
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--certificatesresolvers.le.acme.httpchallenge=true"
      - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.le.acme.email=${LETSENCRYPT_EMAIL}"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./letsencrypt:/letsencrypt
      - ./.htpasswd:/.htpasswd:ro

  portkey:
    image: portkeyai/gateway:1.15.2
    container_name: portkey
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.services.portkey.loadbalancer.server.port=8787"
      - "traefik.http.middlewares.portkey-auth.basicauth.usersfile=/.htpasswd"
      - "traefik.http.middlewares.portkey-ipallow.ipallowlist.sourcerange=${SERVER_IP}"
      - "traefik.http.routers.portkey-console.rule=Host(`${DOMAIN}`) && PathPrefix(`/public`)"
      - "traefik.http.routers.portkey-console.entrypoints=websecure"
      - "traefik.http.routers.portkey-console.tls=true"
      - "traefik.http.routers.portkey-console.tls.certresolver=le"
      - "traefik.http.routers.portkey-console.middlewares=portkey-auth"
      - "traefik.http.routers.portkey-console.service=portkey"
      - "traefik.http.routers.portkey-console.priority=10"
      - "traefik.http.routers.portkey-api.rule=Host(`${DOMAIN}`)"
      - "traefik.http.routers.portkey-api.entrypoints=websecure"
      - "traefik.http.routers.portkey-api.tls=true"
      - "traefik.http.routers.portkey-api.tls.certresolver=le"
      - "traefik.http.routers.portkey-api.middlewares=portkey-ipallow"
      - "traefik.http.routers.portkey-api.service=portkey"
      - "traefik.http.routers.portkey-api.priority=1"
      - "traefik.http.routers.portkey-logs.rule=Host(`${DOMAIN}`) && Path(`/log/stream`)"
      - "traefik.http.routers.portkey-logs.entrypoints=websecure"
      - "traefik.http.routers.portkey-logs.tls=true"
      - "traefik.http.routers.portkey-logs.tls.certresolver=le"
      - "traefik.http.routers.portkey-logs.middlewares=portkey-auth"
      - "traefik.http.routers.portkey-logs.service=portkey"
      - "traefik.http.routers.portkey-logs.priority=10"
Enter fullscreen mode Exit fullscreen mode

Save and close the file.

In the above manifest:

  • traefik: Serves as the reverse proxy and TLS termination point. It listens on ports 80 and 443, automatically redirects HTTP to HTTPS, and provisions Let's Encrypt certificates.
  • portkey: Runs the official portkeyai/gateway image on internal port 8787. Separate Traefik routers apply Basic Authentication to the /public console and an IP allowlist to the API, avoiding conflicts with provider Authorization headers. The stateless gateway requires no environment variables, volumes, or database.

3. Start the services in detached mode:

$ docker compose up -d
Enter fullscreen mode Exit fullscreen mode

4. Verify that all containers are running:

$ docker compose ps
Enter fullscreen mode Exit fullscreen mode

The output displays two running containers with Traefik listening on ports 80 and 443.

Access the Portkey Gateway Console

Traefik routes all HTTPS traffic on the configured domain to the Portkey container, applying Basic Authentication to the console and an IP allowlist to the API.

1. Confirm that the API responds from your server. Replace portkey.example.com with your configured domain:

$ curl -I https://portkey.example.com/v1/chat/completions
Enter fullscreen mode Exit fullscreen mode

A response other than 403 confirms that the request reached the gateway and was not blocked by the IP allowlist. A 403 response means the server's IP does not match the value configured in SERVER_IP in the .env file.

2. Open a web browser and navigate to https://portkey.example.com/public/, replacing portkey.example.com with your configured domain.

3. A browser prompt requests a username and password. Enter the credentials created earlier with htpasswd.

The Portkey Console displays a test request panel and options to configure routing rules such as load balancing, fallbacks, and retries.

Test the Portkey Gateway Console

Validate the deployment by sending a live chat completion request through the gateway using an LLM provider API key.

1. Send a test chat completion request through the gateway. Replace portkey.example.com with your configured domain, your_provider with your LLM provider, and your_api_key and your_model with your provider's key and model name:

$ curl https://portkey.example.com/v1/chat/completions -H "x-portkey-provider: your_provider" -H "Authorization: Bearer your_api_key" -H "Content-Type: application/json" -d '{"messages": [{"role": "user", "content": "Say this is a test."}], "model": "your_model"}'
Enter fullscreen mode Exit fullscreen mode

A successful response returns a chat.completion object containing the model's reply, confirming that the gateway routed the request to the provider and back.

2. Open the Portkey Console at https://portkey.example.com/public/, replacing portkey.example.com with your configured domain and review the request under the Logs section, confirming the gateway recorded the completed request.

Next Steps

Portkey is running behind Traefik, routing requests to your language model provider through a single, OpenAI-compatible endpoint. From here you can:

  • Configure routing rules such as load balancing, fallbacks, and automatic retries across multiple LLM providers
  • Add more LLM providers to the gateway and compare latency, cost, and reliability across them
  • Review the request logs in the console regularly to monitor usage and catch failed requests

For the full guide with additional tips, visit the original article on Vultr Docs.

Top comments (0)