DEV Community

Cover image for LiteLLM: Getting Started with LLM Proxies
Sebastian
Sebastian

Posted on

LiteLLM: Getting Started with LLM Proxies

#ai #llm #litellm

When running multiple applications that interact with LLM provider endpoints, management of keys, token consumption and cost spending, operational aspects like failover and semantic routing, or observability topics like availability, latency, and traces become important. Especially customer-facing apps that expose chat functionality need tight guards to prevent misuse and runaway costs. These aspects are the domain of LLM proxies, dedicated projects that sit between applications and LLM endpoints.

LiteLLM is a sophisticated LLM proxy, available as open-source software for self-hosting, as well as production-grade deployments on infrastructure or even fully managed cloud solutions. This article starts a new blog series and explores the rich feature set of LiteLLM. This first article discusses features, deployment options, and a step-by-step manual for local setup as Docker containers.

The technical context of this article is litellm v1.94.1, published on 2026-07-31. Setup and configuration examples should work with newer versions too.

While I am fascinated by the capabilities of artificial intelligence tools and applications, crafting blog articles remains a personal skill. Every character, number, and symbol in this article was typed manually, with the exception of verbatim copies from log messages and screenshots.

This article originally appeared at my blog admantium.com.

LiteLLM Feature Set at a Glance

The LiteLLM proxy provides the following feature set, grouped into different stages of handling AI tools.

  • API key management: Provide virtual API keys to persons, teams, or applications, and set fixed token or cost budgets.
  • Cost tracking: Actively monitor token consumption and accumulated costs.
  • Guardrails: Filter both incoming and generated messages for inappropriate topics or language.
  • Fallback and retry: Detect upstream provider outages and switch to other available models to keep systems operational.
  • Caching: Add semantic caches to frequent requests, saving both costs and round-trip time for a better user experience.
  • OpenTelemetry traces: Requests and responses are separated into GenAI OTEL spans, allowing precise control over how LLM usage, e.g. by an agent app, is internally structured into thinking loops, tool calls, RAG searches, and answer synthesis.
  • Observability: With traces, also identify latency within the LLM call stack, e.g. by figuring out slow tool calls or a massive context window due to bloated RAG results.
  • Enterprise features: Provide company-wide single sign-on logic, audit logs, and multi-tenant governance by setting up multiple organizations and teams for managed keys. IP restriction and deeper multi-tenant governance are positioned as enterprise features.

Installation Options

LiteLLM is geared towards enterprise usage, and as such, multiple installation options in terms of cloud environments and multiple components for load balancing requests and long-term storage of operational data are supported.

The essential components in each stack are these:

  • LiteLLM microservices: To enhance scalability and improve redundancy, functional aspects of LiteLLM can be broken down into different services
    • Gateway: Central component to which applications connect
    • Backend: Process and storage engine; provides a REST API and frontend APIs for management
    • Frontend: The management GUI of LiteLLM
  • LiteLLM proxy: A single service that combines the functions of the gateway, backend, and configuration GUI
  • PostgreSQL: Stores keys, teams, users, and configuration items

When hosted in cloud environments, these specific components are added:

  • Load balancer
    • AWS: Application Load Balancer (ALB) or Network Load Balancer (NLB)
    • GCP: GKE Ingress (managed Kubernetes)
    • Azure: Application Load Balancer (ALB)
  • Redis: Saves router state and provides global caching

The deployment mechanisms are:

  • Helm chart: Deploy to managed Kubernetes clusters in AWS (Elastic Kubernetes Service), GCP (Google Kubernetes Engine), or Azure (Azure Kubernetes Service)
  • Terraform: Deploy to AWS and GCP by using dedicated Terraform providers
  • Docker: For on-premises and local installations

For more information about cloud-based production deployments, see the official LiteLLM Deployment Guidelines.

Local Setup

LiteLLM can be installed natively, via the LiteLLM CLI that comes packaged with an SDK, or as a Docker container. For local setup, the last option is preferable: The complete application stack, all dependencies, and binaries are versioned inside the container, and essential configurations will be mounted via volumes. You only keep relevant stateful information and can provide safe updates via version pinning.

The LiteLLM Quickstart Guidelines outline all steps in detail. Here is a step-by-step manual, which provide the most recent application versions, and the start for using configuration files effectivly.

  • Docker Compose file
    • Download the official Docker Compose file from the LiteLLM GitHub repository
    • Open the file and change the image tags with the following up-to-date version pinning: 
  image: litellm/litellm:v1.94.1
  image: postgres:18.4-trixie
  image: prom/prometheus:v3.13.1
Enter fullscreen mode Exit fullscreen mode
  • LiteLLM environment variables
    • Create a .env file
    • Generate two random passwords for LiteLLM by running LC_ALL=C tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 20; echo
    • Open the .env file and add these passwords: 
  LITELLM_MASTER_KEY="sk-REDACTED"
  LITELLM_SALT_KEY="REDACTED"
Enter fullscreen mode Exit fullscreen mode
  • Configuration files
    • Create the local directory config/
    • Create config/litellm.yml and paste and edit the following content: 
  general_settings:
    master_key: sk-REDACTED # same key as above
    database_url: "postgresql://llmproxy:dbpassword9090@db:5432/litellm"
Enter fullscreen mode Exit fullscreen mode
  • Configuration files
    • Create config/prometheus.yml and add these lines: 
  global:
    scrape_interval: 15s
    evaluation_interval: 15s


  scrape_configs:
    - job_name: "litellm"
      static_configs:
        - targets: ["litellm:4000"]
Enter fullscreen mode Exit fullscreen mode
  • Docker Compose file changes
    • Edit the container volume mounts to reflect the configuration file locations: 
  litellm:
    volumes:
      - ./config/litellm.yml:/app/litellm-config.yml
    command:
      - "--config=/app/litellm-config.yml"


  # ... 


  prometheus:
    volumes:
      - prometheus_data:/prometheus
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml
Enter fullscreen mode Exit fullscreen mode
  • Edit the Postgres volume target path, required in postgres:18.4: 
  db:
    volumes:
      - postgres_litellm:/var/lib/postgresql
Enter fullscreen mode Exit fullscreen mode
  • Start the Docker Compose stack
    • Run docker compose up, and be patient while all the images are completely loaded: 
  > docker compose up


  # log messages
  [+] Running 15/19
  ✔ prometheus Pulled                                                       1.9s
  ✔ litellm Pulled                                                          2.5s
  ✔ db Pulled                                                               4.2s
  ⠦ open-webui 15 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀⡀⣿⣿⡀] 35.33MB/1.315GB Pulling          8.6s
  ...
Enter fullscreen mode Exit fullscreen mode
  • During startup, the litellm container needs to set up its database, which can also take some time to complete 
  litellm-1     | 2026-07-31 16:54:12,717 - litellm_proxy_extras - INFO - Running prisma migrate deploy
  litellm-1     | 2026-07-31 16:54:12,718 - litellm_proxy_extras - INFO - Using custom Prisma CLI at /opt/prisma/binaries/node_modules/.bin/prisma
  litellm_db    | 2026-07-31 16:54:16.529 UTC  LOG:  could not receive data from client: Connection reset by peer
  litellm-1     | 2026-07-31 16:54:16,645 - litellm_proxy_extras - INFO - prisma migrate deploy stdout: Prisma schema loaded from schema.prisma
  litellm-1     | Datasource "client": PostgreSQL database "litellm", schema "public" at "db:5432"
  litellm-1     |
  litellm-1     | 140 migrations found in prisma/migrations
  litellm-1     |
  litellm-1     | Applying migration `20250326162113_baseline`
  litellm-1     | Applying migration `20250326171002_add_daily_user_table`
  litellm-1     | Applying migration `20250327180120_add_api_requests_to_daily_user_table`


  ...


  litellm-1     | 2026-07-31 16:54:21,890 - litellm_proxy_extras - INFO - ✅ Migration diff applied successfully
  litellm-1     | 2026-07-31 16:54:21,891 - litellm_proxy_extras - INFO - ✅ Post-migration sanity check completed
  litellm-1     | INFO:     Started server process [1]
  litellm-1     | INFO:     Waiting for application startup.
  litellm-1     | LiteLLM Proxy: Using default (v1) migration resolver. If your deployment has seen schema thrashing during rolling deploys, try --use_v2_migration_resolver (safer: avoids the diff-and-force recovery that caused the thrash).
  litellm-1     |
  litellm-1     |    ██╗     ██╗████████╗███████╗██╗     ██╗     ███╗   ███╗
  litellm-1     |    ██║     ██║╚══██╔══╝╔════╝██║     ██║     ████╗ ████║
  litellm-1     |    ██║     ██║   ██║   █████╗  ██║     ██║     ██╔████╔██║
  litellm-1     |    ██║     ██║   ██║   ██╔══╝  ██║     ██║     ██║╚██╔╝██║
  litellm-1     |    ███████╗██║   ██║   ███████╗███████╗███████╗██║ ╚═╝ ██║
  litellm-1     |    ╚══════╝╚═╝   ╚═╝   ╚══════╝╚══════╝╚══════╝╚═╝     ╚═╝
  litellm-1     |
Enter fullscreen mode Exit fullscreen mode

Then, open http://localhost:4000/ui/ and see the dashboard:

Model and Virtual API Key Definition

To start using LiteLLM as a proxy, an LLM provider needs to be configured, and then a virtual API key added that the client applications, or any tool working with an OpenAPI endpoint for this matter, uses.

Model Setup

LiteLLM provides many built-in providers, including direct access to OpenAI, Anthropic, cloud providers like Azure AI, Google Vertex AI, and AWS Bedrock, as well as Mistral, Cohere, and OpenRouter. See the full list at LiteLLM providers.

For demonstration purposes, a direct connection to OpenAI will be configured.

  • On the dashboard, in the left menu, click on Model + Endpoint
  • In the vertical tab list, click on Add Model, and the same-named dialogue appears
  • Fill out the details as follows:
    • Provider: OpenAI
    • LiteLLM Model Name(s): Put in the models of you choice, e.g. the moste recent gpt-5.6-luna, gpt-5.6-terra and gpt-5.6-sol
    • Mode: Chat - /chat/completions
    • OpenAI API key: sk-21113tfa ... no, just kidding, use your own key

Click on Test Connection, and when the success message appears, click on Save Model.

API Key

Virtual API keys are a strong feature of LiteLLM, allowing the provision to individual developers or groups.

Follow these steps for key generation:

  • On the dashboard, in the left menu, click on Model + Endpoint > Virtual Keys.
  • Click on the + Create New Key button
  • In the resulting dialog, fill out these details:
    • Owned By: You
    • Key Name: Provide a custom name, e.g. openai-api-key
    • Models: leave empty
    • Key Type: stick with "AI APIs" (so that it can use only the model APIs)
    • Under Additional Options, several interesting features are provided, but outside the scope of this article

Testing

To use the API key, you can either issue a curl statement to the local endpoint ...

curl http://localhost:4000/v1/chat/completions \
  -H 'Authorization: Bearer sk-REDACTED' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-5.6-luna",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'
Enter fullscreen mode Exit fullscreen mode

... or acces the LiteLLM dashboard and cick on Playground in the vertical menu. Enter the virtual key, select the model, and start chatting.

Conclusion

The proliferation of LLM providers creates both opportunity and overhead. When an increasing array of tools and applications access LLMs unrestrained, excessive token usage and cost explosion is just a matter of time. LLM proxies provide essential key management and budget management options, reducing operational risks tremendously.

In this introductory article, you learned about LiteLLM. Specifically: a) its feature set, b) the production setup options that support the major cloud providers and deployment options via Kubernetes or Terraform, c) a full guide to local setup using Docker containers, and d) the configuration of an LLM provider, an API key, and testing the LiteLLM proxy access through curl or via the buildin playground.

Top comments (1)

Collapse
 
misbahsy profile image
Misbah Syed •

thanks for sharing this Sebastian!