We've been building a robust AI gateway at Portkey for the past 10 months.
Over this time, Portkey has been used by developers to test their prompts and measure costs and performance. We have been interface for shuttling 100B tokens and 10M requests daily to 100+ LLMs.
We've believed that to accelerate Gen AI adoption, companies and AI engineers need strong foundational platforms so going to production is not a nervous affair. The AI gateway in our mind is one of the MOST critical pieces of infrastructure in the new AI stack.
Guess what? Today, we are officially open-sourcing the magic sauce, that got us here.
Portkey's Opensource AI Gateway
Portkey-AI
/
gateway
A Blazing Fast AI Gateway with integrated Guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.
The AI Gateway streamlines requests to 250+ language, vision, audio and image models with a unified API. It is production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.
✅ Blazing fast (9.9x faster) with a tiny footprint (~100kb build)
✅ Load balance across multiple models, providers, and keys
✅ Fallbacks make sure your app stays resilient
✅ Automatic Retries with exponential fallbacks come by default
✅ Configurable Request Timeouts to easily handle unresponsive LLM requests
✅ Multimodal to support routing between Vision, TTS, STT, Image Gen, and more models
✅ Plug-in middleware as needed
✅ Battle tested over 480B tokens
✅ Enterprise-ready for enhanced security, scale, and custom deployments
Setup & Installation
Use the AI gateway through the hosted API or self-host the open-source or enterprise versions…
The AI gateway is an essential component of Portkey's platform. Portkey's AI Gateway helps you route requests to multiple LLMs and enables you to build on a unified API.
Why an AI Gateway?
The Generative AI ecosystem has been expanding rapidly (for good!), leading to a lack of resiliency and a disjointed developer experience when building with different components.
Imagine that you have added a Generative AI feature to your SaaS app, which your users use widely. However, there is a new model available from the Large Language model provider that you want to test before implementing it in production.
Although you are satisfied with the success of your current AI feature, you still lack insights into the costs and errors from your users.
To address these issues, Portkey's AI Gateway is specifically designed to assist developers.
Key Features
Universal API
AI Gateway offers a universal and unified API that allows interaction with over 100 LLMs, whether privately deployed or public LLM vendors. We take care of request and response transformations automatically.
Load Balancing
Portkey's Load Balancing feature efficiently distributes network traffic across multiple Language Model APIs, preventing any LLM from becoming a performance bottleneck and ensuring high availability and optimal performance of your generative AI apps.
Fallbacks
Several Language Model APIs are available in the market, each with its strengths and specialities. It would be very convenient if the APIs could be easily switched between based on their performance or availability. Portkey's Fallback feature enables one to switch between multiple LLMs.
Just imagine your users are not affected by your LLM Vendor's downtime.
Getting Started
The gateway is built in TypeScript and blazingly fast to try out or deploy.
npx @portkey-ai/gateway
Building with AI Gateway
While we understand the promise of universal API, we also acknowledge the learning curve of a new API.
Most of our SDKs support OpenAI convention and work with Llamaindex and LangChain.
| Language | Supported SDKs |
|---|---|
| Node.js / JS / TS |
Portkey SDK OpenAI SDK LangchainJS LlamaIndex.TS |
| Python |
Portkey SDK OpenAI SDK Langchain LlamaIndex |
| Go | go-openai |
| Java | openai-java |
| Rust | async-openai |
| Ruby | ruby-openai |
Portkey's LLM Developer Community
Open-sourcing the gateway can lead to more community involvement and innovation, making it even more useful for developers building AI-enabled experiences.
Please join us with hundreds of developers at our discord to discuss building and contributing with AI.
We believe that Open-sourcing the gateway can lead to more community involvement and innovation, making it even more useful for developers building AI-enabled experiences.
Please consider giving Portkey's AI Gateway a star 🌟. We welcome code and non-code contributions from you - here are some good-first-issues to start.
If you have any questions, feedback. Please let us know in the comments below!
Top comments (1)
Congrats. This is awesome.
Please have some plans on building the official C# SDK.