🎙️We’ve Built the Fastest Way to Run LLMs in Production (50x faster than LiteLLM)🔥

#webdev #javascript #programming #opensource

Today, AI applications are rapidly becoming more complex. Modern systems no longer rely on a single LLM provider, but must also switch between providers like OpenAI, Anthropic, and others, ensuring high availability, minimal latency, and cost control in production.

Direct API integrations or simple proxies no longer scale. Failures lead to downtime, persistent limits lead to errors, and ultimately lead to code rewrites. Full-fledged orchestration platforms and microservice solutions address these issues, but often prove too complex and, most importantly, too slow for applications.

That's why speed is so important today, ensuring clients receive responses as quickly as possible. For large websites, their audiences can reach enormous numbers, and every second saved is an extra cost. So, we created Bifrost to solve this and many other problems.

Well, let's get started! 🏎️

🌐 What is this project?

Bifrost is, first and foremost, a high-performance AI gateway with an OpenAI-compatible API that unifies 15+ LLM providers into a single access point and adds automatic failover, load balancing, semantic caching, and other essential enterprise features, all with virtually zero overhead and launches in seconds.

Since the architecture is Go-based, you can use the project on most stacks, without depending only on Node.js.

⏱️ Speed comparisons

Let's get down to business. Since LiteLLM is one of the most popular projects today, we'll compare it to it and see how Bifrost compares.

To give you an idea, we ran some benchmark tests at 500 RPS to compare performance of Bifrost and LiteLLM. Here are the results:

Both Bifrost and LiteLLM were benchmarked on a single instance for this comparison.

📊 Results on a bar chart

To make the result more visual, let's look at several diagrams in which we have displayed the numbers we obtained.

~9.5x faster, ~54x lower P99 latency, and uses 68% less memory than LiteLLM — on t3.medium instance (2 vCPUs) with tier 5 OpenAI Key.

As you can see, our project uses significantly less memory for calculations. Roughly speaking, without any figures, this means that if you use it, you'll need significantly fewer hosting resources to handle requests from 10,000 users, which means extra money for your plan.

📈 Results on a line chart

Now let's look at this wait time between sending a request and receiving a response.

And this is one of the most straightforward and revealing points. The shorter the wait for a request, the higher the conversion rate. It's simple. We'll wait about 5 seconds for LiteLLM to complete its work, while Bifrost will complete its work in less than a second.