First of all, thanks for writing such detailed comparison, but it seems that some of the things mentioned/assumed about TrueFoundry are not factually correct. Let me clarify a few of them:
The Migration Tax:
This section claims that TrueFoundry enforces users to use our SDK. This is not correct . TrueFoundry unifies all APIs and provides OpenAI compatible APItruefoundry.com/docs/ai-gateway/ch.... Which means you are NEVER vendor locked in into TrueFoundry
The snippet mentioned (from truefoundry sdk) is incorrect. There is no class named "LLMGateway" in truefoundry's client sdk.
Performance Difference
Statement:Gateway shares resources with training, deployment, agent services Fact: Gateway runs in isolation with control plane and provides <5ms of latency for all requests consistently. The gateway pods can auto-scale to handle 5000+ rps easily with no impact on latency truefoundry.com/blog/truefoundry-l...
Statement:Cold start: 60-90% faster (Kubernetes pod startup vs always-on) Fact: System autoscales based on requests, which means that problem of cold start never appears in the first place. TrueFoundry runs a minimum number of replicas and then scales up to handle large amount of traffic.
Statement:Failover: 50-100x faster (<100ms vs 5-10 seconds) Fact: TrueFoundry does intelligent failovers. We maintain health of targets and intelligently fallback based on policy. We never add 5-10 seconds. We would typically add Provider Latency (100-200ms maybe) for failover for first few requests and subsequent request failover is instant. (0ms additional latency!!) truefoundry.com/docs/ai-gateway/lo...
Hot Reload Configuration: We reload all configurations instantly. Thats a base design principle. Our enterprise clients have used to dynamically route traffic in cases of outages of providers.
Hi Debby,
I work and TrueFoundry and I just read it.
First of all, thanks for writing such detailed comparison, but it seems that some of the things mentioned/assumed about TrueFoundry are not factually correct. Let me clarify a few of them:
The Migration Tax:
This section claims that TrueFoundry enforces users to use our SDK. This is not correct . TrueFoundry unifies all APIs and provides OpenAI compatible API truefoundry.com/docs/ai-gateway/ch.... Which means you are NEVER vendor locked in into TrueFoundry
The snippet mentioned (from truefoundry sdk) is incorrect. There is no class named "LLMGateway" in truefoundry's client sdk.
Performance Difference
Statement:
Gateway shares resources with training, deployment, agent servicesFact: Gateway runs in isolation with control plane and provides <5ms of latency for all requests consistently. The gateway pods can auto-scale to handle 5000+ rps easily with no impact on latency
truefoundry.com/blog/truefoundry-l...
Statement:
Cold start: 60-90% faster (Kubernetes pod startup vs always-on)Fact: System autoscales based on requests, which means that problem of cold start never appears in the first place. TrueFoundry runs a minimum number of replicas and then scales up to handle large amount of traffic.
Statement:
Failover: 50-100x faster (<100ms vs 5-10 seconds)Fact: TrueFoundry does intelligent failovers. We maintain health of targets and intelligently fallback based on policy. We never add 5-10 seconds. We would typically add Provider Latency (100-200ms maybe) for failover for first few requests and subsequent request failover is instant. (0ms additional latency!!)
truefoundry.com/docs/ai-gateway/lo...
Cache hits: <2ms vs not availableWe support both semantic and exact match type caching.
truefoundry.com/docs/ai-gateway/ca...
Other incorrect mentions:
TrueFoundry doesn't have semantic caching. At all.: We have this: truefoundry.com/docs/ai-gateway/ca...Intelligent Failover: We have this: truefoundry.com/docs/ai-gateway/vi...Hot Reload Configuration: We reload all configurations instantly. Thats a base design principle. Our enterprise clients have used to dynamically route traffic in cases of outages of providers.All this information is available in docs here: truefoundry.com/docs/ai-gateway/in...
You can also use the "Ask AI" feature to get answers to most of these questions.
Happy to clarify if there is any confusion.
Thanks,
Nikhil