DEV Community: Viktor Novak

7 Best High-Performance LLM Gateways for Low-Latency Workloads

Viktor Novak — Thu, 23 Jul 2026 21:33:38 +0000

An LLM gateway is the control plane for production AI, but adding a new network hop requires a strict latency budget. This guide compares the top AI gateways engineered for high-throughput, low-overhead performance.

Adding an AI gateway between applications and large language models is a standard practice for managing production workloads. This centralized layer handles critical tasks like API key management, cost control, request logging, and routing across multiple providers. However, a gateway also introduces a new network hop, and the overhead it adds to each request can become a significant bottleneck, especially for real-time, latency-sensitive applications. For these workloads, choosing a gateway built for high performance is essential.

An effective low-latency gateway minimizes its own processing time, often measured in microseconds or single-digit milliseconds, ensuring that end-to-end response time is dominated by the LLM's inference speed, not the infrastructure in front of it. This guide examines seven of the best high-performance LLM gateways, comparing them on architecture, published benchmarks, and features that impact speed at scale.

What to Look For in a Low-Latency Gateway

When evaluating gateways for speed, the key metric is gateway overhead: the amount of time the gateway itself spends processing a request before forwarding it and after receiving a response.

Core Architecture: Gateways built in compiled languages like Go or Rust generally offer lower overhead and better concurrency management than those built with interpreted languages like Python.
Benchmarked Performance: Look for published benchmarks that measure latency (p50, p99) and throughput (requests per second) under load. Crucially, these benchmarks should isolate gateway overhead from the upstream model's response time.
Caching Strategy: Caching is critical for reducing latency on repeated queries. Semantic caching offers higher hit rates than simple exact-match caching.
Deployment Model: Self-hosted gateways provide more control over network topology and can be placed closer to your application servers to minimize network latency.

The Top 7 Low-Latency LLM Gateways

Here is a comparison of the top-performing gateways available today, ranked based on their suitability for high-throughput, low-latency workloads.

1. Bifrost

Bifrost is an open-source AI gateway from Maxim AI, written in Go and architected specifically for maximum performance. It consistently appears at the top of independent and internal benchmarks, making it the leading choice for teams that cannot compromise on speed.

Published benchmarks show Bifrost adds as little as 11 microseconds (µs) of overhead per request while sustaining 5,000 requests per second (RPS) on a t3.xlarge instance. In a direct comparison against a popular Python-based alternative, Bifrost demonstrated 54x lower P99 latency and 9.5x higher throughput under a 500 RPS load.

Best for: Mission-critical applications where latency is a primary constraint, such as real-time agents, high-frequency trading, or interactive chatbots. Its performance and self-hosting capabilities make it ideal for enterprises that require both speed and full control over their infrastructure.

Language: Go
Key Features: Automatic failover, load balancing, semantic caching, and native Model Context Protocol (MCP) support for agentic workflows.
Performance: Adds as little as 11µs of overhead at 5,000 RPS.
Deployment: Self-hosted (Docker, Kubernetes).

Beyond routing, the Bifrost AI gateway provides centralized governance and security controls. An associated product, Bifrost Edge, extends these same policies to AI traffic on employee devices, governing desktop and browser AI usage from the same control plane.

2. Kong AI Gateway

Built on top of its widely adopted API management platform, the Kong AI Gateway leverages Kong's high-performance NGINX core to manage AI traffic. Kong is designed for enterprise scale and reliability, and its AI gateway inherits these strengths.

Benchmarks published by Kong show it delivering significantly lower latency and higher throughput compared to several other gateways. While gateway overhead is typically under 10ms, the focus is on managing large-scale, enterprise-wide AI adoption with consistent performance.

Best for: Large enterprises, especially those already invested in the Kong ecosystem for API management. It provides a unified control plane for both traditional microservices and new AI services.

Language: Lua (on NGINX)
Key Features: Unified governance across all APIs, prompt engineering plugins, semantic caching (enterprise tier), and deep observability.
Performance: Benchmarks show high throughput and latency that is 86% lower than some Python-based alternatives.
Deployment: Self-hosted or cloud-managed.

3. LiteLLM

LiteLLM is a very popular open-source proxy that excels at providing a unified, OpenAI-compatible API for over 100 LLM providers. It is written in Python and is highly valued for its simplicity and extensibility.

While its Python architecture can introduce more overhead than Go or Rust-based alternatives, LiteLLM performance has improved significantly with optimizations. Benchmarks show that doubling instances can halve the median latency, dropping it from 200ms to 100ms in one test scenario. For many applications not operating at extreme scale, this level of performance is sufficient.

Best for: Startups and teams prototyping in a Python environment. Its ease of use and broad provider support make it one of the fastest ways to get started with a unified AI gateway.

Language: Python
Key Features: Broad provider support, virtual keys for budget management, and a simple setup process.
Performance: Median overhead can be around 7.5ms, with ongoing work to reduce it further.
Deployment: Self-hosted (Docker, Python package).

4. Cloudflare AI Gateway

Cloudflare AI Gateway is a managed service that leverages Cloudflare's massive global edge network to route and manage AI traffic. Its primary performance advantage comes from this network.

By caching responses at the edge, close to users, Cloudflare can dramatically reduce latency for repeated requests—by up to 90% according to their documentation. However, for uncached requests, the gateway itself adds a baseline latency of 20-60ms as traffic is routed through Cloudflare's infrastructure.

Best for: Applications already running on the Cloudflare ecosystem or those with a global user base that can benefit from edge caching. It offers a simple, managed solution for observability and control.

Architecture: Managed edge network proxy.
Key Features: Global edge caching, rate limiting, and detailed analytics dashboards.
Performance: 20-60ms of added latency for uncached requests, with significant reductions for cached hits.
Deployment: Cloud-managed SaaS.

5. OpenRouter

OpenRouter provides a unified API to a vast catalog of over 300 models from more than 60 providers. It operates as a managed service focused on giving developers the widest possible model selection through a single integration.

As a managed router, it adds a layer of latency for routing and value-add services. However, it also optimizes routing by selecting the best provider for a given model based on real-time performance. Its caching can also significantly reduce response times for popular prompts.

Best for: Teams prioritizing model variety and rapid experimentation. It's an excellent choice for applications that need to test or dynamically route between many different models without managing multiple integrations.

Architecture: Managed cloud service.
Key Features: Extensive model catalog, consolidated billing, and intelligent routing to the best-performing provider.
Performance: Latency varies by model and provider, but the platform is designed to route for speed.
Deployment: Cloud-managed SaaS.

6. NGINX Gateway Fabric

While not a dedicated LLM gateway out of the box, NGINX Gateway Fabric is an implementation of the Kubernetes Gateway API that can be configured for high-performance AI workloads. With the Gateway API Inference Extension, NGINX can perform intelligent, load-aware routing to different model-serving pods.

This approach treats AI traffic as a first-class workload within Kubernetes, leveraging NGINX's proven performance to route requests with minimal overhead. It's a powerful option for teams building their AI platform on Kubernetes.

Best for: Platform engineering and MLOps teams building custom AI infrastructure on Kubernetes. It offers a high-performance, flexible foundation for self-hosting models.

Language: C (NGINX Core)
Key Features: Kubernetes-native configuration, inference-aware routing, and unification of AI and traditional microservice traffic.
Performance: Leverages NGINX's extremely low-latency core.
Deployment: Self-hosted on Kubernetes.

7. Custom FastAPI Gateway

For teams with specific requirements, building a lightweight gateway using a high-performance web framework like FastAPI is a viable option. FastAPI's asynchronous capabilities, built on Starlette and Pydantic, allow it to handle high-concurrency LLM requests efficiently without blocking.

This approach offers maximum control but also requires the most engineering effort. The team is responsible for implementing features like key management, routing logic, and observability from scratch. Performance is excellent for a Python framework but may not match a purpose-built Go or Rust gateway under extreme load.

Best for: Teams with unique routing or integration needs that off-the-shelf gateways don't address, and who have the engineering resources to build and maintain their own infrastructure.

Language: Python
Key Features: Fully customizable, asynchronous by default, and strong data validation with Pydantic.
Performance: High for a Python framework, suitable for many production workloads.
Deployment: Fully self-managed.

Conclusion

For applications where every millisecond counts, the choice of an LLM gateway has a direct impact on the user experience. While many gateways offer similar features for routing and governance, their underlying architecture determines their performance ceiling.

Go-based gateways like Bifrost and established infrastructure like Kong and NGINX are engineered for the lowest possible latency and highest throughput. For teams that need to scale without compromising on speed, these solutions provide the most robust foundation. Teams evaluating options should review the LLM Gateway Buyer's Guide and consider running their own benchmarks to validate performance for their specific workload. The best choice ultimately depends on the balance between raw performance, feature requirements, and operational capacity. Teams can request a Bifrost demo or review its open-source repository to evaluate its performance directly.

Sources

Why AI Infrastructure Investment Is Surging — And Where the Money's Going

Viktor Novak — Tue, 14 Jul 2026 14:50:08 +0000

Global investment in AI infrastructure is experiencing an unprecedented surge, driven by escalating compute demands, expanding model complexity, and the imperative for efficient, scalable AI deployment. This post explores the forces behind this growth and the key areas attracting significant capital across hardware, cloud services, and specialized software.

The rapid advancement of artificial intelligence, particularly generative AI, is fueling an unprecedented surge in demand for specialized infrastructure. This includes not only the physical hardware and data centers but also the intricate software layers required to develop, deploy, and manage AI applications efficiently. The scale of this investment reflects a foundational shift in how organizations are building digital capabilities and competing in an AI-driven economy.

The Unprecedented Surge in AI Infrastructure Spending

The global AI infrastructure market is experiencing remarkable growth, with projections indicating a substantial increase in market size over the coming years. Experts forecast the market to reach approximately $418.8 billion by 2030, growing at a compound annual growth rate (CAGR) of 21.5% from its $158.3 billion valuation in 2025. Other analyses predict an even larger scale, with estimates suggesting the market could grow from $58.78 billion in 2025 to nearly $497.98 billion by 2034, registering a CAGR of 26.60%. Some reports suggest that global cloud and AI infrastructure capital spending could approach $1.5 trillion by 2027.

This rapid expansion is driven by several factors. The increasing complexity and size of AI models, the widespread adoption of AI across industries, and intense competitive pressures are all contributing to the escalating demand for robust underlying infrastructure. Enterprises are increasingly prioritizing investment in AI-centric systems, with IDC estimating an annual growth of 27% in enterprise spending on AI from 2022 to 2026.

Fundamental Drivers of AI Infrastructure Demand

The foundational requirements of modern AI models necessitate a complete rethinking of traditional computing infrastructure. The sheer computational demands, coupled with the need for efficient data handling, are reshaping investment priorities.

The Compute Conundrum: GPUs and Accelerators

Graphics Processing Units (GPUs) have emerged as the backbone of modern AI infrastructure due to their parallel processing capabilities, which are crucial for training and inference workloads in AI models. The demand for GPU acceleration in data centers is directly tied to the rise of AI, machine learning, and data-intensive applications. In 2025, GPUs alone constituted approximately 88.82% of processor architecture revenue.

The power consumption of these specialized chips is a key indicator of their intensity. GPUs designed for generative AI, which operated at around 700 watts in 2023, are expected to see next-generation chips drawing up to 1,200 watts. Major technology companies are deploying these accelerators at scale, with NVIDIA reportedly shipping millions of Blackwell GPUs.

Data Center Expansion and Energy Needs

The immense computational power required by AI translates directly into significant energy demands. AI data centers typically consume 3-5 times more power per square foot than traditional facilities. A single AI server rack can require 50-150 kilowatts of power, in contrast to the 10-15 kilowatts for conventional computing racks. This surge is primarily driven by dense GPU clusters that operate continuously at maximum capacity.

The International Energy Agency (IEA) estimates that global data center electricity consumption, which accounted for around 1.5% of global electricity consumption in 2024, is projected to double to approximately 945 TWh by 2030, representing just under 3% of total global electricity consumption. In the United States, AI data center power consumption could escalate from 3-4% of total U.S. electricity demand today to 8-12% by 2030. Addressing these energy requirements and associated cooling challenges is a critical aspect of AI infrastructure investment, with cooling alone accounting for 30-40% of total data center power use.

The Software and Platform Layer

While hardware forms the foundation, the software stack orchestrating AI workloads is equally vital. The infrastructure needed to deploy AI models reliably involves several layers, including containerization, orchestration, API design, monitoring, and operational reliability engineering. The market for AI software is also expanding, with the software segment of AI infrastructure projected to grow at a 16.02% CAGR through 2031. This layer enables teams to manage the complexity of AI model deployment and ensures models deliver real-world business value at scale.

Where the Capital is Flowing: Key Investment Areas

The significant capital flowing into AI infrastructure is being allocated across several interconnected areas, from chip manufacturing to specialized cloud services and data management platforms.

Hardware Innovation and Manufacturing

Investment in hardware innovation continues to be a primary focus. This includes research and development in chip design, the expansion of semiconductor foundries, and the development of specialized AI accelerators such as Application-Specific Integrated Circuits (ASICs). Companies are exploring custom silicon solutions to reduce reliance on third-party chip suppliers, a trend highlighted by AI developers working on in-house chips. Memory, particularly high-bandwidth memory (HBM), is also capturing an increasing share of AI capital spending, indicating a shift in where dollars concentrate within the supply chain.

Cloud AI Infrastructure and Services

Hyperscale cloud providers are at the forefront of AI infrastructure expansion, with the five largest US hyperscalers (Amazon, Alphabet, Microsoft, Meta, and Oracle) projected to collectively spend around $700 billion on capital expenditure in 2026. A substantial portion of this, approximately 75%, is directly tied to AI infrastructure. Goldman Sachs analysts initially underestimated this spending, with hyperscalers now forecast to devote around $750 billion to capital expenditures in 2026.

This investment fuels the development of dedicated AI cloud offerings and specialized instances designed for compute-intensive AI workloads. The cloud segment is expected to see significant growth due to the demand for scalable, on-demand GPU resources that support fluctuating AI, machine learning, and data analytics workloads. Companies like CoreWeave exemplify the trend of infrastructure verticalization, becoming specialized AI infrastructure providers through substantial financing.

Edge AI Deployment and Specialized Devices

Bringing AI processing closer to the source of data generation, known as edge AI, is another burgeoning area of investment. The global edge AI market is projected to reach $118.69 billion by 2033, growing at a CAGR of 21.7% from 2026 to 2033. This growth is driven by the increasing demand for low-latency data processing and real-time analytics at the network edge, particularly with the proliferation of IoT devices.

Investment in edge AI focuses on specialized hardware for low power consumption, suitable for IoT and mobile devices, and the development of platforms that can support advanced analytics directly at the edge. The hardware segment currently dominates the edge AI industry, holding a 51.8% revenue share in 2025.

Data Pipelines and Governance Platforms

The effective deployment and management of AI models depend heavily on robust data pipelines and governance platforms. While 78% of organizations reported using AI in 2024, only 1% of business leaders reported achieving AI maturity, highlighting a significant gap between model development and production deployment. Challenges such as data drift, lack of monitoring, infrastructure mismatches, and integration complexities often hinder successful AI model deployment.

Investment is therefore flowing into solutions that address these operational challenges. This includes platforms for managing infrastructure complexity, ensuring data quality, and providing comprehensive monitoring and observability. Organizations are actively centralizing data and integrating AI across their technology stacks to overcome siloed data issues and streamline AI accessibility. This emphasis on data management, governance, and MLOps tools is critical for bridging the gap between developing a model and reliably delivering its value in production.

Long-Term Outlook and Future Trends

The surge in AI infrastructure investment is expected to continue, with long-term forecasts pointing towards sustained growth. The imperative for sustainable AI development is also driving innovation in energy efficiency, with liquid cooling systems, for example, offering substantial energy savings compared to traditional air cooling. On-site power generation is also gaining traction to reduce reliance on overburdened utility grids.

As AI becomes more ubiquitous, the focus will broaden beyond just raw compute power to include infrastructure that enables faster time-to-value, optimizes costs, and ensures compliance. The deployment of AI workloads will increasingly embrace hybrid IT operating strategies, combining on-premises and cloud environments, as well as various types of accelerators and processors. The competition among hyperscalers, chip makers, and data center operators for compute, power, and capital will intensify, creating both opportunities and risks in this fast-evolving landscape.

Ultimately, the goal of this infrastructure buildout is to unlock the full potential of AI, driving innovation, improving productivity, and enabling new business models across nearly every sector of the global economy.

Sources

Energy demand from AI - IEA: https://www.iea.org/energy-demand-from-ai
What Are the Power Requirements for AI Data Centers?: https://www.poweranddatacenter.com/ai-data-center-power-requirements/
AI infrastructure: $1.5T CapEx wave and the risks building beneath it: https://seekingalpha.com/article/4617467-ai-infrastructure-1point5t-capex-wave-and-risks-building-beneath-it
Global AI Infrastructure Market Size and Growth Forecast 2030: https://www.bccresearch.com/market-research/artificial-intelligence/ai-infrastructure-market-report.html
AI Infrastructure Market Size, Share | Growth Report: https://www.fortunebusinessinsights.com/ai-infrastructure-market-109033

6 Ways to Load Test an AI Gateway Before Production

Viktor Novak — Thu, 09 Jul 2026 09:39:40 +0000

Evaluating an AI gateway's performance under load is crucial before deploying production AI applications. This guide outlines six essential ways to load test an AI gateway to ensure reliability and scalability.

Production AI applications demand robust infrastructure capable of handling fluctuating traffic, managing diverse models, and ensuring seamless user experiences. An AI gateway sits in the critical path of every LLM request, making its performance under load a deciding factor in overall application reliability and cost. Without rigorous load testing, bottlenecks can emerge, leading to degraded user experience, increased operational costs, and potential outages. This article explores six essential ways to load test an AI gateway, such as Bifrost, an open-source AI gateway from Maxim AI, to guarantee its readiness for real-world traffic.

1. Baseline Load Testing (Throughput and Latency)

The foundation of any performance strategy is understanding an AI gateway's baseline throughput and latency. This involves simulating a steady, expected volume of requests and measuring how the gateway performs under normal operating conditions. Key metrics to track include requests per second (RPS), median latency (p50), and tail latencies (p95, p99). For AI applications, low latency is crucial, with every 100ms of added latency measurably reducing user engagement.

An effective baseline load test should:

Simulate realistic traffic: Use payload sizes, model choices, and API call patterns that mirror actual production use.
Isolate gateway overhead: Measure the time the gateway itself adds to a request, separate from the upstream LLM provider's latency. This helps identify if the gateway is introducing unnecessary delays.
Establish performance goals: Define clear service level agreements (SLAs) for response times and error rates upfront.

For instance, Bifrost is engineered for minimal overhead, consistently adding only 11 microseconds of overhead per request at 5,000 requests per second in sustained benchmarks. This demonstrates its capacity to maintain performance under typical production loads without becoming a bottleneck.

2. Stress Testing (Breaking Point Analysis)

Stress testing pushes an AI gateway beyond its normal operating limits to identify its breaking point and how it behaves under extreme conditions. This type of test gradually increases the load until the gateway's performance degrades significantly, or it starts returning errors. The goal is to determine the maximum capacity the system can handle before becoming unstable. Stress tests help uncover bottlenecks that might not appear under regular load, such as resource exhaustion (CPU, memory, network I/O) or connection pool limits.

When conducting stress tests:

Monitor system resources: Track CPU, memory, and network usage of the gateway instances.
Observe error rates: A sharp increase in errors indicates the system is under stress.
Identify graceful degradation: A robust AI gateway should degrade gracefully, maintaining functionality even as performance declines, rather than outright crashing.

Understanding the breaking point allows teams to provision resources appropriately and to implement safeguards like dynamic scaling or rate limiting before production deployment.

3. Spike Testing (Sudden Traffic Surges)

Spike testing simulates sudden, extreme increases in user traffic over a short period to evaluate how the system responds to unexpected bursts of activity. This is particularly relevant for AI applications that can experience unpredictable demand spikes, such as viral content, breaking news, or agentic workflows that fan out into many simultaneous calls. An AI gateway should be able to absorb these spikes without collapsing, quickly scaling up and stabilizing.

Key aspects of spike testing:

Rapid load increase: Generate a sudden surge of requests, often many times the normal peak load, and then return to baseline.
Recovery time: Measure how quickly the gateway recovers and returns to stable performance after the spike subsides.
Autoscaling validation: For containerized deployments, verify that autoscaling mechanisms (e.g., Horizontal Pod Autoscalers in Kubernetes) trigger and provision new instances quickly enough to handle the increased demand.

Bifrost's Go-based architecture and efficient concurrency model are designed to handle such spikes, leveraging goroutines for lightweight, true parallelism across CPU cores, minimizing scheduling overhead even when thousands of requests arrive concurrently.

4. Failover and Resilience Testing (Provider Degradation)

AI gateways are often deployed to enhance reliability by providing automatic failover across multiple LLM providers. Failover testing validates whether the gateway correctly reroutes requests when a primary provider experiences outages, rate limits, or degradation. This is critical for maintaining application uptime in a multi-provider strategy. LLM provider outages are not uncommon, making robust failover a baseline requirement for production AI.

To test failover and resilience:

Simulate provider failures: Intentionally degrade or block connections to a primary LLM provider. This can involve triggering specific error codes (e.g., HTTP 429 for rate limits, 5xx for server errors).
Measure failover latency: Quantify the time it takes for the gateway to detect a failure and switch to a healthy backup provider.
Verify circuit breaker patterns: Ensure that the gateway stops hammering a failing provider and periodically tests its recovery before resuming full traffic.
Test streaming retries: For streaming responses, check how the gateway handles mid-stream failures. Transparent retries should occur for connection errors before any data is sent to the client.

Bifrost offers automatic fallbacks and adaptive load balancing, actively tracking provider health and routing around degraded services before they impact the application. This functionality ensures continuous operation even when upstream services are unreliable.

5. Concurrency and Rate Limit Testing

AI models often have strict rate limits from providers, and an AI gateway should effectively manage concurrency and enforce internal rate limits to prevent overages and ensure fair usage. Concurrency testing evaluates how many simultaneous requests the gateway can handle without significant performance degradation or triggering upstream rate limits.

This involves:

Varying virtual user counts: Incrementally increase the number of concurrent users or requests simulated against the gateway.
Monitoring provider rate limits: Confirm that the gateway's internal rate limiting and load balancing across API keys successfully prevents hitting external provider limits.
Testing per-consumer limits: For enterprise deployments, verify that the gateway's virtual keys and budget management correctly enforce per-user or per-project rate limits.

The Bifrost AI gateway acts as a control plane for identity, quota, observability, and routing. Bifrost Edge extends this same governance and security to AI traffic on employee machines, with endpoint enforcement on each device. This ensures that even shadow AI usage is subject to the virtual keys, budgets, and rate limits defined at the gateway, preventing uncontrolled concurrency from any source.

6. Soak Testing (Long-Duration Stability)

Soak testing, also known as endurance testing, subjects the AI gateway to a sustained, realistic load over an extended period (hours or even days). The goal is to uncover performance degradation, memory leaks, or other issues that only manifest after prolonged operation. Such issues might include resource exhaustion, database connection pool depletion, or gradual performance slowdowns.

During soak tests, pay close attention to:

Resource utilization trends: Look for steady increases in memory consumption or CPU usage that do not stabilize.
Performance drift: Check if latency or error rates gradually increase over time.
Log analysis: Monitor logs for recurring errors or warnings that might indicate underlying instability.

Soak testing is crucial for ensuring the long-term reliability of an AI gateway and identifying subtle issues that short-duration tests might miss. Bifrost's efficient memory model, a benefit of its Go-based design, helps it maintain consistent performance under load, consuming significantly less memory than many Python-based alternatives in comparable workloads. This efficiency is vital for sustained production readiness and predictable autoscaling.

Load testing an AI gateway is a continuous process that should be integrated into the CI/CD pipeline. Tools like Gatling, Locust, and k6 offer robust capabilities for defining complex scenarios, simulating high concurrency, and collecting detailed metrics. By implementing these six testing methods, teams can build confidence in their AI gateway infrastructure, ensuring that production AI applications remain performant, reliable, and cost-effective.

Sources

Gatling. "API load testing with Gatling". https://gatling.io/blog/api-load-testing-with-gatling/
DeepInspect. "AI Gateway Performance Benchmark: What to Measure and How". https://deepinspect.ai/blog/ai-gateway-performance-benchmark
Maxim AI. "Bifrost vs LiteLLM Benchmarks | 40x Faster LLM Gateway". https://www.getmaxim.ai/bifrost/resources/benchmarks
Maxim AI. "Top LLM Failover Platforms in 2026: A Buyer's Guide". https://www.getmaxim.ai/bifrost/blog/top-llm-failover-platforms-2026
Medium. "Load Testing AI APIs with Locust: A Practical Guide for Data & MLOps Engineers". https://medium.com/@mlops.community/load-testing-ai-apis-with-locust-a-practical-guide-for-data-mlops-engineers-579c372652b0