Cutting ASR Inference Cost with NVIDIA MPS on Amazon EC2

When an ASR pipeline is pushed to production, the interesting question is not only how fast it runs, but how much throughput you can extract from each GPU before latency starts to break. In the setup described here, that tradeoff was the main lever for reducing inference cost by 75% using NVIDIA MPS on Amazon EC2.

This post is a collaboration between AWS, NVIDIA, and Heidi. It also includes input from Jerron Chua, a Deep Learning Architect at the Generative AI Innovation Center at Amazon Web Services (AWS).

Why GPU sharing matters for inference

If you are running speech recognition at scale, a single model instance per GPU is often not the most efficient use of hardware. There are a few common ways to share GPU capacity:

• Time-slicing, where work from multiple processes is interleaved on the same device
• MIG, which partitions supported GPUs into isolated slices
• MPS, which allows multiple CUDA clients to run concurrently through a single GPU context

The key detail here is that NVIDIA CUDA MPS is a binary-compatible alternative implementation of the CUDA API. That matters because it lets existing CUDA-based workloads benefit from concurrent execution behavior without rewriting the application around a different programming model.

For inference workloads, the practical question is whether that concurrency improves utilization without pushing latency outside the acceptable range. That is where the rest of the setup comes in.

The inference pipeline on EC2

The pipeline in this work runs on Amazon EC2 g6e.4xlarge and g7e.4xlarge instances, both using NVIDIA L40S GPUs with 48 GB of memory. The architecture is built from three containerized components, which keeps the deployment modular and easier to reason about when measuring performance and tuning concurrency.

That container-based layout is important for a few reasons:

1. It isolates responsibilities across the pipeline.
2. It makes it simpler to swap configurations while keeping the inference flow comparable.
3. It helps with repeatable benchmarking when evaluating whether MPS is actually improving efficiency.

The article’s focus is not on a brand-new inference stack. It is on taking a practical deployment, then using MPS to improve the cost-to-throughput balance.

What you need before building the image

A key prerequisite is the NVIDIA Triton Inference Server container:

nvcr.io/nvidia/tritonserver:26.03-py3

That base image anchors the serving side of the workflow. From there, the setup builds an all-in-one image for the specific model package used in the test.

The build command is:

docker build -f Dockerfile.single \
  --build-arg LOCAL_NEMO_FILENAME=your_model.nemo \
  -t parakeet-mps:latest .

A few implementation details are worth calling out for builders:

• The Dockerfile is named Dockerfile.single.
• The model file is passed in as a build argument through LOCAL_NEMO_FILENAME.
• The resulting image is tagged as parakeet-mps:latest.

This is a straightforward pattern if you already package models into containers for inference. The main thing is that the model artifact is baked into the image build flow, which makes the runtime environment more reproducible when you compare different GPU-sharing strategies.

Triton plus MPS on g6e.4xlarge

One of the evaluated configurations is Triton + MPS on g6e.4xlarge. The important takeaway from the benchmark is not just that MPS improves utilization, but where the operating point lands once latency is considered.

The source identifies the optimal operating point as the last concurrency level where mean latency stays below 650 ms and p99 stays below 1,000 ms.

That distinction matters because raw throughput alone can be misleading. A configuration can process more requests if you keep increasing concurrency, but if the tail latency climbs too far, it stops being useful for many production ASR scenarios. The benchmark therefore uses a practical latency boundary instead of treating every additional request in flight as a win.

For developers tuning a similar system, this gives a useful rule of thumb:

• Increase concurrency until utilization improves.
• Watch both mean and p99 latency.
• Stop at the last point that still satisfies the service objective.

That approach is more operationally relevant than maximizing throughput in isolation.

Why the result is meaningful

The headline result is a 75% reduction in inference cost. The reason this is interesting is not that the number itself is magical, but that it comes from improving how the GPU is shared rather than replacing the model or redesigning the full pipeline.

That makes the technique attractive when:

• The model is already working correctly.
• The bottleneck is inefficient GPU usage rather than model quality.
• You need better economics without changing the core ASR behavior.

At the same time, MPS is not a universal answer. The same latency constraints that make the configuration viable also limit how far you can push concurrency. If the workload becomes too crowded, latency moves outside the acceptable range and the benefit disappears. That is why the benchmarked operating point matters more than a generic promise of “more throughput.”

Operational cleanup matters too

There is one practical detail that is easy to overlook after benchmarking: clean up the resources.

In this setup, that includes deleting attached Amazon EBS volumes, which may contain:

• model checkpoints
• TensorRT cache

For anyone running iterative performance experiments on EC2, this is not just housekeeping. Leftover volumes can continue to incur cost after the test is over, and they can also leave behind artifacts that confuse future runs if you are trying to reproduce results cleanly.

Takeaway for builders

If you are tuning ASR inference on EC2, the useful lesson from this setup is that GPU sharing can be a cost lever when it is evaluated against explicit latency targets. NVIDIA MPS, used with Triton on L40S-backed EC2 instances, can improve the economics of inference, but only when you choose the concurrency level that still respects both mean and tail latency.

In other words, the optimization is not “run more at any cost.” It is “find the highest concurrency that still behaves like a production service.” That is what makes the 75% cost reduction operationally meaningful rather than just a benchmark number.