DEV Community

Mikhail Savchenko
Mikhail Savchenko

Posted on Originally published at inite.ai

SageMaker now flags latency drift and idle GPUs on its own

Amazon has added default monitoring to SageMaker AI inference endpoints, according to a post on the AWS Machine Learning Blog. The idea is that an endpoint tuned at launch slowly drifts as traffic, prompt length and serving software change, and nobody notices because nothing actually fails.

What is monitored

Monitoring is on for endpoints that use vLLM serving containers on GPU instances. It needs no setup. SageMaker looks at operational signals only: latency, GPU utilization and the endpoint's serving configuration. AWS states it does not read requests or responses.

Three kinds of findings are produced:

  • Latency drift: median end-to-end latency or time to first token rising compared with the previous week.
  • Unused capacity: GPUs you pay for that stay mostly idle.
  • Configuration gaps: a serving container that lacks recent performance enhancements or security fixes, or an improvement like speculative decoding or GPU kernel tuning that the model is capable of using but the endpoint does not employ.

Because latency is compared with the endpoint's own history, you do not set a threshold in advance.

How findings reach you

Notices arrive through AWS Health and by email, one per endpoint, covering all of that endpoint's findings. Each states which endpoint is affected, what was observed, why it matters and a recommended next step. They are specific to an AWS Region, so you see them when signed in to the Region where the endpoint runs. AWS also caps how many an account receives at once.

The recommended action opens SageMaker Studio with the endpoint and its recent traffic already loaded. An agent summarizes the finding and asks for confirmation before acting. It then generates a notebook, with each step in its own cell, that starts an inference recommendation job. The job compares your current setup with alternatives on response time, throughput and cost per million tokens.

What it costs and what it changes

Monitoring itself carries no GPU cost. Benchmarks run on separate compute and are billed at standard SageMaker rates, or against your ML reservation if you hold one. Your production endpoint stays unchanged until you apply a result. The post suggests applying a change to a single production variant first and keeping the old configuration available for rollback.

The example scenarios in the post include a chat assistant slowed by longer retrieval-based prompts, a code-completion model that never enabled speculative decoding, and a summarization endpoint whose busiest GPU used 12 percent or less of its capacity for most of a week. In that last case, AWS notes that keeping the current size can be the right call if growth is planned.

Opting out

To exclude an endpoint, set the OPT_OUT_TRACKING environment variable, with any value, in the model container's environment when calling CreateModel. SageMaker then collects no monitoring telemetry for that endpoint, including engine metrics and GPU metrics, and sends no notifications for it.

Acting on a finding requires access to AWS Health and AWS User Notifications, a SageMaker Studio domain, IAM permissions to run recommendation jobs and update endpoints, and enough service quota for the benchmark instances.

Top comments (0)