Why a Forgotten SageMaker Endpoint Is a Common Surprise Bill

#ai #debugging #llm #machinelearning

The line reads something like Amazon SageMaker AI · USE1-Host:ml.g5.2xlarge · Hrs · 744, and 744 is the number of hours in a 31-day month. That is the shape of the charge: not a per-request price that got out of hand, but a machine that was switched on in a demo and never switched off.

The line item you are looking at

The usage type to look for begins with a Region prefix and the word Host — for example USE1-Host:ml.m5.xlarge. The Host family is SageMaker real-time inference hosting, and it is billed in instance-hours. Two other families are commonly confused with it and neither behaves this way: Notebk for notebook instances, and Inf or AsyncInf usage types for the inference modes that do scale to zero. If your surprise is a Host line, you have a real-time endpoint.

Multiply out the quantity before you go looking. A quantity that is a multiple of the hours in the month, times an InitialInstanceCount greater than one, is the signature of an endpoint that has been up continuously since it was created — often for several months before anyone noticed, because the monthly increase was constant rather than sudden and no anomaly detector flags a flat line.

Why it bills with no traffic

A SageMaker real-time endpoint is provisioned capacity. When you call CreateEndpoint, SageMaker launches the instance count and type named in the endpoint configuration’s ProductionVariants, pulls the inference container, loads the model artefact and holds all of it resident so that an InvokeEndpoint call can be answered in milliseconds. Those instances are yours from the moment the endpoint reaches InService. Nothing in that arrangement is conditional on traffic; the whole point of it is that the capacity is already there when a request arrives.

This is the difference between an endpoint and an API. A per-token model API charges for what you consumed because the provider is multiplexing your traffic across shared capacity. A dedicated endpoint gives you the capacity, and idle capacity is exactly as expensive as busy capacity. The same reasoning applies to any always-on serving tier — the version of this argument for API-versus-hosted generally is in self-host versus API.

What does not stop the charge

This is where the fix goes wrong, and AWS is unambiguous about it. Its guidance opens with “Delete endpoints to stop incurring charges” and then notes that “Deleting an endpoint will not delete the endpoint configuration or the SageMaker AI model” — see AWS on deleting endpoints and resources. The relationship runs one way only, and the inverse is what people try:

DeleteEndpointConfig does not stop the charge. The configuration is a template. Deleting it leaves the running endpoint running, and AWS warns that doing so while the endpoint is live costs you visibility into which instance type it is using — making the bill harder to explain, not cheaper.
DeleteModel does not stop the charge. The model object is a registry entry pointing at artefacts in S3. The endpoint already has what it loaded.
Removing the IAM role does not stop the charge. It breaks the endpoint without releasing the instances.
Setting desired instance count to zero is not available on a plain real-time variant. Application Auto Scaling has a minimum capacity of one for a classic variant; scale-to-zero exists for some newer serving surfaces but is not the default behaviour you get from CreateEndpoint.

Only one call ends it:

aws sagemaker delete-endpoint --endpoint-name my-forgotten-endpoint

Finding every endpoint you own

ListEndpoints is a per-Region call, and a forgotten endpoint is very often in a Region nobody looks at because a tutorial defaulted to it. Sweep all of them:

for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  aws sagemaker list-endpoints --region "$r" --status-equals InService \
    --query 'Endpoints[].[EndpointName,CreationTime]' --output text \
    | sed "s/^/$r\t/"
done

Then decide which are idle, and here is the trap that makes automated detection unreliable. The Invocations metric in the AWS/SageMaker namespace is only emitted when the endpoint is invoked. A completely unused endpoint does not publish Invocations = 0 — it publishes nothing at all. A CloudWatch alarm on Sum of Invocations < 1 therefore sits in INSUFFICIENT_DATA forever and never fires, which is precisely backwards from what you wanted. The alarm has to treat missing data as breaching:

aws cloudwatch put-metric-alarm \
  --alarm-name "sagemaker-idle-my-endpoint" \
  --namespace AWS/SageMaker \
  --metric-name Invocations \
  --dimensions Name=EndpointName,Value=my-endpoint Name=VariantName,Value=AllTraffic \
  --statistic Sum --period 86400 --evaluation-periods 7 \
  --threshold 1 --comparison-operator LessThanThreshold \
  --treat-missing-data breaching \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:cost-alerts

--treat-missing-data breaching is the whole fix. Without it the alarm is decorative.

Stopping it happening again

Endpoints created by hand are the ones that get forgotten, because nothing owns them. The structural answer is that an endpoint should be a resource in a Terraform or CloudFormation stack, so destroying the stack destroys the endpoint and a review of what exists is a review of what is checked in — Terraform for AI infrastructure covers the general shape.

Tag at creation, and require it. An Owner and an ExpiresOn tag turn a monthly sweep into a list of names rather than an archaeology exercise. An SCP or IAM condition on aws:RequestTag/Owner for sagemaker:CreateEndpoint makes it non-optional.
Reach for a mode that does scale to zero. Serverless inference and asynchronous inference bill for what they process; batch transform runs and exits. A real-time endpoint is the right answer only when you genuinely need a warm instance for latency, and an experiment almost never does.
Set a budget scoped to the service. An AWS Budget filtered to Amazon SageMaker with a low absolute threshold catches the first month rather than the fourth — see per-environment budgets for model calls for the tag-filtered version.
Right-size before you rationalise. If the endpoint is genuinely needed, the instance type is usually the larger lever — SageMaker instance right-sizing.

One more thing to check before you close the ticket: an endpoint deleted today stops accruing from the moment it is deleted, but the charge already on the bill is for hours already consumed and is not refundable as a matter of course. Delete first, then work out how long it ran, and only then decide whether to open a support case.