How I Run ML Inference at $0 Idle Compute with SageMaker Async Inference (CDK Included)

A few months ago, I wrote an article “Stop Using Lambda for ML at This Scale”, where I’ve taken a deeper dive into packing a ML model into a Lambda and tested out at what scale does that approach cost more than using a more conventional approach, like using Sagemaker. And that post is great if you are approaching it as a startup which needs to save as much money as possible.

However, in this post, we are going to take a closer look at the architecture which I’ve actually shipped in one of my startups called HeartSense.

HeartSense is a serverless platform which gives a suggestion to the user if they should go and visit a cardiologist, based on a recording of their heartbeats by using their phone’s microphone. The system simply categorizes the sounds by either healthy or unhealthy , by using a ResNet18 model in the background. This product is not meant to be a medical one, and any result of the project should not be taken as medical advice.

In this blog post, I’m giving you my ML pipeline playbook with AWS CDK in Python — every snippet below is from the real-world application. All code, for training the model and CDK infrastructure code, can be found by clicking on the link here.

The first version: one Lambda doing too much

The first version was a single fat Lambda. It downloaded the WAV, generated a spectrogram, ran the ONNX model, and wrote the result — all in one invocation.

It worked in a demo. In practice it had three problems:

1. Coupling. Preprocessing (CPU, fast, cheap) and inference (heavy, slow) shared one timeout, one memory setting, and one failure domain. A slow inference would hold the whole invocation hostage.
2. Cold starts. Loading the model into the Lambda runtime on every cold start added seconds to user-facing latency.
3. No back-pressure. A burst of uploads meant a burst of concurrent heavy Lambdas, with nothing to smooth the spikes.

The fix to all of these problems is to decouple preprocessing code from inference and put a buffer between them.

The architecture

The flow:

1. User requests a presigned URL to upload the WAV file, the API creates an AnalysisRecord in DynamoDB and sets the status to PENDING , and returns the URL for WAV file upload
2. The user uploads a WAV straight to S3
3. S3 creates an ObjectCreated event under the recordings/ prefix, which lands on an SQS queue
4. A preprocessing Lambda (which is configured to use a Docker image, because the scientific Python deps are too big for a zip) pulls the information about the uploaded WAV file, validates the audio, renders a 224×224 spectrogram, uploads it to S3, and calls the async SageMaker endpoint.
5. SageMaker runs inference on its own schedule and drops the JSON which contains results into S3 under results/.
6. That write triggers a tiny result-processor Lambda that parses the prediction and updates DynamoDB.

Nothing here runs unless there’s work to do. The SageMaker endpoint scales to zero instances when idle. Let’s build it.

Step 1 — Buffer uploads with SQS + S3 events

The SQS queue used in this architecture is the shock absorber and gives us batching abilities, partial failure handling and a DLQ — things you don’t get out of the box when using a direct S3-to-Lambda trigger.

When the user uploads the recording of their heartbeat, a S3 event gets created and goes straight to the queue, instead to the Lambda. The reason is to reduce the number of Lambda invocations and to use batching to lower Lambda provisioning and usage cost. One Lambda can handle multiple S3 events and process everything in one or multiple batches, depending on the load on the system.

A very important detail is to know how to handle bad and malicious files too, that’s why this architecture has a DLQ set up, so the data doesn’t get lost anywhere in the system. A platform administrator can take a look into the failing data any time.

The CDK is straightforward — note the dead-letter queue wired in from the start, and the prefix filter so only recordings/ objects trigger the pipeline:

# infra/stacks/queue_stack.py

# Dead Letter Queue — captures messages that fail 3 times (14-day retention)
self.dlq = sqs.Queue(
    self,
    "DeadLetterQueue",
    queue_name=f"heartsense-dlq-{stage}",
    retention_period=Duration.days(SQS_RETENTION_DAYS),
)

# Processing Queue — buffers S3 upload events for the Preprocessing Lambda
self.processing_queue = sqs.Queue(
    self,
    "ProcessingQueue",
    queue_name=f"heartsense-processing-queue-{stage}",
    retention_period=Duration.days(SQS_RETENTION_DAYS),
    visibility_timeout=Duration.seconds(SQS_VISIBILITY_TIMEOUT_SECONDS),
    dead_letter_queue=sqs.DeadLetterQueue(
        max_receive_count=SQS_MAX_RECEIVE_COUNT,  # 3 attempts, then DLQ
        queue=self.dlq,
    ),
)

# S3 → SQS: only ObjectCreated events under recordings/
audio_bucket.add_event_notification(
    s3.EventType.OBJECT_CREATED,
    s3n.SqsDestination(self.processing_queue),
    s3.NotificationKeyFilter(prefix="recordings/"),
)

Two settings matter more than they look:

• visibility_timeout - AWS recommends setting the SQS visibility timeout to at least six times the value of the Lambda function timeout + the number of seconds of the SQS queue batching window. This configuration give the Lambda enough time to handle any throttling and/or retry attempts which could happen - you can read more about this configuration on official AWS documentation web page - https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-configure.html
• max_receive_count = 3 - after three failed attempts a message lands in the DLQ instead of being retried forever. That DLQ is also where your CloudWatch alarm should point (more on failure handling later).

One gotcha worth mentioning: adding an S3 notification that targets a resource in another CDK stack creates a circular dependency. I import the audio bucket by name inside this stack to break the cycle, rather than passing the L2 bucket construct across stacks.

---

Step 2 — The preprocessing Lambda (Docker)

For spectrogram generation, the Python code needs libraries like scipy , Pillow and others, which inflate the Lambda ZIP way above the limit. That’s why, in this architecture, I’ve made the preprocessing Lambda be a container image Lambda. CDK builds the image from the project root and pushes it to ECR for you:

# infra/stacks/preprocessing_stack.py

self.preprocessing_function = _lambda.DockerImageFunction(
    self,
    "PreprocessingFunction",
    function_name=f"heartsense-preprocessing-{stage}",
    code=_lambda.DockerImageCode.from_image_asset(
        directory=project_root,
        file="src/services/preprocessing/Dockerfile",
        platform=ecr_assets.Platform.LINUX_AMD64,
        exclude=["infra/cdk.out", ".git", "node_modules", "ai/data", "..."],
    ),
    timeout=Duration.seconds(PREPROCESSING_LAMBDA_TIMEOUT_SECONDS),  # 300s
    memory_size=PREPROCESSING_LAMBDA_MEMORY_MB,                      # 1024 MB
    environment={
        "AUDIO_BUCKET_NAME": audio_bucket.bucket_name,
        "SPECTROGRAM_BUCKET_NAME": spectrogram_bucket.bucket_name,
        "SAGEMAKER_ENDPOINT_NAME": sagemaker_endpoint_name,
        "IDEMPOTENCY_TABLE_NAME": self.idempotency_table.table_name,
        # ...
    },
    tracing=_lambda.Tracing.ACTIVE,
)

# SQS trigger with partial batch failure reporting
self.preprocessing_function.add_event_source(
    lambda_event_sources.SqsEventSource(
        processing_queue,
        batch_size=SQS_BATCH_SIZE,               # 10 as default, but can be adjusted
        max_batching_window=Duration.seconds(SQS_BATCH_WINDOW_SECONDS),
        report_batch_item_failures=True,                            # <-- important
    )
)

report_batch_item_failures=True is the unsung hero in the configuration. Without it, one bad message in a batch of 10 fails the entire batch and re-drives all 10. With it, the handler can return just the IDs that failed, and SQS only retries those.

The intermediate artifacts which the model sees: a 224×224 mel-spectrogram generated from the uploaded WAV file.

Inside the handler, the batch loop classifies every error as transient (retry via SQS) or permanent (don’t retry, mark the record failed):

# src/services/preprocessing/handler.py

def handler(event: dict, context: LambdaContext) -> dict:
    records = event.get("Records", [])
    batch_item_failures: list[dict[str, str]] = []

    for record in records:
        message_id = record["messageId"]
        try:
            _process_record(record)
        except TransientError:
            # report as failure so SQS retries just this message
            batch_item_failures.append({"itemIdentifier": message_id})
        except PermanentError:
            # already marked FAILED in DynamoDB; consume the message
            pass
        except Exception:
            # unknown errors are treated as transient (safer to retry)
            batch_item_failures.append({"itemIdentifier": message_id})

    return {"batchItemFailures": batch_item_failures}

That transient-vs-permanent distinction is the difference between a self-healing pipeline and one that hammers a dead endpoint forever.

For example, a throttle or a 5xx is transient, while on the other hand, a corrupt WAV is permanent — even if we try and process the audio file a million times, it will still be corrupted.

Step 3 — Make it idempotent

S3-to-SQS delivery is at-least-once — which means that some point of time, the same upload event will delivered twice. Hence, we need to protect our system from processing the same information twice, saving processing time and space.

I use Lambda Powertools idempotency, keyed on the analysis_id, backed by a small DynamoDB table with TTL:

# src/services/preprocessing/handler.py

persistence_layer = DynamoDBPersistenceLayer(table_name=IDEMPOTENCY_TABLE_NAME)
idempotency_config = IdempotencyConfig(
    event_key_jmespath="analysis_id",
    expires_after_seconds=3600,  # 1 hour
)

@idempotent_function(
    data_keyword_argument="data",
    config=idempotency_config,
    persistence_store=persistence_layer,
)
def _process_audio_idempotent(data: dict[str, str]) -> dict[str, Any]:
    _process_audio(**data)
    return {"status": "success", "analysis_id": data["analysis_id"]}

The idempotency table is defined right next to the function, with PAY_PER_REQUEST billing and a TTL attribute so old records expire on their own:

# infra/stacks/preprocessing_stack.py
self.idempotency_table = dynamodb.Table(
    self, "IdempotencyTable",
    partition_key=dynamodb.Attribute(name="id", type=dynamodb.AttributeType.STRING),
    billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
    time_to_live_attribute="expiration",
)

In addition, another layer of idempotency was added too: every DynamoDB status update is a conditional write. The record only moves PENDING → PROCESSING → INFERRING if it’s currently in the expected state. A duplicate invocation simply fails the condition check and no-ops — no exceptions, no double processing.

# src/shared/constants.py — the state machine these conditions enforce
VALID_STATUS_TRANSITIONS = {
    AnalysisStatus.PENDING:    {AnalysisStatus.PROCESSING, AnalysisStatus.FAILED},
    AnalysisStatus.PROCESSING: {AnalysisStatus.INFERRING,  AnalysisStatus.FAILED},
    AnalysisStatus.INFERRING:  {AnalysisStatus.COMPLETED,  AnalysisStatus.FAILED},
    AnalysisStatus.COMPLETED:  set(),  # terminal
    AnalysisStatus.FAILED:     set(),  # terminal
}

Here is the state machine diagram which explains this in a more simple way:

Step 4 — Invoke SageMaker asynchronously

This is one of the crucial parts of this ML processing pipeline — instead of calling the Sagemaker endpoint synchronously and waiting for the result, we are using invoke_endpoint_async to call the endpoint asynchronously, so we don’t have additional Lambda usage for no reason. After sending the processing request to the SageMaker endpoint, Lambda’s job is done, it doesn’t have to wait for the result.

What I’ve learned here is that the ways of calling the SageMaker endpoint changes, based on the way you are calling it — with a synchronous endpoint, you would usually send the payload / request body to it and wait for the response. On the other hand, with SageMaker Async Inference, the approach to processing data is different:

1. you upload the input to S3
2. retrieve the S3 file URI and pass it into the invoke_endpoint_async method as the InputLocation argument
3. let SageMaker write the result back to S3

# src/services/preprocessing/handler.py

response = sagemaker_runtime_client.invoke_endpoint_async(
    EndpointName=SAGEMAKER_ENDPOINT_NAME,
    InputLocation=input_s3_uri,            # s3://.../spectrograms/<user>/<id>.jpg
    ContentType="image/jpeg",
    InferenceId=analysis_id,
    Filename=f"{analysis_id}.out",         # forces a deterministic output key
    InvocationTimeoutSeconds=900,
)

Hard-won detail: set Filename explicitly. If you don’t, SageMaker writes the async output under a random UUID that won’t match the InferenceId you passed. By forcing Filename={analysis_id}.out, the result lands at a deterministic key (results/{analysis_id}.out) and the downstream Lambda can map it straight back to the DynamoDB record — no lookup table, no scan.

The invocation is wrapped in manual exponential backoff (1s, 2s) for transient SageMaker errors, and disables boto3’s built-in retries so the two retry mechanisms don’t fight each other.

Step 5 — The scale-to-zero endpoint (the money shot)

Here’s the part that makes idle cost disappear. The async endpoint is configured with an AsyncInferenceConfig (output + failure S3 paths, one concurrent invocation per instance):

# infra/stacks/inference_stack.py

endpoint_config = sagemaker.CfnEndpointConfig(
    self, "InferenceEndpointConfig",
    production_variants=[
        sagemaker.CfnEndpointConfig.ProductionVariantProperty(
            variant_name="AllTraffic",
            model_name=model.model_name,
            initial_instance_count=1,
            instance_type="ml.t3.medium",
        )
    ],
    async_inference_config=sagemaker.CfnEndpointConfig.AsyncInferenceConfigProperty(
        output_config=sagemaker.CfnEndpointConfig.AsyncInferenceOutputConfigProperty(
            s3_output_path=async_output_s3_uri,    # results/
            s3_failure_path=async_failure_s3_uri,  # failures/
        ),
        client_config=sagemaker.CfnEndpointConfig.AsyncInferenceClientConfigProperty(
            max_concurrent_invocations_per_instance=SAGEMAKER_MAX_CONCURRENT_INVOCATIONS,
        ),
    ),
)

Then Application Auto Scaling with min_capacity=0 lets the endpoint drop to zero instances when there’s nothing in the queue. Async endpoints are the only SageMaker endpoint type that can do this:

def _create_autoscaling(
    self, endpoint_name: str, stage: str, endpoint: sagemaker.CfnEndpoint
) -> None:
    from aws_cdk import aws_applicationautoscaling as appscaling
    from aws_cdk import aws_cloudwatch as cloudwatch
    from aws_cdk import aws_cloudwatch_actions as cw_actions

    # Scale between 0 and 1 instances for this low-cost async endpoint.
    scalable_target = appscaling.ScalableTarget(
        self,
        "EndpointScalableTarget",
        service_namespace=appscaling.ServiceNamespace.SAGEMAKER,
        resource_id=f"endpoint/{endpoint_name}/variant/AllTraffic",
        scalable_dimension="sagemaker:variant:DesiredInstanceCount",
        min_capacity=0,
        max_capacity=1,
    )

    # Wait until the endpoint exists before attaching autoscaling.
    scalable_target.node.add_dependency(endpoint)

    endpoint_dimensions = {"EndpointName": endpoint_name}

    # Target tracking should follow queued async work, not invocation rate.
    backlog_per_instance_metric = cloudwatch.Metric(
        namespace="AWS/SageMaker",
        metric_name="ApproximateBacklogSizePerInstance",
        dimensions_map=endpoint_dimensions,
        statistic="Average",
        period=Duration.minutes(1),
    )

    scalable_target.scale_to_track_metric(
        "BacklogPerInstancePolicy",
        policy_name=f"heartsense-backlog-per-instance-{stage}",
        target_value=float(SAGEMAKER_BACKLOG_TARGET_PER_INSTANCE),
        custom_metric=backlog_per_instance_metric,
        scale_in_cooldown=Duration.minutes(SAGEMAKER_IDLE_TIMEOUT_MINUTES),
        scale_out_cooldown=Duration.seconds(SAGEMAKER_SCALE_OUT_COOLDOWN_SECONDS),
    )

    # Target tracking alone cannot wake a zero-instance endpoint.
    # HasBacklogWithoutCapacity becomes 1 when backlog > 0 and capacity == 0.
    scale_from_zero_action = appscaling.StepScalingAction(
        self,
        "ScaleFromZeroAction",
        scaling_target=scalable_target,
        policy_name=f"heartsense-scale-from-zero-{stage}",
        adjustment_type=appscaling.AdjustmentType.CHANGE_IN_CAPACITY,
        metric_aggregation_type=appscaling.MetricAggregationType.MAXIMUM,
        cooldown=Duration.seconds(SAGEMAKER_SCALE_OUT_COOLDOWN_SECONDS),
    )
    # +1 instance; max_capacity=1 keeps the endpoint at a single instance.
    scale_from_zero_action.add_adjustment(adjustment=1, lower_bound=0)

    has_backlog_without_capacity_alarm = cloudwatch.Alarm(
        self,
        "HasBacklogWithoutCapacityAlarm",
        alarm_name=f"heartsense-has-backlog-without-capacity-{stage}",
        alarm_description=(
            "SageMaker async inference has queued requests but zero instances — "
            "triggers the scale-from-zero step scaling policy."
        ),
        metric=cloudwatch.Metric(
            namespace="AWS/SageMaker",
            metric_name="HasBacklogWithoutCapacity",
            dimensions_map=endpoint_dimensions,
            statistic="Maximum",
            period=Duration.minutes(1),
        ),
        threshold=1,
        comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
        evaluation_periods=SAGEMAKER_SCALE_FROM_ZERO_EVALUATION_PERIODS,
        datapoints_to_alarm=SAGEMAKER_SCALE_FROM_ZERO_EVALUATION_PERIODS,
        treat_missing_data=cloudwatch.TreatMissingData.MISSING,
    )
    # Alarm fires → step scaling policy → endpoint scales from 0 to 1.
    has_backlog_without_capacity_alarm.add_alarm_action(
        cw_actions.ApplicationScalingAction(scale_from_zero_action)
    )

    return scalable_target

There is one very useful metric in CloudWatch for Sagemaker instances for target-tracking called ApproximateBacklogSizePerInstance , which represents the queued requests relative to the available endpoint capacity, and AWS recommends the usage for this metric for all autoscaling asynchronous endpoints. You can read about it in more detail by clicking on the link here.

There is one important scale-to-zero edge case when using this configuration. When the endpoint already has zero instances, a small backlog may not be enough for normal target-tracking policy to scale it out right away, and for that reason, I used already available metric HasBacklogWithoutCapacity which is a second scaling signal. The value of the metric becomes 1 when the requests are waiting, but there is no compute instances available to process the information, which triggers a step-scaling policy and it starts the endpoint from zero instances to one.

It’s important to mention that in this stack, the max number of provisioned SageMaker endpoints is going to be 1 — just to cut down on cost and have a POC ready.

When an upload arrives after a quiet period, SageMaker spins an instance back up, processes the queued request, and scales back to zero ~15 minutes after the last invocation. The async queue in front of the endpoint means callers never get a “no capacity” error — requests just wait for the instance to warm up.

The interesting part was not that the model saturated the instance. It did the opposite. During this test, CPU utilization stayed under ~2%, and memory utilization stayed around ~2.3%. It was surprising to me, as the ml.t3.medium instance is not a very strong instance and it’s only CPU dependent, so I expected a bigger impact on it’s hardware.

That is exactly why idle cost matters. If the endpoint is lightly used, the waste does not come from a single expensive inference. The waste comes from keeping model-serving compute warm while there are no requests. For this workload, scale-to-zero mattered more than squeezing every last percent out of the instance.

Another interesting metric which is available to see inside the console is the model latency metric and it looks like this:

The actual model execution was not the slow part. SageMaker model latency stayed roughly in the 30–37 ms range during this run.

That is an important distinction: model latency is not the same as user-visible pipeline latency. The end-to-end path also includes S3 upload, S3 event delivery, SQS batching, preprocessing, async endpoint queueing, result writing, result processing, and frontend polling.

Step 6 — Close the loop with an S3 event

When SageMaker finishes, it writes the result JSON to results/. Same as for WAV file upload, that is considered a S3 event, and the information about that event is put inside the SQS queue. When the queue gets some data, it triggers a small, plain-zip Lambda (no Docker needed here) that parses the prediction and finishes the record:

# infra/stacks/inference_stack.py
# Buffer SageMaker result objects so the Lambda can process them in batches.
self.result_processor_queue = sqs.Queue(
    self,
    "ResultProcessorQueue",
    queue_name=f"heartsense-result-processor-queue-{stage}",
    retention_period=Duration.days(SQS_RETENTION_DAYS),
    visibility_timeout=Duration.seconds(
        RESULT_QUEUE_VISIBILITY_TIMEOUT_SECONDS
    ),
    dead_letter_queue=sqs.DeadLetterQueue(
        max_receive_count=SQS_MAX_RECEIVE_COUNT,
        queue=self.result_processor_dlq,
    ),
)

# SageMaker writes to results/ → S3 notifies the result-processing SQS queue.
local_inference_output_bucket.add_event_notification(
    s3.EventType.OBJECT_CREATED,
    s3n.SqsDestination(self.result_processor_queue),
    s3.NotificationKeyFilter(prefix=RESULTS_PREFIX),
)

# Result Processor Lambda consumes the queue with partial batch failure reporting.
self.result_processor_function.add_event_source(
    lambda_event_sources.SqsEventSource(
        self.result_processor_queue,
        batch_size=SQS_BATCH_SIZE,
        max_batching_window=Duration.seconds(SQS_BATCH_WINDOW_SECONDS),
        report_batch_item_failures=True,
    )

Before the ResultProcessor Lambda, which has a SQS queue in-front of it - the primary objective of this queue is to absorb the burst of completed SageMaker predictions.

The result processor validates the payload and writes the final COMPLETED state — again with a conditional update, so a fast inference that beats preprocessing’s INFERRING write (or a duplicate S3 event) can never clobber the record:

# src/services/inference/result_processor.py

table.update_item(
    Key={"analysis_id": analysis_id},
    UpdateExpression=(
        "SET #status = :completed, #prediction = :prediction, "
        "#confidence = :confidence, #model_version = :model_version, "
        "#completed_at = :completed_at"
    ),
    ExpressionAttributeValues={
        ":completed": AnalysisStatus.COMPLETED.value,
        ":prediction": prediction,
        ":confidence": Decimal(str(confidence)),
        # ...
        ":inferring": AnalysisStatus.INFERRING.value,
        ":processing": AnalysisStatus.PROCESSING.value,
    },
    # only complete if we're mid-flight — idempotent against duplicates
    ConditionExpression="#status IN (:inferring, :processing)",
)

Here are the invocation statistics of the Results Processor Lambda:

This image shows the result processor is tiny:

• 145 invocations
• average duration ~257 ms
• max ~586 ms
• min ~44 ms

This Lambda is intentionally boring — it doesn’t generate audio features, load a model or do any heavy computation. It simply reads the SageMaker output JSON, validates that everything is alright with the payload and updates the DynamoDB table containing the results.

During multiple test runs, it handled ~140 invocations with an average duration ~260ms and a max duration of just under 600ms. That kind of work for a Lambda is excellent and excels at: short, event-driven glue code around managed services.

The frontend polls GET /analyses/{id} and flips from a spinner to the result the moment the record hits COMPLETED.

Failure handling, in one place

Because every stage is decoupled, failures are isolated instead of cascading:

Failure	What happens
Corrupt / invalid WAV	PermanentError → record marked FAILED, message consumed (no retry)
S3 throttle / 5xx	TransientError → SQS redelivers, up to 3×
SageMaker throttle	manual backoff (1s, 2s); if exhausted → FAILED
Poison message	after 3 receives → DLQ → CloudWatch alarm
Duplicate S3 event	idempotency table + conditional writes → no-op

What the metrics showed

After running a small batch through the pipeline, the metrics matched the architecture:

• Preprocessing was the heavier Lambda stage, averaging ~8.9s and peaking around ~27.6s.
• Result processing was tiny, averaging ~257ms.
• SageMaker model latency stayed around 30–37ms.
• Endpoint CPU and memory utilization stayed low, which made scale-to-zero more important than raw instance utilization.

That confirmed the main design decision: this was not one workload. It was several smaller workloads with different scaling profiles.

What this buys you

• The expensive model-serving compute scales to zero.
• Independent tuning. Preprocessing is 1 GB / 300s; inference has its own instance type and timeout. Neither blocks the other.
• Burst tolerance. SQS flattens spikes; the async queue means callers never see capacity errors.
• Self-healing. Transient errors retry, permanent errors fail fast, duplicates are absorbed, and poison messages park in a DLQ you can replay.

The whole thing is a few hundred lines of CDK. If you’ve been forcing ML into a single Lambda and watching it strain, this is the shape I’d reach for instead.