Oleksandr Kuryzhev

Posted on Oct 24 Originally published at kuryzhev.cloud

Lambda Cold Start in Python: Diagnose and Cut Init Time

#aws #python #devops

Originally published on kuryzhev.cloud

An API Gateway endpoint backed by a Python function answers in 80 ms most of the time, then occasionally takes 3 seconds. The slow requests cluster after idle periods, after deployments, and during traffic bursts. That pattern is most often a Lambda cold start, and this runbook covers how to confirm it, find where the time goes, and fix it in order of increasing cost.

Symptoms: how a Lambda cold start shows up

Look for these signs before changing anything:

p50 latency is fine, but p99 or max latency is several times higher.
Slow requests follow idle gaps, deployments, configuration changes, or sudden traffic increases.
Clients behind short timeouts (API Gateway, Step Functions task timeouts, webhooks) fail intermittently while retries succeed.
The REPORT log line for slow invocations contains an Init Duration field. Warm invocations do not have it.

The last point is the reliable test. Use this CloudWatch Logs Insights query against the function's log group to measure how many invocations are cold and how slow they are:

filter @type = "REPORT"
| stats count(*) as invocations,
        count(@initDuration) as coldStarts,
        avg(@initDuration) as avgInitMs,
        pct(@initDuration, 95) as p95InitMs,
        pct(@duration, 99) as p99DurationMs
  by bin(1h)
# @initDuration exists only on cold invocations; @duration excludes init time

If you later enable SnapStart, restored environments report Restore Duration instead of Init Duration. This query does not count those restores, so extend it or parse @message to track them.

Watch out for: the cold-start share depends on the workload and is often small relative to total invocations. Average latency dashboards hide it. Always read percentiles, not means.

Root cause: where init time goes

A new execution environment goes through several steps before your handler runs: code download, environment start, runtime start, and your initialization code (everything at module top level). Only the last step is under your control, and in Python it is frequently the largest part. The documented lifecycle is in the Lambda execution environment documentation.

Typical contributors in Python workloads:

Heavy imports. Libraries such as pandas, numpy, scikit-learn, or large ORM stacks execute substantial code on import, even when a given request never uses them.
Eager clients and config. Creating boto3 clients, loading secrets, or opening database connections at module level adds work to every cold start, including network round trips.
Package size. Larger deployment packages and container images take longer to load into a new environment.
Low memory setting. CPU scales with memory, so a 128 MB function runs its imports on a fraction of a vCPU.

Two budgeting points matter here. First, work done in init adds directly to cold-start latency. Second, as of 2025 AWS bills the init phase for on-demand functions in all packaging types, so slow init costs money as well as time. Verify the current rules on the Lambda pricing page.

On-demand cold starts cannot be avoided as concurrency grows, because each concurrent request needs its own environment. The goal is to make them short, or to remove them from the request path.

Fix #1: Shrink the Lambda cold start with leaner init

This is the cheapest fix and should come first. Measure imports locally before guessing:

# Per-module import cost in microseconds, written to stderr
python -X importtime -c "import app" 2> imports.txt
# Show the most expensive cumulative imports (last column of each line is the module)
sort -t'|' -k2 -n -r imports.txt | head -15

Then apply the changes the results justify. Keep reusable clients at module level so warm invocations reuse them, but import rarely used heavy libraries inside the code path that needs them:

import io
import json
import boto3

# Created once per environment; reused across warm invocations
s3 = boto3.client("s3")

def handler(event, context):
    if event.get("report"):
        # Lazy import: only paid by requests that need it
        import pandas as pd
        df = pd.read_json(io.StringIO(event["report"]))
        return {"statusCode": 200, "body": json.dumps({"rows": len(df)})}
    return {"statusCode": 200, "body": json.dumps({"ok": True})}

Further steps in the same category:

Remove unused dependencies, and do not bundle test tooling. The Lambda Python runtime already includes boto3, though pinning your own version is a valid choice if you need it.
Test arm64 and memory sizes between 512 MB and 1769 MB. A higher setting often shortens init enough to offset its cost, but verify with your own measurements.
Defer secret and config fetches until first use, then cache them after the first call.

Watch out for: lazy imports move cost to the first request that uses them. If that request is also latency-sensitive, you have only relocated the delay.

Fix #2: Enable SnapStart for Python

If init remains slow after trimming, for example because of unavoidable large dependencies, use SnapStart. Lambda runs your init code once when you publish a version, snapshots the memory and disk state, and restores from that snapshot on a cold start. Python support was added for newer runtimes (Python 3.12 and later at launch). Check the SnapStart documentation for the current runtime and region list.

Resources:
  ApiFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.13
      Architectures: [arm64]
      MemorySize: 1024
      Handler: app.handler
      CodeUri: src/
      AutoPublishAlias: live        # SnapStart applies only to published versions
      SnapStart:
        ApplyOn: PublishedVersions  # not active on $LATEST

Constraints to confirm in the docs before rollout: SnapStart cannot be combined with provisioned concurrency, and it has restrictions around Amazon EFS and ephemeral storage larger than 512 MB. Unlike Java, Python SnapStart carries cache and restore charges, so check pricing.

Watch out for: state captured in the snapshot is reused across many environments. Random seeds, UUIDs generated at import time, cached credentials, and open network connections can be duplicated or stale after restore. Generate unique values inside the handler. Re-establish connections after restore, using runtime hooks if needed. Test this explicitly, because it is a frequent class of SnapStart defect.

Fix #3: Provisioned concurrency for hard latency targets

Some latency objectives cannot tolerate any cold start, such as a user-facing checkout call. For those, keep environments initialized in advance. Provisioned concurrency pre-warms a fixed number of environments for a published version or alias. Because it cannot be combined with SnapStart, disable SnapStart on the function before configuring it. See the provisioned concurrency documentation for billing and scaling behavior.

# Keep 5 initialized environments behind the "live" alias
aws lambda put-provisioned-concurrency-config \
  --function-name my-api-fn \
  --qualifier live \
  --provisioned-concurrent-executions 5

# Check status until it reports READY
aws lambda get-provisioned-concurrency-config \
  --function-name my-api-fn --qualifier live

Size the number from observed peak concurrency (the ConcurrentExecutions metric), not from guesswork. Use Application Auto Scaling scheduled actions for predictable daily peaks. For variable load, use target tracking on provisioned concurrency utilization.

Watch out for: provisioned concurrency applies only to the version or alias you configure.

Invoking $LATEST bypasses it, and the cold starts return.
If you configure a specific version number and then route traffic to a newer version, that traffic also bypasses it.
Configuring the alias avoids this. When the alias moves, Lambda provisions environments for the new version, which takes time to reach READY.
Traffic above the provisioned number spills over to on-demand environments, which still start cold.

Prevention

Treat init time as a budgeted metric rather than something investigated only during incidents.

Alert on init, not just duration. Build a dashboard or alarm from @initDuration percentiles using the query above, and compare after each release.
Gate dependencies. Fail CI when the deployment package grows past an agreed size, or when python -X importtime shows a new top-level import above a threshold.
Keep init deterministic. Make no network calls to unstable dependencies at import time. A slow or failing call there turns every cold start into an incident.
Split functions by weight. Put heavy data-processing code in separate functions from latency-sensitive API handlers, so the API path never imports pandas.
Choose the cheapest fix that meets the target. Leaner init comes first. SnapStart comes second. Provisioned concurrency comes last, and it replaces SnapStart rather than stacking with it. Re-measure after each step, and stop when the p99 target is met.
Re-check after runtime upgrades. Init behavior, SnapStart eligibility, and pricing rules change between runtime versions and over time. Verify against current AWS documentation when you move to a new Python runtime.

For more infrastructure and serverless runbooks, browse the rest of kuryzhev.cloud.

DEV Community