Originally published on kuryzhev.cloud
Your Python Lambda's p99 latency graph has a spike that looks random. It isn't. Pull up the REPORT log line in CloudWatch and check Init Duration — that's your lambda cold start latency, and it almost always traces back to one of a handful of fixable causes. We've chased this on a handful of production APIs, and the fixes below are what actually moved the needle, not what we assumed would.
Right-size memory before you touch anything else
Lambda allocates CPU proportionally to memory. Below roughly 1024MB, your function is often CPU-starved during init, and that shows up directly as slow cold starts — not just slow execution. Before rewriting any code, run AWS Lambda Power Tuning, a Step Functions state machine that benchmarks your function across memory sizes and gives you an actual cost/latency curve.
Watch out for the common overcorrection here: teams see cold starts, max out memory to 3GB "just to be safe," and end up paying for GB-seconds they didn't need. We did this on one function for three months before someone ran the power tuning report and found 1024MB was the sweet spot — 1536MB bought us nothing but a bigger bill.
Move imports and clients outside the handler
This is the fix that surprises people the most because it affects warm invocations too, not just cold ones. If you instantiate a boto3 client or open a DB connection inside the handler function, you're re-paying that setup cost on every single call — cold or warm.
Initialize SDK clients, DB connections, and parsed config at module scope so the execution environment reuses them across invocations. Heavy libraries you only need occasionally — pandas, numpy, an ML SDK — should be imported lazily inside the specific code path that needs them, not at the top of the file.
# --- BAD: re-created on every invocation, adds latency to warm calls too ---
def handler(event, context):
import boto3 # heavy import inside handler
s3 = boto3.client("s3") # new client every call
conn = create_db_connection() # new connection every call
return s3.get_object(Bucket="my-bucket", Key=event["key"])
# --- GOOD: init once at module scope, reused across warm invocations ---
import boto3
s3 = boto3.client("s3") # created once per execution environment
_db_conn = None
def get_db_connection():
global _db_conn
if _db_conn is None:
_db_conn = create_db_connection()
return _db_conn
def handler(event, context):
# heavy, rarely-used dependency imported only when actually needed
if event.get("needs_analytics"):
import pandas as pd # lazy import, skipped on most invocations
return process_with_pandas(pd, event)
conn = get_db_connection()
return s3.get_object(Bucket="my-bucket", Key=event["key"])
One more gotcha: warm environments persist between invocations, so don't cache long-lived secrets or credentials in global scope without expiry logic. Use the Secrets Manager caching client instead of hardcoding a token that quietly goes stale.
Trim the deployment package aggressively
Package size directly affects how fast Lambda unpacks and initializes your code. We've seen zips creep past 60MB unzipped just from stray test fixtures and docs that never got cleaned out of the build.
Strip test files, documentation, and __pycache__ directories from your deployment artifact. Use pip install --no-cache-dir --target with a minimal requirements file instead of dragging in a full framework's extras. For container-image functions, multi-stage Docker builds with slim base images cut both image size and cold start time — check the Lambda container image docs for the current base image tags.
Try SnapStart where it's available for Python
SnapStart restores a pre-initialized, cached execution environment instead of running your init code from scratch. For functions with a heavy import graph, this is often a bigger win than any code trimming, because you're skipping the init phase entirely rather than optimizing it.
Here's the gotcha that bites people: SnapStart takes a snapshot of your initialized environment, so anything non-deterministic set up at init time — random seeds, unique IDs, cached short-lived tokens — needs to be refreshed after restore, not baked into the snapshot. And SnapStart is mutually exclusive with Provisioned Concurrency per function, so this is a strategy choice, not an add-on.
Reserve Provisioned Concurrency for latency-critical, predictable traffic
Provisioned Concurrency guarantees warm environments, but you're billed per allocated concurrency-hour whether it's invoked or not. This is a cost decision as much as a performance one, and treating it as a free cold-start eraser is how bills quietly balloon.
Pair it with Application Auto Scaling schedules so concurrency ramps up before known traffic spikes — business-hours APIs, scheduled batch triggers — and scales down overnight. The mistake we made early on was applying it blanket across every function "just in case." Half of those functions got a handful of invocations a day; the Provisioned Concurrency cost dwarfed the actual compute cost.
# Example: scheduled scaling target for a business-hours API function
Resources:
ScalableTarget:
Type: AWS::ApplicationAutoScaling::ScalableTarget
Properties:
MinCapacity: 5
MaxCapacity: 5
ResourceId: !Sub "function:${FunctionName}:${FunctionAlias}"
ScalableDimension: lambda:function:ProvisionedConcurrency
ServiceNamespace: lambda
ScheduledActions:
- ScheduledActionName: scale-up-morning
Schedule: "cron(0 8 * * ? *)" # 8am UTC, before traffic ramps
ScalableTargetAction:
MinCapacity: 5
MaxCapacity: 5
- ScheduledActionName: scale-down-evening
Schedule: "cron(0 20 * * ? *)" # 8pm UTC, off-hours drop
ScalableTargetAction:
MinCapacity: 0
MaxCapacity: 0
Skip the VPC unless you truly need one
Hyperplane ENIs shrank the VPC cold-start penalty considerably, but VPC-attached functions still initialize slower than public ones — especially when outbound calls route through a NAT gateway. If you don't need RDS or ElastiCache directly, don't attach a VPC just because it feels "more secure."
If you do need one, add VPC endpoints for the AWS services you call — S3, DynamoDB, Secrets Manager — so traffic doesn't take a NAT round trip. And don't use "no VPC" as an excuse to skip least-privilege IAM; the network boundary disappearing doesn't mean the resource policy boundary should too.
Choose Graviton (ARM64) as the default architecture
Switching a Python function to arm64 is usually a one-line change in your IaC, and it's typically around 20% cheaper per GB-second while matching or beating x86 on init time for pure Python workloads. There's close to no reason not to default to it on new functions.
The gotcha is compiled dependencies. numpy, cryptography, and similar packages need arm64-compatible wheels, and if one's missing, the function won't fail at deploy — it'll fail silently at import time in production. Test the switch in a staging environment first, not directly on the function that's paging you at 2am.
# Terraform: switching architecture is a one-line change
resource "aws_lambda_function" "api" {
function_name = "orders-api"
architectures = ["arm64"] # was ["x86_64"] — verify compiled deps have arm64 wheels first
runtime = "python3.13"
memory_size = 1024
handler = "app.handler"
}
None of these fixes work as a substitute for measuring first. Keep-warm cron pings every 5 minutes are a common workaround we still see, but they don't guarantee the same execution environment gets reused once concurrency scales out — they're an unreliable, wasteful stand-in for actual Provisioned Concurrency. Use the decision checklist below to match the fix to your traffic pattern instead of applying all seven blindly.
Cold start mitigation decision checklist:
Traffic pattern -> Recommended strategy
------------------------------------------------------
Spiky, unpredictable -> Right-size memory + trim package + arm64
Predictable schedule (biz hrs)-> Provisioned Concurrency + Auto Scaling schedule
Heavy import graph, steady -> SnapStart (if supported) over Provisioned Concurrency
Rare/low-traffic internal job -> Do nothing extra; cold start cost is negligible
VPC-required (RDS/ElastiCache)-> VPC endpoints + connection reuse, avoid NAT hops
Compiled deps (numpy/crypto) -> Verify arm64 wheels before switching architecture
If you're debugging lambda cold start latency on a serverless API right now, start with Init Duration in CloudWatch, not the total Duration metric — it tells you exactly which phase is slow before you touch memory, packaging, or Provisioned Concurrency. For more serverless patterns and AWS troubleshooting notes, check the DevOps_DayS archive.
Top comments (0)