Postmortem: When AWS Lambda Versioning Was Not Enough
Summary
During a recent release, we ran into deployment and rollback challenges while relying on AWS Lambda versioning. While versioning worked well for changes limited to business logic, it broke down when changes involved Lambda layers and underlying architecture.
This postmortem documents what went wrong, why it happened, and how we fixed it.
Impact
- Increased deployment risk during releases involving dependency changes
- Inability to safely reuse existing Lambda versions
- Rollbacks became unreliable due to layer incompatibility
- Slower release cycles for architectural changes
No customer-facing outage occurred, but the risk of runtime failures increased.
What Changed
The release included changes to:
- Lambda layers containing shared libraries
- Dependency versions used across multiple functions
- Runtime behavior dependent on updated layer code
Although the function interface remained unchanged, the underlying execution environment did not.
Expected Behavior
We expected AWS Lambda versioning to:
- Support safe deployments
- Allow rollbacks using Lambda aliases
- Work for both code and dependency updates
This assumption proved incorrect.
Actual Behavior
AWS Lambda versioning creates immutable snapshots:
- Each published version is permanently bound to specific layer versions
- Published versions cannot be updated to reference new layer versions
- Lambda aliases cannot compensate for architectural incompatibilities
As a result:
- Updating layers required publishing new Lambda versions
- Rollbacks could fail due to dependency mismatches
- Versioning alone was insufficient for architecture-level changes
Root Cause
We treated AWS Lambda versioning as a full deployment lifecycle solution.
In reality, Lambda versioning is a code versioning mechanism, not an architecture versioning mechanism. The tight coupling between Lambda versions and Lambda layer versions was the core issue.
Resolution
We introduced a separate Lambda function to represent the new architecture:
- New function name
- New layer references
- Independent deployment and testing
- Gradual traffic migration via upstream services (API Gateway, EventBridge, SQS)
This approach restored deployment safety and rollback confidence.
Lessons Learned
- Lambda versioning works best for business logic changes
- Lambda layers introduce hidden coupling across versions
- Architectural changes require architectural isolation
- Rollbacks must consider dependency compatibility, not just code
Preventive Actions
- Use Lambda versions only for logic-level changes
- Create new Lambda functions for dependency or architecture changes
- Limit shared logic in Lambda layers where possible
- Add dependency impact analysis to release reviews
- Document decision criteria: version vs new function
Final Takeaway
Serverless does not remove architectural responsibility. Knowing when to version and when to isolate is critical for building reliable, production-grade Lambda systems.
If this post helped you, feel free to share or discuss your own Lambda versioning experiences below.
Top comments (1)
Good one martin