Feature flags have evolved far beyond simple on/off switches. At AWS re:Invent 2025, Steve Rice (General Manager, AWS AppConfig & Parameter Store) and Ben Shumpert (Senior SDE, AWS) demonstrated how Amazon leverages feature flags at scale to accelerate releases and dramatically reduce mean time to recovery (MTTR).
Their code-focused session DEV320 took attendees through real production examples, live coding demonstrations, and practical patterns that can be implemented immediately in any tech stack.
Watch the Full Session:
Why Feature Flags Are Your Secret Weapon
Steve opened with a powerful perspective: "Feature flags are the secret weapon of moving fast when you're developing software, as well as responding to issues on production."
The session used a creative approach with Steve role-playing as a coffee shop owner (Lambda Latte) and Ben as the engineer implementing features. This setup perfectly illustrated how feature flags bridge the gap between business requirements and technical implementation, especially when requirements change frequently.
Real Production Examples from Lambda Latte
The Lambda Latte demo application showcased three progressively sophisticated feature flag use cases:
Basic Feature Promotion
The first demo showed the simplest use case: promoting a seasonal product (Peppermint Candy Cane Latte) with a banner. But even this "simple" flag demonstrated the power of AWS AppConfig's attribute system.
Rather than just an on/off toggle, the flag included attributes for description, name, and price. When Steve wanted to change the promotion text, Ben simply updated the attribute values and redeployed through AppConfig. No code changes, no application redeployment. The banner updated instantly across the application.
Multi-Variant Flags for Customer Segmentation
The second demo elevated the complexity significantly. The business requirement: offer different rewards to customers based on their loyalty status (platinum, gold, or default).
This is where AWS AppConfig's multi-variant flags shine. Ben created three variants, each with:
- A rule defining which customers see that variant (based on loyalty status)
- Custom attribute values (different reward codes for each tier)
- Independent on/off toggles
The key technical insight: when requesting a multi-variant flag, you pass request context to the AppConfig agent. The agent evaluates the rules locally and returns the appropriate variant, with no additional network calls after the initial configuration fetch.
Ben's live coding showed how to pass this context via HTTP headers to the AppConfig agent. The result: platinum members saw four rewards, gold members saw two, and default members saw one, all driven by a single feature flag with intelligent segmentation.
Operational Flags for Logging Verbosity
The third demo addressed a critical operational use case: dynamically adjusting log verbosity during production incidents.
Steve explained the scenario perfectly: "You normally are gonna be running your logging at info level, but something might be going wrong, you're not sure, and you want to get a lot more information, so you can actually flip with a feature flag the verbosity from info to debug."
Ben demonstrated flipping the log level from INFO to DEBUG through AppConfig. Within seconds, the application console showed detailed debug logs without any code deployment or application restart.
AWS AppConfig's attribute constraints added an extra safety layer. The log level attribute used an enum constraint, preventing typos like "DEBG" or "DEBUG " (with a space) that could cause issues during a 3 AM incident response.
Architecture: How AppConfig Integrates
The Lambda Latte application architecture demonstrated AppConfig's flexibility:
Frontend: Lambda Latte web application Backend: AWS Lambda function Feature Flag Layer: AppConfig agent (as a Lambda extension/layer) Service: AWS AppConfig
The AppConfig agent acts as a local cache and rule evaluator, providing:
- High availability even during network issues
- Low latency through local evaluation
- Automatic background polling for configuration updates (default: every 45 seconds)
- Multiple deployment options: Lambda layers, container images, or RPM packages
Ben emphasized: "AppConfig works anywhere. We vend this agent in Lambda as a layer. However, we also vend the agent as a container image for ECS, EKS, or your own Kubernetes cluster, or just docker running on your laptop. We also vend the agent as an RPM for EC2 instances or on-prem instances."
Beyond Release Flags: Three Critical Use Cases
Steve outlined three major feature flag categories:
Release Flags
The traditional use case: hide new functionality behind a toggle, deploy the code, then gradually enable for 1%, 10%, 20% of traffic. If issues arise at scale, roll back instantly.
Steve shared that during AWS re:Invent announcement week, new capabilities are deployed weeks in advance but hidden behind feature flags. As executives announce features on stage, engineers simply flip flags to make them available. No risky deployments during keynotes.
Operational Flags
Respond to production issues immediately without code deployments:
- Block bad actors by account ID
- Throttle background processes during peak load
- Adjust recommendation counts (like amazon.com showing 3 instead of 7 product suggestions during Black Friday)
- Control logging verbosity for debugging
Peak Event Management
Steve revealed how amazon.com uses AppConfig during Black Friday: "You might only see three suggestions of things that you might wanna buy instead of the normal five to seven things. And you can imagine why that might be on a peak event. When people are using the website a lot, we wanna actually have less expensive queries to the backend."
Deployment Safety: The Critical Lesson
While Ben's demos used instant deployments for speed, Steve emphasized this is an anti-pattern for production: "Every change that you make, including a configuration change, could break your production environment. And if you think back to the last 18 months, a lot of the major outages of providers have been related to configuration changes that were deployed everywhere all at once."
AWS' internal policy: minimum 30-minute deployment and bake time for any configuration change. AppConfig supports gradual deployments with automatic rollback capabilities.
Steve referenced the 2017 S3 outage in US-EAST-1, caused by an extra semicolon in a configuration change: "That was before AppConfig existed, and that was part of the reason we launched AppConfig internally."
Feature Flag Lifecycle Management
The session addressed a common concern: feature flag sprawl. Ben demonstrated AppConfig's short-term flag feature, which allows setting a target deprecation date when creating a flag. Teams can then filter and sort by deprecation date to identify flags ready for cleanup.
However, the speakers emphasized that AppConfig won't automatically delete flags (too risky). Instead, it provides:
- Deprecation date tracking and filtering
- Deletion protection for actively used configurations
- Reminders to clean up overdue flags
The recommendation: treat feature flags like code. When a flag is no longer needed, delete it from AppConfig AND remove the branching logic from your codebase to avoid accumulating technical debt.
Advanced Features Demonstrated
Attribute Constraints: Enums and regular expressions prevent invalid values during updates, especially valuable during incident response.
Request Context: Pass arbitrary metadata (user type, loyalty status, region, etc.) to enable sophisticated segmentation rules.
Multi-Variant Rules: Support complex conditions with AND/OR logic, multiple operators, and compound rules for precise targeting.
Resiliency Layers: In-memory caching, optional on-disk backup (for non-Lambda environments), and graceful degradation during network issues.
Key Takeaways
Treat configuration like code: Configuration changes can be as dangerous as code changes. Deploy gradually and monitor carefully.
Start simple, scale complexity: Begin with basic flags, then adopt multi-variant flags as segmentation needs emerge.
Operational flags reduce MTTR: Pre-deploy operational controls (logging, throttling, feature toggles) to respond instantly to production issues.
Clean up aggressively: Set deprecation dates, track flag usage, and remove both the flag and the code when no longer needed.
Use the right tool: While homegrown solutions work initially, purpose-built services like AWS AppConfig provide critical safety features, deployment controls, and resiliency patterns.
Steve's closing advice captured the session's essence: "Feature flags are very seductive. People think it's no big deal just to flip a feature flag. If something goes wrong, I'm gonna flip it back. But you can see from how we have set up our service, they're very dangerous. 99.9% of the time, no problems. But when they go wrong, things can go really wrong."
About This Series
This post is part of DEV Track Spotlight, a series highlighting the incredible sessions from the AWS re:Invent 2025 Developer Community (DEV) track.
The DEV track featured 60 unique sessions delivered by 93 speakers from the AWS Community - including AWS Heroes, AWS Community Builders, and AWS User Group Leaders - alongside speakers from AWS and Amazon. These sessions covered cutting-edge topics including:
- 🤖 GenAI & Agentic AI - Multi-agent systems, Strands Agents SDK, Amazon Bedrock
- 🛠️ Developer Tools - Kiro, Kiro CLI, Amazon Q Developer, AI-driven development
- 🔒 Security - AI agent security, container security, automated remediation
- 🏗️ Infrastructure - Serverless, containers, edge computing, observability
- ⚡ Modernization - Legacy app transformation, CI/CD, feature flags
- 📊 Data - Amazon Aurora DSQL, real-time processing, vector databases
Each post in this series dives deep into one session, sharing key insights, practical takeaways, and links to the full recordings. Whether you attended re:Invent or are catching up remotely, these sessions represent the best of our developer community sharing real code, real demos, and real learnings.
Follow along as we spotlight these amazing sessions and celebrate the speakers who made the DEV track what it was!
Top comments (0)