DEV Community: Håkon Eriksen Drange

AWS Summit Stockholm 2026 talk: Using Amazon Bedrock for creative Christmas spirit ideas

Håkon Eriksen Drange — Fri, 08 May 2026 12:57:38 +0000

On May 7th 2026 I got the opportunity to hold a talk at the Developer Community Zone at the AWS Summit in Stockholm, where I shared my learnings developing the creative Christmas Elf project as described in blog post https://hedrange.com/2026/03/07/generative-ai-for-creativity-lessons-learned-from-building-a-christmas-elf-idea-platform-with-amazon-bedrock/.

The audience was very engaged and raised many good questions after the presentation. The slides are available for download below.

AWS-Summit-Stockholm-2026-COM302-English Download

The post AWS Summit Stockholm 2026 talk: Using Amazon Bedrock for creative Christmas spirit ideas first appeared on Håkon Eriksen Drange.

Bytefest developer conference talk: From Cowboy Code to Production Gold with Kiro

Håkon Eriksen Drange — Sat, 28 Mar 2026 17:38:52 +0000

On Thursday March 17th I held yet another talk about Spec-Driven development, this time at Sopra Steria’s Bytefest developer conference. As the capabilities in this field are evolving fasted I showed a practical demonstration on how to go from solo mode to full team visibility by leveraging the Github MCP server to automagically create and update Github issues for spec refinement and task execution. This ensures ongoing work is visible on a Kanban board, for full transparency to everyone involved, from developers to stakeholders.

Abstract

Vibe coding can be very useful for prototyping, but do you trust it for production? Get an introduction to the AI-assisted software development lifecycle and how you can use Kiro to ship your product using professional craftmanship practices.

Official program directory: https://bytefest.soprasteria.no/bytefest-med-venner/talks/1118863

Github repo with code and examples for the demonstrated weather forecast app: https://github.com/haakond/terraform-aws-weather-forecast

Slides

2026-03-19-bytefest-from-cowboy-code-to-production-gold-with-kiro Download

The post Bytefest developer conference talk: From Cowboy Code to Production Gold with Kiro first appeared on Håkon Eriksen Drange.

Generative AI for creativity: Lessons learned from building a Christmas Elf idea platform with Amazon Bedrock

Håkon Eriksen Drange — Sat, 07 Mar 2026 18:05:54 +0000

Many Generative and Agentic AI blogs these days are about document processing and business workflow automation. This one is about elves. Read on to learn more about how I built this platform and the lessons that I learned.

Introduction
The unique value proposition of Naughty Elf Ideas
Solution architecture of the Naughty Elf Ideas platform
Idea generation flow walk-through
- Idea generation orchestration with AWS Step Functions
Video generation flow walkthrough
Choosing the right tool/model for each job
Looking into the creative idea generation process
Lessons learned
- Lesson 1: Construct prompts according to model provider guidance
- Lesson 2: Prompt engineering alone wasn't enough, RAG changed everything
- Lesson 3: LLMs doesn't follow "don't do X" – Use positive context instead
- Lesson 4: Multilingual translation challenges (Norwegian and Swedish)
- Lesson 5: AWS Step Functions vs. Amazon Bedrock Agents – Why I chose deterministic orchestration
- Comparison of orchestration workflow alternatives
Conclusion
- Key takeaways
- Future enhancements

Introduction

With the increasingly popular tradition with the Elf on the Shelf (Naughty/Mischevious Elf, or “Rampenissen” in Norwegian), as December approaches, millions of parents all over the world face a similar challenge: what creative scenario should the Christmas elf get into tonight, to surprise their children? After a few years of the tradition, you’ve exhausted the classics; elf in flour mess, elf making snow angels in powdered sugar, elf zip-lining across the living room and toilet paper all over. You find yourself doom-scrolling Facebook groups and Pinterest until midnight, desperately searching for something your kids haven’t seen before.

At Sopra Steria, we help customers optimize processes and achieve business outcomes with technology. But in this case, I wanted to explore a more creative side – what would be possible when you push the boundaries of generative AI for inspiration rather than automation. This is my take on an innovation sandbox.

To solve this creative idea problem (primarily for my wife, but other parents could also probably profit ;-), just before AWS re:Invent 2025 I got the idea to dive deep into Generative and Agentic AI capabilities with Amazon Bedrock. I figure that a really useful tool would not be a static database of recycled ideas, but a complete platform that generates unlimited unique elf scenarios on demand — tailored to kids’ age, rooms in your house, and the mood you’re after (naughty, nice, nostalgia/pop culture). The result is https://naughtyelfideas.com/.

The unique value proposition of Naughty Elf Ideas

While traditional blog posts usually cover 20-80 recycled ideas, the Naughty Elf Ideas platform provides unlimited creative AI-generated ideas, based on the latest and most innovative Large Language Models (LLMs) from providers such as Anthropic and Amazon.

Traditional blogs have 0 personalization. This platform generates ideas according to your preferences for age group, room, mood, and it even accepts custom prompts. The site and ideas are tailored for local culture and language currently in English, Spanish, Norwegian and Swedish, with more to come!

Regular sites uses common stock photos or poorly lit cell phone snapshots. Here we generate unique images and video snippets of each idea.

Freshness is also on a whole new level, since new ideas are added by the community continuously. Users can vote for and rank the best ideas and share them in social media.

Solution architecture of the Naughty Elf Ideas platform

This is a real, live site which naturally receives the majority of it’s traffic in the months of late November and December. After Christmas, there’s no use for it. That’s why I chose pure serverless components from Amazon Web Services, to be able to scale according to demand. The remaining time of the year the baseline running costs are minimal.

Key components

Fully serverless – no servers or databases to manage, scales automatically
AWS Step Functions provides visual workflow orchestration with built-in error handling
Multiple Amazon Bedrock models selected for specific tasks (more on this below)
Amazon DynamoDB for idea storage with single-table design
Amazon S3 + Amazon CloudFront for static assets (images, videos) with cache invalidation
Next.js on AWS Amplify for SEO-optimized frontend with SSR/SSG

Overview of AWS architecture components

Idea generation flow walk-through

This is how the idea generation dialogue looks like:

Excited users are not patient, so a challenge has been to optimize end-to-end processing to reduce waiting times to a minimum. Websockets dialogue mimics interaction during processing. The current total end-to-end processing time is about 30 seconds from request to complete idea, including translations to multiple languages and unique image generation. Parallel branches for translation and image generation reduces overall time-to-idea.

Idea generation orchestration with AWS Step Functions

One of the key architectural decisions was to identify the optimal workflow orchestration. I started out with Bedrock Agents, but quickly discovered it took well over 60-80 seconds every time. The AI reasoning overhead added latency at each decision point (which also incurred cost).

For a well-defined and repeatable workflow, there’s actually no need for AI Agent reasoning. The fastest and most cost efficient approach turned out to be a good, old State Machine. In addition, visual insight and debugging with log integrations in AWS Step Functions proved invaluable for troubleshooting.

Recommendation : Use Step Functions when the workflow is well-defined and latency matters; consider Bedrock Agents when you need dynamic reasoning and tool selection.

End user clicks Generate idea along with preferred properties
A POST /generate-idea call is sent to Amazon API Gateway which starts execution of an AWS Step Functions State Machine for Idea Generation
To avoid bankrupting me in case the site goes really viral, I’ve set a cap on generation of max 100 ideas per day.
If a custom prompt was provided in another language than English, translate it to English, which is the primary language of the system.
Lambda function GenerateCreativeIdea invokes an Amazon Bedrock Agent wired up with a Bedrock Knowledge Base, to provide relevant context of historical ideas.
- I found the most creative ideas were provided by the latest version of Anthropic Sonnet.
- An Amazon Bedrock Guardrail is applied to ensure family friendly and safe ideas.
When we have an original idea we enter a ParallelProcessing state.
- Translations to Norwegian, Spanish and Swedish – two-pass using Anthropic Haiku.
- Image generation with Amazon Nova Canvas based on idea description.
Upon successful results the generated idea is stored in Amazon DynamoDB along with relevant metadata.
The daily idea generator counter is incremented and relevant website pages are invalidated from Amazon Cloudfront.
The final step is to put a message on an Amazon Simple Queue System (SQS) queue to generate a video snippet. This takes 2-3 minutes, and I didn’t want to keep the user waiting, so this is the reason for the decoupled, asynchronous process.

Video generation flow walkthrough

Same approach here; repeatable process => AWS Step Functions.

The Amazon SQS queue which receives the message from the Idea Generation State Machine is configured with an AWS Lambda trigger which invokes the Video Generation State Machine.
The AWS Lambda function GenerateVideo invokes Amazon Bedrock for video generation with Amazon Nova Reel.
- The original image is supplied for relevance.
- Raw video output is stored in Amazon S3.
AWS Step Function logic waits and polls for video generation success or failure.
When video generation completes, another Docker based AWS Lambda function uses FFmpeg to transcode the raw video file from S3 for web and file size optimization (for fast loading and data transfer cost control. An optimized version is saved back to Amazon S3 in a path configured as Amazon Cloudfront origin.
The original idea is updated with metadata that a video is available.
The main idea web page is invalidated from Amazon Cloudfront and the next page load displays a video overlay instead of the original image.

End-to-end this takes about 2 minutes and 25 seconds for a 6 second clip (current limitation in Amazon Nova Reel). Videos are progressive enhancements, they do not block the main idea flow.

Choosing the right tool/model for each job

To be able to get the results you are looking for it’s imperative to evaluate and benchmark relevant LLMs for different purposes. I ended up with the following:

Task	Model selected	Rationale	Alternatives considered
Creative idea generation	Anthropic Claude Sonnet 4.6	Most creative, contextually appropriate ideas	Claude Haiku (less creative), Nova Pro (good but less nuanced)
Norwegian and Swedish translation	Anthropic Claude Haiku 4.5 (two-pass)	Natural-sounding Norwegian/Swedish via two-pass approach	Nova models (did not produce acceptable results, suspect low training on these languages), Amazon Translate (too literal)
Spanish translation	Anthropic Claude Haiku 4.5 (two-pass)	Consistent quality across languages	Nova 2 Lite (acceptable, but standardized on Haiku as this was required for NO/SE)
Image generation	Amazon Nova Canvas	Cost-effective, good quality, AWS-native	Stability AI (higher quality, higher cost)
Video generation	Amazon Nova Reel	Only AWS-native option, 6-second clips (without complex stitching)	None (unique capability)

Anthropic Claude Sonnet for ideas : Tested multiple models – Sonnet consistently produced the most creative, contextually appropriate elf ideas

Claude Haiku for Norwegian and Swedish : This was the hardest problem. Amazon Nova models were not acceptable out of the box for Scandinavian languages, and that’s not a surprise, taking into account the small population compared to the world’s major languages. Training on material from these languages have probably not been prioritized. Amazon Translate was even worse (too literal). The solution: a two-pass approach with Claude Haiku that first translates, then refines for natural flow, using Anthropic’s prompt guidelines. Low temperature (0.1) provides consistent translations. I discovered that higher temperature lead to creative interpretations and invention of new words :-).

Amazon Nova Canvas for images : Good balance of quality and cost. Stability AI was said to produce very high quality, but at 3x the cost, I went with Nova Canvas. I am looking forward to full Nova V2 suite availability.

Amazon Nova Reel for videos : The only AWS-native option for video generation. 6-second clips are perfect for social media sharing and preserving bandwidth charges.

Key take-away : Model selection should be validated through experimentation for each use case — don’t assume based on benchmarks. As new minor versions are released from the providers, expect upgrades in quality and relevance.

Looking into the creative idea generation process

Each idea on the site is generated on demand by an Amazon Bedrock Agent backed by an Amazon Bedrock Knowledge Base of existing elf ideas. When a request comes in, the agent doesn’t just freestyle, it receives a structured prompt that includes the target age group, the mood of the idea (naughty, nice, or popular culture/meme/nostalgia inspired), the desired room (optional) and a feed of the 20 most recently generated titles. This last part is key: rather than telling the model “don’t repeat yourself,” we hand it positive context; “here’s what already exists”, which language models handle far more reliably than negation constraints.

The Amazon Bedrock Knowledge Base grounds the output in proven, family-friendly formats, while the session attributes steer it toward something fresh and specific. The result is ideas that feel handpicked and personalized, rather than randomly generated.

Sequence diagram for the creative idea generation process

Let’s take a closer look at the Lambda function for idea creation. As it is hundreds of line of code, I’ve included a simplified pseudo-code example below to illustrate my approach.

<textarea tabindex="-1" aria-hidden="true" readonly>function generate_creative_idea(language, custom_prompt, parameters, attempt):

  1. READ agent config from env vars (AGENT_ID, AGENT_ALIAS_ID)
     → raise error if missing

  2. RESOLVE parameters (category, mood, room)
     → randomly pick category from [toddlers, preschoolers, school-age, tweens] if not given
     → randomly pick mood from [naughty, nice, pop] if not given
     → map "naughty" → "mischievous" (safe alias to avoid content filter)

  3. BUILD session attributes dict
     → always include: language, customPrompt, attempt
     → conditionally include: category, mood (safe alias), room

  4. FETCH last 20 idea titles from DB
     → add as "recentIdeaTitles" to session attributes (positive context, not a prohibition)

  5. RETRY LOOP (up to 3 attempts, exponential backoff):

     a. BUILD input prompt via build_agent_input()
        → adjusts prompt wording if previous attempt was content-blocked

     b. INVOKE Bedrock Agent (streaming)
        → pass session attributes + input text

        # Actual Python snippet below

        # Get agent configuration from environment
        agent_id = os.environ.get('AGENT_ID')
        agent_alias_id = os.environ.get('AGENT_ALIAS_ID')
        knowledge_base_id = os.environ.get('KNOWLEDGE_BASE_ID')

        if not agent_id or not agent_alias_id:
            raise ValueError("Agent not configured: AGENT_ID and AGENT_ALIAS_ID environment variables required")

        # Create Bedrock Agent Runtime client
        bedrock_agent_runtime = boto3.client('bedrock-agent-runtime', region_name='eu-central-1')

        # Retry configuration with exponential backoff
        max_retries = 3
        base_delay = 2 # seconds

        # Track if previous attempt was content-blocked (to adjust prompt on retry)
        content_blocked_retry = False

        for retry_attempt in range(max_retries):
            try:
                # Rebuild input text each attempt — adjusts prompt if previous was blocked
                input_text = build_agent_input(
                    category=category,
                    mood=safe_mood,
                    language=language,
                    custom_prompt=custom_prompt,
                    room=room,
                    attempt=attempt,
                    content_blocked=content_blocked_retry,
                    recent_titles=recent_titles
                )

                agent_start = time.time()

                # Invoke agent with streaming response
                response = bedrock_agent_runtime.invoke_agent(
                    agentId=agent_id,
                    agentAliasId=agent_alias_id,
                    sessionId=str(uuid.uuid4()),
                    sessionState={
                        'sessionAttributes': session_attributes
                    },
                    inputText=input_text
                )

                # Parse streaming response
                idea_data = parse_agent_response(response)

     c. PARSE streaming response → idea_data (title, description, metadata)

     d. PUBLISH metrics (agent latency, KB citations)

     e. ON SUCCESS → break loop

     f. ON ContentBlockedError
        → if retries remain: set content_blocked_retry=True, wait, retry
        → else: raise

     g. ON ThrottlingException
        → if retries remain: wait, retry
        → else: raise

     h. ON ResourceNotFoundException / AccessDeniedException
        → raise immediately (no retry)

     i. ON generic Exception
        → if retries remain: wait, retry
        → else: raise

  6. WARN if total generation time &gt; 45s (near Lambda timeout)

  7. IF custom_prompt given:
     → score how well the output incorporated it
     → warn if score &lt; 0.3

  8. BUILD result dict:
     → title, description
     → metadata: category, mood (mapped back from safe alias), room, materials_type,
                  time_available, custom_prompt_score, kb_used flag
     → generation_time_ms

  9. RETURN result</textarea>

function generate_creative_idea(language, custom_prompt, parameters, attempt):

  1. READ agent config from env vars (AGENT_ID, AGENT_ALIAS_ID)
     → raise error if missing

  2. RESOLVE parameters (category, mood, room)
     → randomly pick category from [toddlers, preschoolers, school-age, tweens] if not given
     → randomly pick mood from [naughty, nice, pop] if not given
     → map "naughty" → "mischievous" (safe alias to avoid content filter)

  3. BUILD session attributes dict
     → always include: language, customPrompt, attempt
     → conditionally include: category, mood (safe alias), room

  4. FETCH last 20 idea titles from DB
     → add as "recentIdeaTitles" to session attributes (positive context, not a prohibition)

  5. RETRY LOOP (up to 3 attempts, exponential backoff):

     a. BUILD input prompt via build_agent_input()
        → adjusts prompt wording if previous attempt was content-blocked

     b. INVOKE Bedrock Agent (streaming)
        → pass session attributes + input text

        # Actual Python snippet below

        # Get agent configuration from environment
        agent_id = os.environ.get('AGENT_ID')
        agent_alias_id = os.environ.get('AGENT_ALIAS_ID')
        knowledge_base_id = os.environ.get('KNOWLEDGE_BASE_ID')

        if not agent_id or not agent_alias_id:
            raise ValueError("Agent not configured: AGENT_ID and AGENT_ALIAS_ID environment variables required")

        # Create Bedrock Agent Runtime client
        bedrock_agent_runtime = boto3.client('bedrock-agent-runtime', region_name='eu-central-1')

        # Retry configuration with exponential backoff
        max_retries = 3
        base_delay = 2 # seconds

        # Track if previous attempt was content-blocked (to adjust prompt on retry)
        content_blocked_retry = False

        for retry_attempt in range(max_retries):
            try:
                # Rebuild input text each attempt — adjusts prompt if previous was blocked
                input_text = build_agent_input(
                    category=category,
                    mood=safe_mood,
                    language=language,
                    custom_prompt=custom_prompt,
                    room=room,
                    attempt=attempt,
                    content_blocked=content_blocked_retry,
                    recent_titles=recent_titles
                )

                agent_start = time.time()

                # Invoke agent with streaming response
                response = bedrock_agent_runtime.invoke_agent(
                    agentId=agent_id,
                    agentAliasId=agent_alias_id,
                    sessionId=str(uuid.uuid4()),
                    sessionState={
                        'sessionAttributes': session_attributes
                    },
                    inputText=input_text
                )

                # Parse streaming response
                idea_data = parse_agent_response(response)

     c. PARSE streaming response → idea_data (title, description, metadata)

     d. PUBLISH metrics (agent latency, KB citations)

     e. ON SUCCESS → break loop

     f. ON ContentBlockedError
        → if retries remain: set content_blocked_retry=True, wait, retry
        → else: raise

     g. ON ThrottlingException
        → if retries remain: wait, retry
        → else: raise

     h. ON ResourceNotFoundException / AccessDeniedException
        → raise immediately (no retry)

     i. ON generic Exception
        → if retries remain: wait, retry
        → else: raise

  6. WARN if total generation time > 45s (near Lambda timeout)

  7. IF custom_prompt given:
     → score how well the output incorporated it
     → warn if score < 0.3

  8. BUILD result dict:
     → title, description
     → metadata: category, mood (mapped back from safe alias), room, materials_type,
                  time_available, custom_prompt_score, kb_used flag
     → generation_time_ms

  9. RETURN result

Infrastructure-as-Code for the Amazon Bedrock Agent implementation is handled with AWS Cloud Development Kit in Typescript, with specific and targeted agent instructions for the use-case. Tuning the prompts along with the Knowledge Base context was where I spent a lot of time to ensure relevant responses.

As there are many things that can go wrong in this process, content blocking by Amazon Bedrock Guardrails or safety filter in the language models themselves and Amazon Bedrock capacity constraints in certain regions, it is vital to remember the Well-Architected principle of REL05-BP03 Control and limit retry calls and implement proper retry and error handling.

Below is an example of the implemented logic for idea generation.

Lessons learned

As the concept of the Naughty/Mischievous Elf is creative, but with very specific boundaries about what’s acceptable and not, also taking into account cultural differences across countries, it was challenging to get the platform to produce relevant results in the beginning. As with most projects involving Artificial Intelligence, scoping, context and iterative testing and validation makes all the difference. Below I share my top five lessons learned.

Lesson 1: Construct prompts according to model provider guidance

Different models may expect prompts to be designed and structured in a particular way. Amazon Nova models and Anthropic Claude models expect input in different formats for optimal and relevant results. To get the results you’re after, check the model provider’s guidance. Keep the prompts version controlled so that you know what changed when results suddenly deviate. Test and benchmark until you’re happy. Amazon Bedrock Prompt Management may also be something to consider for more static prompts.

For Amazon Nova models see User Guide: Creating precise prompts. For Anthropic Claude models such as Opus, Sonnet or Haiku see Claude API Docs: Prompting best practices.

Lesson 2: Prompt engineering alone wasn’t enough, RAG changed everything

Problem : Initial prompts produced generic, uninspired elf ideas. Despite adding constraints, few-shot examples, and negative prompts, the process kept generating the same 3-5 ideas repeatedly. Most ideas agitated towards spillage, flour mess, toilet paper and so on.

What didn’t work :

Adding more constraints to the prompt logic.
Including few-shot examples of “good” ideas.
Negative constraints (“avoid messy setups”, “no flour ideas”) – This amplified it and led to even more of them!
Temperature adjustments (higher = more random, not more creative).

What actually worked: Retrieval Augmented Generation :

To help the model with more context I sourced ~150 unique ideas from other elf idea blogs and Pinterest boards and set up an Amazon Bedrock Knowledge Base with these ideas as inspiration (not to copy, but to provide a more comprehensive understanding of the creative space).

It had to be cost efficient, so Amazon OpenSearch was ruled out in favour of Amazon S3 Vectors. The agent now retrieves relevant examples before generating, dramatically improving variety and relevancy.

The feedback loop :

In the final part of the orchestration flow newly generated ideas are added back to the knowledge base, which creates a growing corpus of unique ideas. The more ideas generated, the more diverse future generations become. After 150+ ideas in the library, repetition dropped from ~30% to <5%.

Outcome : When I provided relevant Perceive context with Retrieval Augment Generation (RAG) and Amazon Bedrock Knowledge Bases, RAG expanded the creative space. This was a major breakthrough.

The language models themselves are supposed to be generic, based on the material they’re trained on. Prompt engineering on specialized or generic language models will only take you so far. My approach is to use the language models for reasoning, not the actual content (which also has a cut-off date after the model is trained and published). This is what most people don’t get. RAG is also faster and more cost-efficient than fine-tuning a model.

To illustrate the concept of Perceive, Reason and Act, the AWS Prescriptive Guidance: Foundations of agentic AI on AWS explains:

“At the core of every software agent is a cognitive cycle that is often described as the perceive, reason, act loop. This process is illustrated in the following diagram. It defines how agents operate autonomously in dynamic environments.”.

_ Perceive : Agents gather information (for example, events, sensor inputs, or API signals) from the environment and update their internal state or beliefs._
_ Reason : Agents analyze current beliefs, goals, and contextual knowledge by using a plan library or logic system. This process might involve goal prioritization, conflict resolution, or intention selection._
_ Act : Agents select and execute actions that move them closer to achieving their delegated goals._

Reference: https://docs.aws.amazon.com/prescriptive-guidance/latest/agentic-ai-foundations/perceive-reason-act.html

AWS Prescriptive Guidance: Generative AI agents: replacing symbolic logic with LLMs

“The following diagram illustrates how large language models (LLMs) now serve as a flexible and intelligent cognitive core for software agents. In contrast to traditional symbolic logic systems, which rely on static plan libraries and hand-coded rules, LLMs enable adaptive reasoning, contextual planning, and dynamic tool use, which transform how agents perceive, reason, and act.”

Reference: https://docs.aws.amazon.com/prescriptive-guidance/latest/agentic-ai-foundations/generative-ai-agents.html

Lesson 3: LLMs doesn’t follow “don’t do X” – Use positive context instead

Problem : After a few hundred ideas, the system started generating duplicates. Two ideas generated 32 minutes apart were essentially the same concept — “elf swaps everyone’s phone contact names” — despite the Amazon Bedrock Knowledge Base similarity check that was supposed to catch this.

Why the Knowledge Base similarity check wasn’t enough : The check fired after generation. By then, the Amazon Bedrock call had already been made and billed. Worse, the first generated idea got added back to the Amazon Bedrock Knowledge Base, so the second generation retrieves it as a reference, and the model riffs on it rather than diverging from it.

The instinct and why it doesn’t work : The obvious fix is to pass recent idea titles to the agent with a “don’t repeat these” instruction. This is a well-documented failure mode: LLMs are poor at negation constraints. “Don’t generate X” makes X more salient in the model’s attention, not less. The model acknowledges the constraint, then gravitates toward the same concepts anyway.

What actually works: positive context framing : According to Claude API Docs: Add context to improve performance: Providing context or motivation behind your instructions, such as explaining to Claude why such behavior is important, can help Claude better understand your goals and deliver more targeted responses.

_ Less effective: NEVER use ellipses_

_ More effective: Your response will be read aloud by a text-to-speech engine, so never use ellipses since the text-to-speech engine will not know how to pronounce them._

Taking this into consideration, instead of “avoid these ideas”, I framed it as “here’s what already exists, explore a different domain”:

<textarea tabindex="-1" aria-hidden="true" readonly># <img src="https://s.w.org/images/core/emoji/17.0.2/72x72/274c.png" alt="❌" style="height: 1em; max-height: 1em;"> Negation framing — doesn't work reliably
parts.append("Do NOT generate ideas similar to these recent ones:")
for title in recent_titles:
    parts.append(f" - {title}")

# <img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2705.png" alt="✅" style="height: 1em; max-height: 1em;"> Positive context framing — works with how LLMs process context
parts.append("STEP 3: These ideas have been generated recently — create something in a DIFFERENT domain:")
for title in recent_titles[:10]:
    parts.append(f" - {title}")
</textarea>

# ❌ Negation framing — doesn't work reliably
parts.append("Do NOT generate ideas similar to these recent ones:")
for title in recent_titles:
    parts.append(f" - {title}")

# ✅ Positive context framing — works with how LLMs process context
parts.append("STEP 3: These ideas have been generated recently — create something in a DIFFERENT domain:")
for title in recent_titles[:10]:
    parts.append(f" - {title}")

The model uses the list as context about what’s already covered and naturally diverges. It’s the same information, but framed as situational awareness rather than a prohibition.

The implementation : Before invoking the agent, fetch_recent_idea_titles() queries DynamoDB’s StatusIndex for the 20 most recent active ideas (newest first, projection on translations only for efficiency). The titles are joined as a pipe-separated string in sessionAttributes['recentIdeaTitles'] and also injected into the prompt as step 3. Fails are open, a DynamoDB error never blocks generation.

Defense in depth : The pre-generation context reduces the probability of duplicates. The post-generation check_similar_ideas() in catches anything that slips through, and the Step Function retries with attempt=2.

Key insight : LLMs process context, not rules. “Here’s what exists” is context. “Don’t do X” is a rule. Context wins.

Lesson 4: Multilingual translation challenges (Norwegian and Swedish)

Problem : Spanish translations were okay, but Norwegian and Swedish translations were awkward and unnatural. Results with Amazon Nova Pro/Lite models were not acceptable out of the box. Amazon Translate was even worse – too literal, missing cultural nuance.

Approach : I tested multiple models: Amazon Nova Pro, Amazon Nova 2 Lite, Amazon Translate, Anthropic Claude Haiku and Anthropic Claude Sonnet.

From the AWS AI Service Card overview on Amazon Nova Models – Language:

Amazon Nova models are released as generally available for English, German, French, Spanish, Italian, Portuguese, Japanese, Hindi, and Arabic. Amazon Nova model’s guardrails are intended for use in the aforementioned languages only, however Amazon Nova models are trained and will attempt to provide completions in up to 200 languages. In use cases beyond the languages identified above, customers should carefully check completions for effectiveness, including safety.

Reference: https://docs.aws.amazon.com/ai/responsible-ai/nova-micro-lite-pro/overview.html

From Anthropic Claude API Docs: Multilingual support:

Claude excels at tasks across multiple languages, maintaining strong cross-lingual performance relative to English.

Claude demonstrates robust multilingual capabilities, with particularly strong performance in zero-shot tasks across languages. The model maintains consistent relative performance across both widely-spoken and lower-resource languages, making it a reliable choice for multilingual applications.

Reference: https://platform.claude.com/docs/en/build-with-claude/multilingual-support

A two-pass approach with Claude Haiku produced the best results at a reasonable price point:

First pass: Direct translation
Second pass: “Refine this Norwegian text to sound natural, as if written by a native speaker”

Outcome : Natural Norwegian and Swedish like a native speaker would write it. Two-pass added ~2-3 seconds of latency, but it was definitely worth it. Claude’s broader multilingual training data made the difference, and the most lightweight option with Haiku did the trick, at the best performance and price.

Lesson 5: AWS Step Functions vs. Amazon Bedrock Agents – Why I chose deterministic orchestration

Problem : The first version of the idea generation workflow was a mess. Too many fine-grained states, no parallel processing, and a failed experiment with Amazon Bedrock Agents that added 40-70 seconds of latency I couldn’t explain at first.

First iteration: The Amazon Bedrock Agent experiment

In the initial iteration the orchestration layer was based on Amazon Bedrock Agents. The appeal was obvious; natural language instructions, dynamic tool selection, no explicit state machine to maintain. In theory, you describe what you want and the agent figures out the steps.

In practice, it was the wrong tool for this job. The agent’s reasoning loop – deciding which tool to call, in what order, with what parameters – added ~45-70 seconds of overhead on top of the actual model invocations. For a workflow where the involved steps are known in advance (generate idea → translate → generate image → store ++), that reasoning overhead provides zero value. You’re paying for flexibility you don’t need.

Next iteration: AWS Step Functions

Defining all steps in the process in an AWS Step Functions State Machine was relatively straight forward, and i set up the idea generation step is a separate AWS Lambda function, invoking the Bedrock Agent for the creative reasoning process only. This cut the processing time in half and also provided better visibility into the end-to-end workflow.

Comparison of orchestration workflow alternatives

Dimension	AWS Step Functions	Amazon Bedrock Agents
Control & predictability	Deterministic, explicit state transitions	AI-driven, non-deterministic paths
Latency	~15-25s for idea generation	~45-90s due to reasoning overhead
Cost model	Pay per state transition	Pay per token + reasoning tokens
Flexibility	Requires code changes for new flows	Natural language instructions
Operational complexity	Visual debugging, built-in retries	Black-box reasoning, harder to debug
Best for	Well-defined, latency-sensitive workflows	Dynamic, exploratory tasks

The latency difference alone was disqualifying. Parents using the site at 11 PM won’t have the patience to wait two minutes seconds for an idea. But the operational complexity gap matters just as much in practice; when an Amazon Bedrock Agent fails, you get a vague error from the reasoning loop (out of the box) and you have to add more logging for debugging capabilities. When an AWS Step Functions state fails, you see exactly which state, with the full input/output, on a visual timeline.

Experiences with AWS Step Functions orchestration :

The initial Step Functions workflow had its own problems — too many fine-grained states that should have been a single Lambda call, and no parallel processing. Consolidating related operations and adding parallel branches for translation and image generation brought execution time from ~45s down to ~20s.

Consolidated related operations into single Lambda functions (fewer state transitions, less overhead)
Used parallel branches for translation and image generation — they’re independent, so there’s no reason to run them sequentially
Added explicit error handling states with retry logic instead of relying on default behavior
Implemented cost limit checks at the start to fail fast before any expensive model invocations

Key takeaway: Use AWS Step Functions when the workflow is well-defined and the steps are known in advance. Use Amazon Bedrock Agents when you genuinely need dynamic reasoning – when the agent needs to decide which tools to call based on context, or when the workflow varies significantly between executions. For content generation pipelines with fixed steps, AWS Step Functions wins on every dimension that matters: latency, cost, debuggability, and predictability.

Outcome : Overall execution time ~90-60 seconds → ~30 seconds. The AWS Step Functions visual debugger has saved me hours of troubleshooting – being able to replay a failed execution with the exact input/output at each state is invaluable.

Conclusion

Building this site just for fun to push the boundaries of creative capabilities with Generative AI on AWS has been really cool and inspiring. I’ve learned a lot during the process which will also be extremely valuable when implementing customer solutions.

Key takeaways

As Generative and Agentic AI technology matures, more and more use-cases emerge – But, to meet expectations, solution design requires careful orchestration, model benchmarking and output quality assurance.
Model capabilities are also developing fast – Stay up to date on the recent advances and availability of new model versions to experience even more accurate and high quality results.
Model selection requires empirical testing – benchmarks don’t tell the whole story, especially for sub-major languages.
AWS Step Functions beats Amazon Bedrock Agents for deterministic workflows – the latency difference could be significant (~30s vs ~60-120s in this case). Use Agent orchestration for non-deterministic scenarios or where you need to handle unknown edge cases.
Content safety is solvable – Amazon Bedrock Guardrails provides enterprise-grade filtering with minimal latency impact. Also, some model providers have built-in content safety filters.
Serverless scales effortlessly – the architecture handles traffic spikes during the Christmas season without intervention.
Spec-driven development with Kiro boost innovation – Leveraging Kiro along with a maturing base of agent/project steering definitions, MCP servers and subagents really amplified development velocity and learning outcomes.

Future enhancements

Some ideas currently in the backlog:

User accounts for saving favorite ideas and tracking which ones you’ve used.
Personalization based on past preferences and household setup.
December planning mode: Generate a full month of ideas in one go – no more nightly scrambling.
Weekly elf calendar: Get a curated week of ideas that build on each other (storyline mode).
Difficulty progression: Start simple in early December, escalate to elaborate setups as Christmas approaches.
Integration with smart home devices (imagine the elf controlling your speakers and Christmas tree lights)!
Collaborative features: Share your elf calendar with your partner so you can take turns.
Portal support for additional languages (German, French).

Try it out yourself!

Visit naughtyelfideas.com to see it in action and generate your own unique elf idea.

Let me know what you think, feedback is welcome!

The post Generative AI for creativity: Lessons learned from building a Christmas Elf idea platform with Amazon Bedrock first appeared on Håkon Eriksen Drange.

AWS Immersion Day talk: Web Application Firewall and AWS Shield Advanced

Håkon Eriksen Drange — Tue, 25 Nov 2025 20:17:00 +0000

I was invited by AWS Norway to contribute to their AWS Immersion Day on Zero Trust which took place on Tuesday November 25th 2025.

Elshan Hasanov from AWS started with an introduction to the concept of Zero Trust and how Amazon Verified Access and Amazon VPC Lattice can be used for the purpose. Next up I presented approaches for application and infrastructure security with AWS Web Application Firewall and AWS Shield Advanced.

You can find the slides below.

2025-11-25-aws-immersion-day-zero-trust-waf-shield-haakon-eriksen-drange Download

About the event

In today’s dynamic and distributed cloud environments, traditional perimeter security alone is no longer suﬃcient to protect against advanced threats. This session explores the paradigm shift towards Zero Trust architecture and its implementation within AWS ecosystems, covering the use of AWS services and features to adopt Zero Trust principles alongside traditional security approaches for fine-grained access control, network segmentation, secure data access, and logging. Join us to learn how to transition to a more secure, resilient, and comprehensive security approach tailored for your AWS environment.

What to Expect from the Event

This session and hands-on workshop provides an in-depth exploration of Zero Trust security principles and their practical application within AWS environments using AWS Verified Access and VPC Lattice, complemented by modern perimeter security strategies including WAF and firewall inspection.

Core Focus Areas

Identity-aware, least-privilege access for both human users and microservices
Integration of Zero Trust with strategic perimeter controls including WAF
Practical implementation using AWS Verified Access and Amazon VPC Lattice

Hands-on Components

Configure AWS Verified Access for secure remote user access
Implement Amazon VPC Lattice for service-to-service communication

Original AWS Experience North event page:

2025-11-25-zero-trust-event-page Download

The post AWS Immersion Day talk: Web Application Firewall and AWS Shield Advanced first appeared on Håkon Eriksen Drange.

AWS User Group Oslo talk: From Vibe Coding to Spec-Driven Development with Kiro

Håkon Eriksen Drange — Wed, 19 Nov 2025 13:39:13 +0000

On Tuesday November 18th at the AWS User Group Oslo Meetup I shared my views on why builders should pivot from Vibe Coding to Spec-Driven Development with Kiro.

Highlights

How AI assistants are transforming the Software Development Lifecycle (all code, not only frontend and backend, Infrastructure-as-Code as well)
Introduction to Vibe Coding, the good and the not so good parts
How traditional and established software craftmanship is becoming more relevant than ever
How Kiro and the Spec-Driven Development workflow helps bring more structure, quality and speed
A practical demonstration; adding a new feature to a serverless weather forecast application, deployed on AWS with Terraform and GitHub Actions
Best practices for specs, agent steering context, MCP and token optimization based on my experience
Predictions for the future

You can find the slides below. Thanks to AWS Community Builders and the Amazon Kiro team for valuable insights and supporting material.

For a detailed walk-through check out https://hedrange.com/2025/08/11/how-to-use-kiro-for-ai-assisted-spec-driven-development/ .

2025-11-18-aws-user-group-oslo-from-vibe-coding-to-spec-driven-development-with-kiro Download

Additional resources

The post AWS User Group Oslo talk: From Vibe Coding to Spec-Driven Development with Kiro first appeared on Håkon Eriksen Drange.

Demonstrate practical AWS skills with new microcredentials

Håkon Eriksen Drange — Fri, 14 Nov 2025 07:58:42 +0000

AWS has announced a new skills validation program called Microcredentials. These are more practical and lightweight approaches for validating knowledge than a full, comprehensive certification exam. It’s rewarding to be able to go beyond theoretical knowledge and prove what you’ve actually learned. AWS Certifications and Microcredentials complement each other; validating both your deep technical knowledge and hands-on skills.

Image credit: AWS – https://www.aboutamazon.com/news/aws/aws-ai-certification-learning-tools-skills-development

This is how it works

The exam labs are in a live AWS-provisioned environment, similar to regular lab/SimuLearn tasks on AWS Skill Builder. During the course of 90 minutes candidates are presented with a set of challenges to be achieved by practical implementation in the AWS Console (ClickOps, no IaC). You will have to diagnose issues and implement solutions on your own, there are no hints or guidance provided. The exam lab cannot be paused or restarted, so if you have to quit you need to start over again. Not much different than an actual certification exam. Candidates failing to meet the passing score objective can go for a re-take after 25 days.

As of November 2025 the available training options are:

Microcredential Preview Experience

This microcredential validates your ability to configure and connect basic AWS services in hands-on scenarios. Key focus areas include S3 static hosting, API Gateway connections, Lambda integrations, and DynamoDB storage. This is not a real microcredential exam lab. This is a trial version you can use to familiarize yourself with the interface.

I recommend you start here to become familiar with the concept to set yourself up for success for the actual lab exam.

AWS Skill Builder link: https://skillbuilder.aws/learn/JBTFY8M6S8/microcredential-preview-experience/YJFR7KHKR3

AWS Agentic AI Demonstrated

_ AWS Agentic AI Demonstrated is a hands-on exam lab designed to help you validated your AWS skills in the Agentic AI domain_.

Objectives

Troubleshoot and repair supervisor and specialist Bedrock Agents
Troubleshoot and repair an Amazon Bedrock knowledge base
Integrate and fix a Bedrock Agent
Enhance Bedrock Agent capabilities
Integrate Bedrock Guardrails with a Bedrock Agent
Connect a web application chat client with a Bedrock Agent

AWS Skill Builder link: https://skillbuilder.aws/learn/32Y249P272/aws-agentic-ai-demonstrated/TTAJ5WKYTS

AWS Serverless Demonstrated

AWS Serverless Demonstrated is a hands-on exam lab designed to help you validate your AWS skills in the Serverless domain.

Objectives

Configure an AWS Lambda function
Deploy a REST API
Configure a Step Functions state machine
Design and implement an event-driven system
Optimize AWS Lambda functions for various scenarios
Configure a CI/CD pipeline for serverless applications
Configure and analyze monitoring and telemetry

AWS Skill Builder Link: https://skillbuilder.aws/learn/XV3B4RGA8Q/aws-serverless-demonstrated/BYD5SH8R5C

Tip: Completing the Serverless Knowledge Badge Readiness Path course first can be helpful (you will also get a badge upon completing a multiple-choice assessment).

Result

Passing the microcredential lab exams will unlock some nice new Credly badges you can share with your manager, colleagues and on social media to prove the skills you’ve acquired.

My reference: https://www.credly.com/badges/bc214808-641d-4fb2-b196-e78b530af563

My reference: https://www.credly.com/badges/80b6282b-026e-482e-bc14-9aa801536435

Good luck!

The post Demonstrate practical AWS skills with new microcredentials first appeared on Håkon Eriksen Drange.

How to use Kiro for AI assisted spec-driven development

Håkon Eriksen Drange — Mon, 11 Aug 2025 13:35:31 +0000

Read on to learn how to use Kiro for AI assisted spec-driven development of a serverless weather forecasting app, using Terraform for deployment to AWS.

Table of contents

Introduction
Kiro introduces the AI assisted spec-driven development workflow
Kiro core concepts
- Core capabilities
- Specs: Plan and build features using structured specifications
- Hooks: Automate repetitive tasks with intelligent triggers
- Agentic chat: Build features through natural conversation with AI
- Steering: Guide AI with custom rules and context
- MCP Servers: Connect external tools and data sources
Getting Kiro
Starting your first Kiro project
- Vibe
- Spec
Setting up Kiro
- Model Context Protocol (MCP) servers
Setting up agent steering context
Writing your product spec
- Step 1: Define requirements
- Step 2: Define design
- Step 3: Implement
Deploying the final solution
- Workflow for adding a new feature
Learnings and key takeaways
- Reflections on context
Kiro pricing
Conclusion
Resources

Introduction

Since the advent of Generative AI, coding assistants and their evolution has been a topic of much discussion. As the Large Language Models became increasingly more precise, the AI-based coding assistants or companions have been able to produce increasingly more relevant suggestions, which has been incorporated into extensions and new IDEs such as Cursor in addition to the CLI. We’ve evolved from simple suggestions of a few lines to transformation of complete codebases. The focus is pivoting from assistance to resolution of a particular problem and outcomes, what we instruct the software to achieve. Vibe Coding will probably go down in history as one of the main terms of 2025. It can be efficient for prototypes and proof-of-concepts, but how can we know what assumptions and decisions the agent made to get to that result?

Traditional software development processes are based on initial specification. We need to know the purpose and functionality of what to build before we starting building.

Project Rules and Customizations in Amazon Q Developer were a step in the right direction, but some key challenges I experienced with AI coding assistants earlier in 2025:

Don’t remember context or state. If you shut down your laptop and continue tomorrow the context may be lost (was improved with Q Developer)
How to share context across multiple developers in a team
How to get more valuable output according to personal/company preference

Research referenced by AWS shows that addressing issues during the development phase is 5 to 7 times more costly than resolving them during the planning phase of the software development lifecycle. Similarly, it’s less complex and costly to change a system before going to production.

This principle holds true even with AI coding assistants. When you take the time to discuss requirements and design with Kiro during the planning phase, a single specification request will often accomplish what would otherwise require multiple vibe iterations during implementation.

Garbage in, garbage out, you know.

Luckily, Kiro can now help incorporate that well-known structure into AI assistant coding, in a consistent manner.

Kiro introduces the AI assisted spec-driven development workflow

Kiro is a new software development IDE based on Visual Studio Code that turns prompts into clear requirements, structured designs and implementation tasks.

The whole process is validated by tests. Code is generated by revolutionary AI agents utilizing the latest and most up-to-date Large Language Models (LLMs).

Kiro leverages Anthropic’s Claude Sonnet 4 under the hood, with the option to fall back to 3.7 (prefer the newest one). These models are specialized in agentic coding and tasks across the entire software development lifecycle from initial planning, implementation, bug fixing, maintenance and refactoring.

I recommend reading Introducing Kiro and Kiro and the future of AI spec-driven software development to get up to speed.

Kiro core concepts

Kiro introduces two main modes: Vibe and Spec.

Image courtesy of Kiro

Core capabilities

You can read more about my experiences with these capabilities in the next chapter. Let me introduce the concepts first.

Specs: Plan and build features using structured specifications

Specs or specifications are structured artifacts that formalize the development process for complex features in your application. They provide a systematic approach to transform high-level ideas into detailed implementation plans with clear tracking and accountability.

With Kiro’s specs, you can:

Break down requirements into user stories with acceptance criteria

Build design docs with sequence diagrams and architecture plans

Track implementation progress across discrete tasks

Collaborate effectively between product and engineering teams

Reference: https://kiro.dev/docs/specs/

Hooks: Automate repetitive tasks with intelligent triggers

Agent Hooks are automated triggers that execute predefined agent actions when specific events occur in the Kiro IDE. When files are created, saved or deleted you can configure hooks to be run for common tasks to:

Maintain consistent code quality

Prevent security vulnerabilities

Reduce manual overhead

Standardize team processes

Create faster development cycles

Reference: https://kiro.dev/docs/hooks/

This is an area I have not explored in detail yet.

Agentic chat: Build features through natural conversation with AI

Kiro offers a chat panel where you can interact with your code through natural language conversations. Just tell Kiro what you need. Ask questions about your codebase, request explanations for complex logic, generate new features, debug tricky issues, and automate repetitive tasks—all while Kiro maintains complete context of your project.

Steering: Guide AI with custom rules and context

I believe this is one of the most powerful capabilities Kiro introduces. To quote the documentation:

Steering gives Kiro persistent knowledge about your project through markdown files in .kiro/steering/. Instead of explaining your conventions in every chat, steering files ensure Kiro consistently follows your established patterns, libraries, and standards.

Consistent Code Generation – Every component, API endpoint, or test follows your team’s established patterns and conventions.

Reduced Repetition – No need to explain project standards in each conversation. Kiro remembers your preferences.

Team Alignment – All developers work with the same standards, whether they’re new to the project or seasoned contributors.

Scalable Project Knowledge – Documentation that grows with your codebase, capturing decisions and patterns as your project evolves.

Reference: https://kiro.dev/docs/steering/

MCP Servers: Connect external tools and data sources

In my opinion the main inputs for valuable and tailored results is your combination of Steering and Model Context Protocol servers. MCP extends Kiro’s capabilities by connecting to specialized servers that provide additional tools and context, tailored to your environment.

MCP is a protocol that allows Kiro to communicate with external servers to access specialized tools and information. For example, the AWS Documentation MCP server provides tools to search, read, and get recommendations from AWS documentation directly within Kiro.

With MCP, you can:

Access specialized knowledge bases and documentation

Integrate with external services and APIs

Extend Kiro’s capabilities with domain-specific tools

Create custom tools for your specific workflows

Reference: https://kiro.dev/docs/mcp/

Getting Kiro

As of early August 2025 Kiro is still in public preview with limited availability and a waiting list.

Assuming you have been able to get Kiro from https://kiro.dev/ go through their Get started guide to learn more about the basic concepts.

Starting your first Kiro project

Open a new folder to start a new project and you are presented with the option build Vibe or Spec style.

Vibe

Chat first, then build. Explore ideas and iterate as you discover needs.

Great for:

Rapid exploration and testing
Building when requirements are unclear
Implementing a task

Spec

Plan first, then build. Create requirements and design before coding starts.

Great for:

Thinking through feature in-dept
Projects needing upfront planning
Building features in a structured way

In my case I’ve used Amazon Q Developer in VS Code and CLI quite a lot for coding acceleration, so I went directly for Spec. Let’s get back to working with specifications after we have configured the remaining parts of our Kiro workspace.

Setting up Kiro

Model Context Protocol (MCP) servers

Enriching your environment with relevant MCP servers can be a massive boost. Take a look at the official MCP Servers from AWS at https://github.com/awslabs/mcp , there is already a ton available.

The ones I currently enjoy in Kiro and Amazon Q Developer focusing on Terraform development and AWS are:

AWS Core provides tools for prompt understanding and translation to AWS services
AWS Docs and AWS Knowledge can read, search and recommend from the official, up-to-date AWS Documentation.
AWS API can suggest for you and call AWS CLI commands on your behalf.
AWS CDK can provide guidance and generate CDK stacks.
AWS Terraform can search AWS and AWSCC provider docs, Execute Terraform and Terragrunt Commands, run Checkov scans and search user provided Terraform registry modules. Super valuable!
AWS Serverless can provide guidance, search schemas, deploy serverless applications, get metrics and so on.
AWS Diagram get generate diagrams, get diagram examples and list icons. You can generate architecture diagrams with official AWS icons, flow and sequence charts and so on. Content can be provided as a static image or in Draw.IO XML format, so that you can finishing up the final touches and corrections yourself.

- AWS Pricing can analyze CDK and Terraform projects, query the official pricing API and generate cost reports, much more efficient than manually working with the AWS Cost Calculator.

The MCP configuration feature in Kiro supports two modes: User Config (global) and Workspace Config.

Personally I prefer Workspace Config and keeping all context in the project. This makes it easier and predictable for other colleagues having the same MCP server configuration. It also yields more consistent outputs. Think about it, if not all team members have the same context settings, the results are not guaranteed to be consistent and could lead to implementation differences and bugs.

Here is my current Workspace MCP Config, which I have added to a common agent-steering-bootstrap Git repo:

<textarea tabindex="-1" aria-hidden="true" readonly>{
  "mcpServers": {
    "fetch": {
      "command": "uvx",
      "args": [
        "mcp-server-fetch"
      ],
      "env": {},
      "disabled": false,
      "autoApprove": []
    },
    "aws-docs": {
      "command": "uvx",
      "args": [
        "awslabs.aws-documentation-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": [
        "search_documentation",
        "read_documentation"
      ]
    },
    "aws-core": {
      "command": "uvx",
      "args": [
        "awslabs.core-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": []
    },
    "aws-api": {
      "command": "uvx",
      "args": [
        "awslabs.aws-api-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": []
    },
    "aws-knowledge-mcp-server": {
      "command": "uvx",
      "args": [
        "mcp-proxy",
        "--transport",
        "streamablehttp",
        "https://knowledge-mcp.global.api.aws"
      ],
      "disabled": false,
      "autoApprove": []
    },
    "aws-cdk": {
      "command": "uvx",
      "args": [
        "awslabs.cdk-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": []
    },
    "aws-terraform": {
      "command": "uvx",
      "args": [
        "awslabs.terraform-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": []
    },
    "aws-serverless": {
      "command": "uvx",
      "args": [
        "awslabs.aws-serverless-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": []
    },
    "awslabs-diagram": {
      "command": "uvx",
      "args": [
        "awslabs.aws-diagram-mcp-server"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": [
        "get_diagram_examples",
        "generate_diagram"
      ]
    },
    "awslabs-pricing": {
      "command": "uvx",
      "args": [
        "awslabs.aws-pricing-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR",
        "AWS_PROFILE": "default",
        "AWS_REGION": "eu-west-1"
      },
      "disabled": false,
      "autoApprove": [
        "get_pricing_service_codes",
        "get_pricing_service_attributes",
        "get_pricing_attribute_values",
        "get_pricing"
      ]
    }
  }
}</textarea>

{
  "mcpServers": {
    "fetch": {
      "command": "uvx",
      "args": [
        "mcp-server-fetch"
      ],
      "env": {},
      "disabled": false,
      "autoApprove": []
    },
    "aws-docs": {
      "command": "uvx",
      "args": [
        "awslabs.aws-documentation-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": [
        "search_documentation",
        "read_documentation"
      ]
    },
    "aws-core": {
      "command": "uvx",
      "args": [
        "awslabs.core-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": []
    },
    "aws-api": {
      "command": "uvx",
      "args": [
        "awslabs.aws-api-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": []
    },
    "aws-knowledge-mcp-server": {
      "command": "uvx",
      "args": [
        "mcp-proxy",
        "--transport",
        "streamablehttp",
        "https://knowledge-mcp.global.api.aws"
      ],
      "disabled": false,
      "autoApprove": []
    },
    "aws-cdk": {
      "command": "uvx",
      "args": [
        "awslabs.cdk-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": []
    },
    "aws-terraform": {
      "command": "uvx",
      "args": [
        "awslabs.terraform-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": []
    },
    "aws-serverless": {
      "command": "uvx",
      "args": [
        "awslabs.aws-serverless-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": []
    },
    "awslabs-diagram": {
      "command": "uvx",
      "args": [
        "awslabs.aws-diagram-mcp-server"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "disabled": false,
      "autoApprove": [
        "get_diagram_examples",
        "generate_diagram"
      ]
    },
    "awslabs-pricing": {
      "command": "uvx",
      "args": [
        "awslabs.aws-pricing-mcp-server@latest"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR",
        "AWS_PROFILE": "default",
        "AWS_REGION": "eu-west-1"
      },
      "disabled": false,
      "autoApprove": [
        "get_pricing_service_codes",
        "get_pricing_service_attributes",
        "get_pricing_attribute_values",
        "get_pricing"
      ]
    }
  }
}

Setting up agent steering context

Steering gives Kiro persistent knowledge about your project through markdown files in .kiro/steering/. Instead of explaining your conventions in every chat, steering files ensure Kiro consistently follows your established patterns, libraries, and standards.

Key benefits:

Consistent Code Generation – Every component, API endpoint, or test follows your team’s established patterns and conventions.

Reduced Repetition – No need to explain project standards in each conversation. Kiro remembers your preferences.

Team Alignment – All developers work with the same standards, whether they’re new to the project or seasoned contributors.

Scalable Project Knowledge – Documentation that grows with your codebase, capturing decisions and patterns as your project evolves.

Reference: https://kiro.dev/docs/steering/

MCP agent steering contains more background information and guidance about how Kiro can leverage the active MCP servers.

Product focuses on common product development context and principles.

Structure focuses on how the files in your codebase are structured, to align with company standards and preferences.

In Tech I define general patterns and principles for approaching solutions deployed to AWS with Terraform.

Most companies have product development principles, architecture and software guidelines documented in their internal wikis. This context is crucial to get into Kiro Agent Steering context, to get outputs matching with company and team preferences.

Example from tech.md:

<textarea tabindex="-1" aria-hidden="true" readonly># Technology Stack

This document outlines the technical foundation and tooling for the project.

## Build System &amp; Tools
- CI/CD is out of scope of this Terraform module.

## Application Tech Stack
- Python3, Boto3, Jinja templating etc.
- Unit testing of core functionality
- Basic testing of Terraform code
- Provide /health endpoint for REST APIs
- Python operations should take place in a virtual environment where optimal Python version is installed with pyenv

## Infrastructure Tech Stack
- Terraform for Infrastructure-as-Code
- Terraform providers aws and awscc, if necessary
- Leverage community modules from https://github.com/terraform-aws-modules as relevant
- AWS Serverless architecture options are preferred for minimal operational overhead
- The AWS infrastructure is Well-Architected
- The AWS infrastructure is secure as per the latest CIS AWS Security Hub control standard
- Terraform code is unit tested Terraform's native testing framework, HCL-based tests.

## Observability
- For serverless components, AWS X-Ray is leveraged for tracing
- Logs are directed to AWS CloudWatch Logs. CloudWatch Logs groups have a retention period of 180 days. 
- A solution specific AWS Cloudwatch Dashboard which includes relevant CloudWatch metrics for reliability, performance and cost, in addition to a list over the last failing requests

### Pre-commit for Terraform
- Pre-commit is installed and leveraged for validation and formatting. 
  - terraform_fmt
  - terraform_docs in main README.md
  - check-merge-conflict
  - trailing-whitespace
  - mixed-line-ending

Example .pre-commit-config.yaml located in the root directory:

repos:

repo: https://github.com/antonbabenko/pre-commit-terraform rev: v1.77.3 hooks:
- id: terraform_fmt
- id: terraform_docs args: ["--args=--sort-by required"]
- id: terraform_checkov args:
  - --args=--quiet
  - --args=--download-external-modules false
repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.4.0 hooks:
- id: check-merge-conflict
- id: trailing-whitespace args: [--markdown-linebreak-ext=md]
- id: mixed-line-ending args: ["--fix=lf"]


## Documentation
- AWS Labs Diagram MCP server is used to produce relevant architecture, flow and sequence diagrams, included in main README.md
- AWS Labs Pricing MCP server is used to perform a basic cost calculation of the solution, included in the main README.md
- Every task should also ensure relevant and clear documentation is created or up to date. Prefer simple and user friendly documentation, don't overcomplicate.
- All documentation follows markdown format and is stored in the `docs/` directory
- Architecture diagrams are generated programmatically using the AWS diagram MCP server
- Cost analysis documentation includes detailed breakdowns, usage projections, and optimization recommendations
- Documentation includes deployment guides, troubleshooting guides, and operational runbooks
- There should be an examples folder with README.md explaining how to include the Terraform module call in an existing CI/CD codebase.
- In documentation, provide TL;DR to make it easy and quick for developers to get up to speed. 
- In high level project documentation, include an executive summary for target group project owners, to articulate functionality and the value the solution provides.

## Cost Management
- AWS Labs Pricing MCP server provides accurate cost calculations for the infrastructure components of the solution.
- Cost analysis should include environment-specific projections (staging and production).
- Cost analysis should include AWS region comparison of eu-west-1, eu-central-1 and eu-north-1 for the production environment.
- CloudWatch cost metrics and dashboards provide real-time cost monitoring.
- A solution specific AWS Budget is deployed, based on infrastructure tag Key Service. Budget alerts prevents unexpected charges.
- Guidance is provided for the top three cost items that may increase with heavy production load. 
- Cost documentation is included in the main `README.md`

## Principles
- Favor KISS over complexity, simplicity over comprehensibility
- Respect and adopt well-known cloud based architecture and integration patterns

</textarea>

# Technology Stack

This document outlines the technical foundation and tooling for the project.

## Build System & Tools
- CI/CD is out of scope of this Terraform module.

## Application Tech Stack
- Python3, Boto3, Jinja templating etc.
- Unit testing of core functionality
- Basic testing of Terraform code
- Provide /health endpoint for REST APIs
- Python operations should take place in a virtual environment where optimal Python version is installed with pyenv

## Infrastructure Tech Stack
- Terraform for Infrastructure-as-Code
- Terraform providers aws and awscc, if necessary
- Leverage community modules from https://github.com/terraform-aws-modules as relevant
- AWS Serverless architecture options are preferred for minimal operational overhead
- The AWS infrastructure is Well-Architected
- The AWS infrastructure is secure as per the latest CIS AWS Security Hub control standard
- Terraform code is unit tested Terraform's native testing framework, HCL-based tests.

## Observability
- For serverless components, AWS X-Ray is leveraged for tracing
- Logs are directed to AWS CloudWatch Logs. CloudWatch Logs groups have a retention period of 180 days. 
- A solution specific AWS Cloudwatch Dashboard which includes relevant CloudWatch metrics for reliability, performance and cost, in addition to a list over the last failing requests

### Pre-commit for Terraform
- Pre-commit is installed and leveraged for validation and formatting. 
  - terraform_fmt
  - terraform_docs in main README.md
  - check-merge-conflict
  - trailing-whitespace
  - mixed-line-ending

Example .pre-commit-config.yaml located in the root directory:

repos:

repo: https://github.com/antonbabenko/pre-commit-terraform rev: v1.77.3 hooks:
- id: terraform_fmt
- id: terraform_docs args: ["--args=--sort-by required"]
- id: terraform_checkov args:
  - --args=--quiet
  - --args=--download-external-modules false
repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.4.0 hooks:
- id: check-merge-conflict
- id: trailing-whitespace args: [--markdown-linebreak-ext=md]
- id: mixed-line-ending args: ["--fix=lf"]


## Documentation
- AWS Labs Diagram MCP server is used to produce relevant architecture, flow and sequence diagrams, included in main README.md
- AWS Labs Pricing MCP server is used to perform a basic cost calculation of the solution, included in the main README.md
- Every task should also ensure relevant and clear documentation is created or up to date. Prefer simple and user friendly documentation, don't overcomplicate.
- All documentation follows markdown format and is stored in the `docs/` directory
- Architecture diagrams are generated programmatically using the AWS diagram MCP server
- Cost analysis documentation includes detailed breakdowns, usage projections, and optimization recommendations
- Documentation includes deployment guides, troubleshooting guides, and operational runbooks
- There should be an examples folder with README.md explaining how to include the Terraform module call in an existing CI/CD codebase.
- In documentation, provide TL;DR to make it easy and quick for developers to get up to speed. 
- In high level project documentation, include an executive summary for target group project owners, to articulate functionality and the value the solution provides.

## Cost Management
- AWS Labs Pricing MCP server provides accurate cost calculations for the infrastructure components of the solution.
- Cost analysis should include environment-specific projections (staging and production).
- Cost analysis should include AWS region comparison of eu-west-1, eu-central-1 and eu-north-1 for the production environment.
- CloudWatch cost metrics and dashboards provide real-time cost monitoring.
- A solution specific AWS Budget is deployed, based on infrastructure tag Key Service. Budget alerts prevents unexpected charges.
- Guidance is provided for the top three cost items that may increase with heavy production load. 
- Cost documentation is included in the main `README.md`

## Principles
- Favor KISS over complexity, simplicity over comprehensibility
- Respect and adopt well-known cloud based architecture and integration patterns

Writing your product spec

Product specs or specifications are structured artifacts that formalize the development process. They provide a systematic approach to transform high-level ideas into detailed implementation plans with clear tracking and accountability.

The workflow is illustrated below:

From the Kiro pane, click the + button under Specs. Alternatively, choose Spec from the chat pane.

Describe your project idea.

A requirements Markdown file is created in a folder with the spec name weather-forecast-app.

Step 1: Define requirements

The requirements.md file should define user stories with acceptance criterias in EARS notation, similar to common agile development practice. Define what we would like to achieve and which problems we propose to solve. HOW we plan to solve it will come afterwards in the Design phase.

EARS, which stands for Easy Approach to Requirements Syntax, is a method for writing clear and unambiguous requirements using a structured set of rules and keywords. Alistair Mavin and colleagues at Rolls-Royce PLC developed EARS whilst analysing the airworthiness regulations for a jet engine’s control system. The structured format makes it easy to understand what is expected, reducing misinterpretations. Clearer requirements lead to better test cases and easier verification of application functionality.

<textarea tabindex="-1" aria-hidden="true" readonly>WHEN [condition/event]
THE SYSTEM SHALL [expected behavior]</textarea>

WHEN [condition/event]
THE SYSTEM SHALL [expected behavior]

Starting writing your User Stories as acceptance criterias in EARS format. Remember, as described in the Introduction, the more complete context you provide, including what your team usually have in their minds and have learned by experience how you do you things in your company, be as specific as you can. Investing more time in producing well-crafted specifications can reduce time spent on modifications and troubleshooting.

Remember, common principles and guidelines are defined as Agent Steering Context. Product Spec focuses on the functionality of the application. As the system grows you can create additional specifications and manage requirements in logical separation.

This is how the .kiro/specs/weather-forecast-app/requirements.md for the example weather forecast application looks like:

<textarea tabindex="-1" aria-hidden="true" readonly># Requirements Document

## Introduction

This specification will create a weather forecast application, to be deployed with Terraform on AWS serverless infrastructure.

### Requirement 1

**User Story:** As an end user, I will access a web site which compares the weather forecast for tomorrow for the European cities Oslo (Norway), Paris (France), London (United Kingdom) and Barcelona (Spain).

#### Acceptance Criteria

1. WHEN an end user is accessing the service THEN the system SHALL display a simple web site with a fancy design for the weather forecast for the cities as described in the User Story
2. WHEN an end user is accessing the service THEN the system SHALL be snappy and respond fast
3. WHEN an end user is accessing the service on a mobile device THEN the design SHALL be optimized for display on a small screen
4. WHEN static content is served to end users THEN the system SHALL set Cache-Control headers with Max-Age of 900 seconds (15 minutes) to optimize performance and reduce server load

### Requirement 2

**User Story:** As a developer, my application requirements are as follows:

#### Acceptance Criteria

1. WHEN the weather-forecast-app is generated THEN the system SHALL provide a modern front-end application
2. WHEN the weather-forecast-app is generated THEN the system SHALL look up weather forecasts from https://api.met.no/weatherapi/locationforecast/2.0/documentation and cache the results for 1 hour
3. WHEN the weather-forecast-app is generated THEN the system SHALL respect the Terms of Service defined at https://developer.yr.no/doc/TermsOfService/
4. WHEN the weather-forecast-app is generated THEN the system SHALL be tested
5. WHEN the Lambda function successfully retrieves weather data from the backend API THEN the system SHALL set cache-control: max-age=60 on the HTTP response
6. WHEN the Lambda function fails to retrieve weather data from the backend API THEN the system SHALL set cache-control: max-age=0 on the HTTP response
7. WHEN the frontend displays weather data THEN the system SHALL show the Last updated timestamp from the lastUpdated field in the API response
8. WHEN weather data is cached in DynamoDB and the weather API does not provide timestamp information THEN the system SHALL use the DynamoDB cache timestamp as the lastUpdated value in the API response

### Requirement 3

**User Story:** As a developer, my cloud infrastructure requirements are as follows:

#### Acceptance Criteria

1. WHEN the infrastructure is generated THEN the codebase SHALL be organized as one, self-contained Terraform module
2. WHEN the infrastructure is generated THEN the system SHALL require basic unit tests and infrastructure-as-code validation to be successful
3. WHEN the infrastructure is deployed THEN the system SHALL create AWS resources with appropriate tags like Service:weather-forecast-app.
4. WHEN the infrastructure is deployed THEN the system SHALL package and deploy the weather-forecast-app code
5. WHEN the infrastructure is deployed THEN the system SHALL provide accessible endpoints for testing
6. WHEN the infrastructure is deployed THEN the system SHALL include required IAM roles and permissions
7. WHEN the infrastructure is deployed THEN the system SHALL output relevant URLs or connection information
8. WHEN the infrastructure is deployed THEN the system SHALL be configured for high availability
9. WHEN the CloudFront distribution is deployed THEN the system SHALL use price class 100 to optimize costs while covering Europe and the United States edge locations
10. WHEN the CloudFront distribution is deployed THEN the system SHALL allow only GET, HEAD, and OPTIONS HTTP methods for security and performance optimization
11. WHEN the CloudFront distribution is deployed THEN the system SHALL configure caching policies based on query parameters to optimize cache efficiency
12. WHEN the CloudFront distribution is deployed THEN the system SHALL set the default TTL to 900 seconds (15 minutes) to align with static content caching requirements</textarea>

# Requirements Document

## Introduction

This specification will create a weather forecast application, to be deployed with Terraform on AWS serverless infrastructure.

### Requirement 1

**User Story:** As an end user, I will access a web site which compares the weather forecast for tomorrow for the European cities Oslo (Norway), Paris (France), London (United Kingdom) and Barcelona (Spain).

#### Acceptance Criteria

1. WHEN an end user is accessing the service THEN the system SHALL display a simple web site with a fancy design for the weather forecast for the cities as described in the User Story
2. WHEN an end user is accessing the service THEN the system SHALL be snappy and respond fast
3. WHEN an end user is accessing the service on a mobile device THEN the design SHALL be optimized for display on a small screen
4. WHEN static content is served to end users THEN the system SHALL set Cache-Control headers with Max-Age of 900 seconds (15 minutes) to optimize performance and reduce server load

### Requirement 2

**User Story:** As a developer, my application requirements are as follows:

#### Acceptance Criteria

1. WHEN the weather-forecast-app is generated THEN the system SHALL provide a modern front-end application
2. WHEN the weather-forecast-app is generated THEN the system SHALL look up weather forecasts from https://api.met.no/weatherapi/locationforecast/2.0/documentation and cache the results for 1 hour
3. WHEN the weather-forecast-app is generated THEN the system SHALL respect the Terms of Service defined at https://developer.yr.no/doc/TermsOfService/
4. WHEN the weather-forecast-app is generated THEN the system SHALL be tested
5. WHEN the Lambda function successfully retrieves weather data from the backend API THEN the system SHALL set cache-control: max-age=60 on the HTTP response
6. WHEN the Lambda function fails to retrieve weather data from the backend API THEN the system SHALL set cache-control: max-age=0 on the HTTP response
7. WHEN the frontend displays weather data THEN the system SHALL show the Last updated timestamp from the lastUpdated field in the API response
8. WHEN weather data is cached in DynamoDB and the weather API does not provide timestamp information THEN the system SHALL use the DynamoDB cache timestamp as the lastUpdated value in the API response

### Requirement 3

**User Story:** As a developer, my cloud infrastructure requirements are as follows:

#### Acceptance Criteria

1. WHEN the infrastructure is generated THEN the codebase SHALL be organized as one, self-contained Terraform module
2. WHEN the infrastructure is generated THEN the system SHALL require basic unit tests and infrastructure-as-code validation to be successful
3. WHEN the infrastructure is deployed THEN the system SHALL create AWS resources with appropriate tags like Service:weather-forecast-app.
4. WHEN the infrastructure is deployed THEN the system SHALL package and deploy the weather-forecast-app code
5. WHEN the infrastructure is deployed THEN the system SHALL provide accessible endpoints for testing
6. WHEN the infrastructure is deployed THEN the system SHALL include required IAM roles and permissions
7. WHEN the infrastructure is deployed THEN the system SHALL output relevant URLs or connection information
8. WHEN the infrastructure is deployed THEN the system SHALL be configured for high availability
9. WHEN the CloudFront distribution is deployed THEN the system SHALL use price class 100 to optimize costs while covering Europe and the United States edge locations
10. WHEN the CloudFront distribution is deployed THEN the system SHALL allow only GET, HEAD, and OPTIONS HTTP methods for security and performance optimization
11. WHEN the CloudFront distribution is deployed THEN the system SHALL configure caching policies based on query parameters to optimize cache efficiency
12. WHEN the CloudFront distribution is deployed THEN the system SHALL set the default TTL to 900 seconds (15 minutes) to align with static content caching requirements

If you prefer, you can write your requirements the way you are used to and then click Refine to have Kiro help you format them in EARS format, but I would say it’s good team practice to align to the EARS format.

Step 2: Define design

Switch to the design tab and click Refine to generate design specification based on the defined requirements, merged with the Agent Steering context configuration.

Then Kiro starts working.

Kiro has now populated design.md. When I take a closer look I don’t see anything addressing the Weather API Service Terms of Service rule: You must identify yourself (set custom HTTP User-Agent).

You can have Kiro help refine and add more requirements as you go along.

Now the design looks good to me, so let’s move on to generate the implementation plan.

<textarea tabindex="-1" aria-hidden="true" readonly># Weather Forecast App Design Document

## Overview

The weather forecast application is a serverless web application that displays tomorrow's weather forecast for four European cities: Oslo (Norway), Paris (France), London (United Kingdom), and Barcelona (Spain). The application will be deployed on AWS using Terraform infrastructure-as-code and will integrate with the Norwegian Meteorological Institute's weather API.

### Key Design Principles
- **Serverless-first architecture** for minimal operational overhead
- **Mobile-responsive design** for optimal user experience across devices
- **Fast response times** through efficient caching and CDN distribution
- **Well-architected AWS infrastructure** following security and reliability best practices

## Architecture

### High-Level Architecture
The application follows a serverless architecture pattern with the following components:

1. **Frontend** : Static web application hosted on S3 with CloudFront distribution
2. **Backend API** : AWS Lambda functions for weather data processing
3. **Data Layer** : DynamoDB for caching weather data and API rate limiting
4. **External Integration** : Norwegian Meteorological Institute API (api.met.no)

### Architecture Rationale
- **Static hosting with S3/CloudFront** : Provides fast global content delivery and handles traffic spikes efficiently
- **Lambda functions** : Serverless compute eliminates server management and scales automatically
- **DynamoDB** : NoSQL database perfect for caching weather data with TTL capabilities
- **API Gateway** : Provides managed API endpoints with built-in throttling and monitoring

## SNIP END</textarea>

# Weather Forecast App Design Document

## Overview

The weather forecast application is a serverless web application that displays tomorrow's weather forecast for four European cities: Oslo (Norway), Paris (France), London (United Kingdom), and Barcelona (Spain). The application will be deployed on AWS using Terraform infrastructure-as-code and will integrate with the Norwegian Meteorological Institute's weather API.

### Key Design Principles
- **Serverless-first architecture** for minimal operational overhead
- **Mobile-responsive design** for optimal user experience across devices
- **Fast response times** through efficient caching and CDN distribution
- **Well-architected AWS infrastructure** following security and reliability best practices

## Architecture

### High-Level Architecture
The application follows a serverless architecture pattern with the following components:

1. **Frontend** : Static web application hosted on S3 with CloudFront distribution
2. **Backend API** : AWS Lambda functions for weather data processing
3. **Data Layer** : DynamoDB for caching weather data and API rate limiting
4. **External Integration** : Norwegian Meteorological Institute API (api.met.no)

### Architecture Rationale
- **Static hosting with S3/CloudFront** : Provides fast global content delivery and handles traffic spikes efficiently
- **Lambda functions** : Serverless compute eliminates server management and scales automatically
- **DynamoDB** : NoSQL database perfect for caching weather data with TTL capabilities
- **API Gateway** : Provides managed API endpoints with built-in throttling and monitoring

## SNIP END

The design turned out to be grow into more than 300 lines, so I’m just including a snippet here. You can review the complete file here: .kiro/specs/weather-forecast-app/design.md

This looks good to me, let’s proceed to generate the implementation plan.

Step 3: Implement

Kiro has now generated a task list for implementation based on the previous steps.

.kiro/specs/weather-forecast-app/tasks.md now looks like this:

<textarea tabindex="-1" aria-hidden="true" readonly># Implementation Plan

- [] 1. Set up project structure and configuration
  - Create Terraform module directory structure following best practices
  - Set up Python virtual environment with pyenv for application development
  - Configure pre-commit hooks for Terraform validation and formatting
  - Create basic project documentation structure with docs/ directory
  - _Requirements: 3.1, 3.2_

- [] 2. Implement simplified Lambda weather service
  - [] 2.1 Create embedded weather service in Lambda handler
    - Implement weather data fetching directly in lambda_handler.py using urllib
    - Define city configuration with coordinates for Oslo, Paris, London, Barcelona
    - Create weather data processing and transformation logic embedded in handler
    - Implement proper User-Agent header with configurable company website
    - Add rate limiting with simple delays between API calls
    - _Requirements: 2.2, 2.3, 1.1_

  - [] 2.2 Implement weather API integration and processing
    - Create fetch_weather_data function for met.no API calls
    - Implement extract_tomorrow_forecast for parsing API responses
    - Add weather condition mapping and error handling
    - Create process_city_weather for individual city processing
    - Implement get_weather_summary for all cities with delay between calls
    - _Requirements: 2.2, 1.2_

- [] 3. Build Lambda function infrastructure
  - [] 3.1 Create simplified Lambda function handler
    - Implement main Lambda handler with embedded weather service
    - Add environment variable configuration for company website
    - Implement proper error handling and logging with standardized responses
    - Create /health endpoint for monitoring with environment information
    - Add CORS support and OPTIONS request handling
    - _Requirements: 2.1, 2.4, 3.6_

  - [] 3.2 Add DynamoDB caching to simplified Lambda handler
    - Implement DynamoDB caching directly in the lambda_handler.py file
    - Add cache check before API calls and cache storage after successful API responses
    - Implement 1-hour TTL (3600 seconds) for cached weather data
    - Add error handling for DynamoDB operations with fallback to API calls
    - Use boto3 client for DynamoDB operations embedded in the handler
    - _Requirements: 2.2, 1.2, 3.6_

- [] 4. Update Terraform infrastructure for simplified approach
  - [] 4.1 Maintain DynamoDB table configuration for caching
    - Keep existing DynamoDB table from Terraform backend module
    - Maintain DynamoDB-related IAM permissions for Lambda role
    - Ensure DynamoDB table name is passed to Lambda via environment variable
    - Keep TTL configuration for 1-hour cache expiration
    - Maintain existing tests for DynamoDB validation
    - _Requirements: 3.1, 3.6, 3.8_

  - [] 4.2 Update Lambda function Terraform module for simplified deployment
    - Update Terraform configuration for simplified Lambda function
    - Maintain DynamoDB environment variables (table name) and permissions
    - Keep COMPANY_WEBSITE environment variable configuration
    - Maintain X-Ray tracing and CloudWatch logging
    - Keep IAM role with DynamoDB permissions for caching
    - _Requirements: 3.1, 3.4, 3.6, 3.8_

  - [] 4.3 Maintain API Gateway configuration
    - Keep existing API Gateway REST API configuration
    - Maintain CORS settings and rate limiting
    - Keep Lambda integration with proper error handling
    - Maintain CloudWatch logging configuration
    - Keep existing tests for the API Gateway setup
    - _Requirements: 3.1, 3.5, 3.6_

- [] 5. Create frontend application
  - [] 5.1 Build responsive weather display components
    - Create React components for weather card display
    - Implement responsive grid layout for four cities
    - Add loading states and error handling UI
    - Ensure mobile-optimized design with proper breakpoints
    - _Requirements: 1.1, 1.2, 1.3, 2.1_

  - [] 5.2 Implement API integration and state management
    - Create API client for backend weather service
    - Implement data fetching with error handling and retries
    - Add browser-side caching strategy respecting 1-hour backend cache
    - Create loading and error state management
    - _Requirements: 1.2, 2.1, 2.2_

  - [] 5.3 Add weather icons and styling
    - Implement weather condition icon mapping
    - Create CSS styling for responsive design
    - Add animations and transitions for better UX
    - Ensure accessibility compliance (WCAG)
    - _Requirements: 1.1, 1.3, 2.1_

  - [] 5.4 Optimize frontend build for caching
    - Configure build process to generate static assets optimized for 15-minute caching
    - Ensure proper file naming and versioning for cache busting when needed
    - Validate that all static assets (HTML, CSS, JS, images) are properly configured
    - _Requirements: 1.2, 1.4_

- [] 6. Configure static hosting infrastructure
  - [] 6.1 Create S3 bucket for static hosting
    - Implement Terraform module for S3 bucket configuration
    - Configure bucket policies for static website hosting
    - Set up versioning and lifecycle policies
    - Add proper IAM permissions for deployment
    - Write basic tests for the S3 configuration
    - _Requirements: 3.1, 3.4, 3.6_

  - [] 6.2 Set up CloudFront distribution
    - Create Terraform module for CloudFront CDN
    - Configure cache behaviors and TTL settings
    - Set up origin failover for high availability
    - Add security headers and HTTPS redirection
    - Write basic tests for the Cloudfront configuration
    - _Requirements: 1.2, 3.1, 3.8_

  - [] 6.4 Configure CloudFront price class and optimization settings
    - Update CloudFront distribution to use price class 100 (PriceClass_100)
    - Configure allowed HTTP methods to GET, HEAD, and OPTIONS only
    - Set up caching policy configuration based on query parameters
    - Configure default TTL to 900 seconds (15 minutes)
    - Ensure coverage includes Europe and United States edge locations
    - Validate cost optimization while maintaining performance for target regions
    - Update Terraform configuration with appropriate price_class, allowed_methods, and caching parameters
    - Test CloudFront distribution functionality with new configuration
    - _Requirements: 3.9, 3.10, 3.11, 3.12_

  - [] 6.3 Configure Cache-Control headers for static content
    - Configure S3 bucket metadata to set Cache-Control: max-age=900 for all static assets
    - Update CloudFront cache behaviors to respect and forward Cache-Control headers
    - Ensure consistent 15-minute caching for HTML, CSS, JavaScript, and image files
    - _Requirements: 1.2, 1.4_

- [] 7. Implement monitoring and observability
  - [] 7.1 Create simple and intuitive CloudWatch dashboard and alarms
    - Implement Terraform module for CloudWatch dashboard
    - Configure the most important alarms for Lambda errors, API Gateway 5xx, and DynamoDB throttling
    - Set up custom metrics for weather API success rates
    - Add log retention policies (180 days)
    - _Requirements: 3.6, 3.7_

  - [] 7.2 Set up AWS Budget and cost monitoring
    - Create Terraform module for AWS Budget with Service tag filter
    - Configure budget alerts for cost thresholds
    - Implement simple and intuitive cost monitoring Cloudwatch dashboard
    - _Requirements: 3.3, 3.7_

- [] 8. Create deployment and testing automation
  - [] 8.1 Implement Terraform module packaging
    - Create main Terraform module with all sub-modules
    - Configure variable definitions and outputs
    - Add module documentation with terraform-docs
    - Create examples/ directory with usage examples
    - _Requirements: 3.1, 3.2, 3.7_

  - [] 8.2 Add basic integration and end-to-end tests
    - Create integration tests for complete weather data flow
    - Implement end-to-end tests for user journey with CloudWatch synthetics
    - Create basic infrastructure deployment tests
    - Write basic test automation scripts with cleanup
    - _Requirements: 2.4, 3.2_

  - [] 8.3 Add cache header validation tests
    - Create automated tests to verify Cache-Control headers are properly set
    - Test that static assets return max-age=900 in response headers
    - Validate cache behavior across different asset types (HTML, CSS, JS, images)
     - _Requirements: 1.4, 2.4_

  - [] 8.4 Fix CI/CD deployment path issues
    - Resolve frontend build path problems in CI/CD environments where working directory structure differs
    - Update Terraform frontend module to handle different working directory structures and missing directories
    - Add proper error handling and path validation for frontend build process
    - Ensure frontend directory and package.json are found correctly in CI/CD pipelines
    - Test build process works in both local development and CI/CD environments
    - _Requirements: 3.1, 3.4_

  - [] 8.5 Update unit tests for simplified Lambda implementation
    - Update existing unit tests to work with the simplified embedded Lambda handler
    - Remove tests for separate weather service modules (api_client, cache, processor, etc.)
    - Create focused tests for the main Lambda handler functions
    - Test weather data fetching, processing, response formatting, and DynamoDB caching
    - Ensure tests cover error handling, cache hits/misses, and edge cases
    - _Requirements: 2.4, 3.2_

  - [] 8.6 Add frontend error loop prevention safeguards
    - Implement circuit breaker pattern in useWeatherData hook to prevent infinite retry loops
    - Add exponential backoff with maximum delay caps for failed requests
    - Implement request rate limiting to prevent rapid successive API calls on errors
    - Add error threshold detection to disable auto-retry after consecutive failures
    - Create user-friendly error states that prevent automatic retry loops
    - _Requirements: 1.2, 2.1_

  - [] 8.7 Configure reasonable Lambda concurrency limits
    - Set Lambda reserved concurrency to 5 concurrent executions (reasonable for weather API)
    - Update backend module variables to reflect appropriate concurrency limits
    - Add documentation explaining concurrency limits and cost implications
    - Ensure concurrency limits prevent runaway costs while maintaining service availability
    - Test concurrency limits under load to ensure proper throttling behavior
    - _Requirements: 3.6, 3.8_

  - [] 8.8 Implement dynamic cache-control headers in Lambda function
    - Update Lambda handler to set cache-control: max-age=60 for successful weather API responses
    - Set cache-control: max-age=0 for failed weather API responses or error conditions
    - Ensure cache-control headers are properly included in HTTP response headers
    - Test cache-control behavior for both success and failure scenarios
    - _Requirements: 2.5, 2.6_

  - [] 8.9 Implement lastUpdated timestamp handling in Lambda function
    - Update Lambda handler to include lastUpdated timestamp in all API responses
    - Use weather API timestamp when available in the met.no API response
    - Fall back to DynamoDB cache timestamp when weather API timestamp is not provided
    - Ensure timestamp is in ISO 8601 format for consistent frontend display
    - Test timestamp handling for both fresh API calls and cached responses
    - _Requirements: 2.7, 2.8_

  - [] 8.10 Update frontend to display lastUpdated timestamp
    - Modify weather display components to show the lastUpdated timestamp from API responses
    - Format timestamp for user-friendly display (e.g., "Last updated: 2 minutes ago")
    - Handle cases where lastUpdated is null or missing
    - Ensure timestamp display is responsive and accessible
    - _Requirements: 2.7_

- [] 9. Generate documentation and cost analysis
  - [] 9.1 Create architecture diagrams
    - Generate AWS architecture diagram using MCP diagram server
    - Create sequence diagrams for weather data flow
    - Add deployment flow diagrams
    - Include diagrams in main README.md
    - _Requirements: 3.7_

  - [] 9.2 Perform cost analysis and optimization
    - Use AWS Labs Pricing MCP server for cost calculations
    - Compare costs across eu-west-1, eu-central-1, eu-north-1 regions
    - Create cost projections for staging and production environments
    - Document top three cost optimization opportunities
    - Include cost analysis in main README.md
    - _Requirements: 3.7_

- [] 10. Finalize project documentation
  - Create crisp and clear README.md with TL;DR section
  - Add executive summary for project stakeholders
  - Create basic deployment guide and troubleshooting documentation
  - Write operational runbooks for maintenance
  - Add basic examples for CI/CD integration and how to configure relevant variables
  - _Requirem</textarea>

# Implementation Plan

- [] 1. Set up project structure and configuration
  - Create Terraform module directory structure following best practices
  - Set up Python virtual environment with pyenv for application development
  - Configure pre-commit hooks for Terraform validation and formatting
  - Create basic project documentation structure with docs/ directory
  - _Requirements: 3.1, 3.2_

- [] 2. Implement simplified Lambda weather service
  - [] 2.1 Create embedded weather service in Lambda handler
    - Implement weather data fetching directly in lambda_handler.py using urllib
    - Define city configuration with coordinates for Oslo, Paris, London, Barcelona
    - Create weather data processing and transformation logic embedded in handler
    - Implement proper User-Agent header with configurable company website
    - Add rate limiting with simple delays between API calls
    - _Requirements: 2.2, 2.3, 1.1_

  - [] 2.2 Implement weather API integration and processing
    - Create fetch_weather_data function for met.no API calls
    - Implement extract_tomorrow_forecast for parsing API responses
    - Add weather condition mapping and error handling
    - Create process_city_weather for individual city processing
    - Implement get_weather_summary for all cities with delay between calls
    - _Requirements: 2.2, 1.2_

- [] 3. Build Lambda function infrastructure
  - [] 3.1 Create simplified Lambda function handler
    - Implement main Lambda handler with embedded weather service
    - Add environment variable configuration for company website
    - Implement proper error handling and logging with standardized responses
    - Create /health endpoint for monitoring with environment information
    - Add CORS support and OPTIONS request handling
    - _Requirements: 2.1, 2.4, 3.6_

  - [] 3.2 Add DynamoDB caching to simplified Lambda handler
    - Implement DynamoDB caching directly in the lambda_handler.py file
    - Add cache check before API calls and cache storage after successful API responses
    - Implement 1-hour TTL (3600 seconds) for cached weather data
    - Add error handling for DynamoDB operations with fallback to API calls
    - Use boto3 client for DynamoDB operations embedded in the handler
    - _Requirements: 2.2, 1.2, 3.6_

- [] 4. Update Terraform infrastructure for simplified approach
  - [] 4.1 Maintain DynamoDB table configuration for caching
    - Keep existing DynamoDB table from Terraform backend module
    - Maintain DynamoDB-related IAM permissions for Lambda role
    - Ensure DynamoDB table name is passed to Lambda via environment variable
    - Keep TTL configuration for 1-hour cache expiration
    - Maintain existing tests for DynamoDB validation
    - _Requirements: 3.1, 3.6, 3.8_

  - [] 4.2 Update Lambda function Terraform module for simplified deployment
    - Update Terraform configuration for simplified Lambda function
    - Maintain DynamoDB environment variables (table name) and permissions
    - Keep COMPANY_WEBSITE environment variable configuration
    - Maintain X-Ray tracing and CloudWatch logging
    - Keep IAM role with DynamoDB permissions for caching
    - _Requirements: 3.1, 3.4, 3.6, 3.8_

  - [] 4.3 Maintain API Gateway configuration
    - Keep existing API Gateway REST API configuration
    - Maintain CORS settings and rate limiting
    - Keep Lambda integration with proper error handling
    - Maintain CloudWatch logging configuration
    - Keep existing tests for the API Gateway setup
    - _Requirements: 3.1, 3.5, 3.6_

- [] 5. Create frontend application
  - [] 5.1 Build responsive weather display components
    - Create React components for weather card display
    - Implement responsive grid layout for four cities
    - Add loading states and error handling UI
    - Ensure mobile-optimized design with proper breakpoints
    - _Requirements: 1.1, 1.2, 1.3, 2.1_

  - [] 5.2 Implement API integration and state management
    - Create API client for backend weather service
    - Implement data fetching with error handling and retries
    - Add browser-side caching strategy respecting 1-hour backend cache
    - Create loading and error state management
    - _Requirements: 1.2, 2.1, 2.2_

  - [] 5.3 Add weather icons and styling
    - Implement weather condition icon mapping
    - Create CSS styling for responsive design
    - Add animations and transitions for better UX
    - Ensure accessibility compliance (WCAG)
    - _Requirements: 1.1, 1.3, 2.1_

  - [] 5.4 Optimize frontend build for caching
    - Configure build process to generate static assets optimized for 15-minute caching
    - Ensure proper file naming and versioning for cache busting when needed
    - Validate that all static assets (HTML, CSS, JS, images) are properly configured
    - _Requirements: 1.2, 1.4_

- [] 6. Configure static hosting infrastructure
  - [] 6.1 Create S3 bucket for static hosting
    - Implement Terraform module for S3 bucket configuration
    - Configure bucket policies for static website hosting
    - Set up versioning and lifecycle policies
    - Add proper IAM permissions for deployment
    - Write basic tests for the S3 configuration
    - _Requirements: 3.1, 3.4, 3.6_

  - [] 6.2 Set up CloudFront distribution
    - Create Terraform module for CloudFront CDN
    - Configure cache behaviors and TTL settings
    - Set up origin failover for high availability
    - Add security headers and HTTPS redirection
    - Write basic tests for the Cloudfront configuration
    - _Requirements: 1.2, 3.1, 3.8_

  - [] 6.4 Configure CloudFront price class and optimization settings
    - Update CloudFront distribution to use price class 100 (PriceClass_100)
    - Configure allowed HTTP methods to GET, HEAD, and OPTIONS only
    - Set up caching policy configuration based on query parameters
    - Configure default TTL to 900 seconds (15 minutes)
    - Ensure coverage includes Europe and United States edge locations
    - Validate cost optimization while maintaining performance for target regions
    - Update Terraform configuration with appropriate price_class, allowed_methods, and caching parameters
    - Test CloudFront distribution functionality with new configuration
    - _Requirements: 3.9, 3.10, 3.11, 3.12_

  - [] 6.3 Configure Cache-Control headers for static content
    - Configure S3 bucket metadata to set Cache-Control: max-age=900 for all static assets
    - Update CloudFront cache behaviors to respect and forward Cache-Control headers
    - Ensure consistent 15-minute caching for HTML, CSS, JavaScript, and image files
    - _Requirements: 1.2, 1.4_

- [] 7. Implement monitoring and observability
  - [] 7.1 Create simple and intuitive CloudWatch dashboard and alarms
    - Implement Terraform module for CloudWatch dashboard
    - Configure the most important alarms for Lambda errors, API Gateway 5xx, and DynamoDB throttling
    - Set up custom metrics for weather API success rates
    - Add log retention policies (180 days)
    - _Requirements: 3.6, 3.7_

  - [] 7.2 Set up AWS Budget and cost monitoring
    - Create Terraform module for AWS Budget with Service tag filter
    - Configure budget alerts for cost thresholds
    - Implement simple and intuitive cost monitoring Cloudwatch dashboard
    - _Requirements: 3.3, 3.7_

- [] 8. Create deployment and testing automation
  - [] 8.1 Implement Terraform module packaging
    - Create main Terraform module with all sub-modules
    - Configure variable definitions and outputs
    - Add module documentation with terraform-docs
    - Create examples/ directory with usage examples
    - _Requirements: 3.1, 3.2, 3.7_

  - [] 8.2 Add basic integration and end-to-end tests
    - Create integration tests for complete weather data flow
    - Implement end-to-end tests for user journey with CloudWatch synthetics
    - Create basic infrastructure deployment tests
    - Write basic test automation scripts with cleanup
    - _Requirements: 2.4, 3.2_

  - [] 8.3 Add cache header validation tests
    - Create automated tests to verify Cache-Control headers are properly set
    - Test that static assets return max-age=900 in response headers
    - Validate cache behavior across different asset types (HTML, CSS, JS, images)
     - _Requirements: 1.4, 2.4_

  - [] 8.4 Fix CI/CD deployment path issues
    - Resolve frontend build path problems in CI/CD environments where working directory structure differs
    - Update Terraform frontend module to handle different working directory structures and missing directories
    - Add proper error handling and path validation for frontend build process
    - Ensure frontend directory and package.json are found correctly in CI/CD pipelines
    - Test build process works in both local development and CI/CD environments
    - _Requirements: 3.1, 3.4_

  - [] 8.5 Update unit tests for simplified Lambda implementation
    - Update existing unit tests to work with the simplified embedded Lambda handler
    - Remove tests for separate weather service modules (api_client, cache, processor, etc.)
    - Create focused tests for the main Lambda handler functions
    - Test weather data fetching, processing, response formatting, and DynamoDB caching
    - Ensure tests cover error handling, cache hits/misses, and edge cases
    - _Requirements: 2.4, 3.2_

  - [] 8.6 Add frontend error loop prevention safeguards
    - Implement circuit breaker pattern in useWeatherData hook to prevent infinite retry loops
    - Add exponential backoff with maximum delay caps for failed requests
    - Implement request rate limiting to prevent rapid successive API calls on errors
    - Add error threshold detection to disable auto-retry after consecutive failures
    - Create user-friendly error states that prevent automatic retry loops
    - _Requirements: 1.2, 2.1_

  - [] 8.7 Configure reasonable Lambda concurrency limits
    - Set Lambda reserved concurrency to 5 concurrent executions (reasonable for weather API)
    - Update backend module variables to reflect appropriate concurrency limits
    - Add documentation explaining concurrency limits and cost implications
    - Ensure concurrency limits prevent runaway costs while maintaining service availability
    - Test concurrency limits under load to ensure proper throttling behavior
    - _Requirements: 3.6, 3.8_

  - [] 8.8 Implement dynamic cache-control headers in Lambda function
    - Update Lambda handler to set cache-control: max-age=60 for successful weather API responses
    - Set cache-control: max-age=0 for failed weather API responses or error conditions
    - Ensure cache-control headers are properly included in HTTP response headers
    - Test cache-control behavior for both success and failure scenarios
    - _Requirements: 2.5, 2.6_

  - [] 8.9 Implement lastUpdated timestamp handling in Lambda function
    - Update Lambda handler to include lastUpdated timestamp in all API responses
    - Use weather API timestamp when available in the met.no API response
    - Fall back to DynamoDB cache timestamp when weather API timestamp is not provided
    - Ensure timestamp is in ISO 8601 format for consistent frontend display
    - Test timestamp handling for both fresh API calls and cached responses
    - _Requirements: 2.7, 2.8_

  - [] 8.10 Update frontend to display lastUpdated timestamp
    - Modify weather display components to show the lastUpdated timestamp from API responses
    - Format timestamp for user-friendly display (e.g., "Last updated: 2 minutes ago")
    - Handle cases where lastUpdated is null or missing
    - Ensure timestamp display is responsive and accessible
    - _Requirements: 2.7_

- [] 9. Generate documentation and cost analysis
  - [] 9.1 Create architecture diagrams
    - Generate AWS architecture diagram using MCP diagram server
    - Create sequence diagrams for weather data flow
    - Add deployment flow diagrams
    - Include diagrams in main README.md
    - _Requirements: 3.7_

  - [] 9.2 Perform cost analysis and optimization
    - Use AWS Labs Pricing MCP server for cost calculations
    - Compare costs across eu-west-1, eu-central-1, eu-north-1 regions
    - Create cost projections for staging and production environments
    - Document top three cost optimization opportunities
    - Include cost analysis in main README.md
    - _Requirements: 3.7_

- [] 10. Finalize project documentation
  - Create crisp and clear README.md with TL;DR section
  - Add executive summary for project stakeholders
  - Create basic deployment guide and troubleshooting documentation
  - Write operational runbooks for maintenance
  - Add basic examples for CI/CD integration and how to configure relevant variables
  - _Requirem

In the IDE you will see an option to trigger to start tasks:

You can either trigger them one by one or ask Kiro in the chat to get started.

Let’s start the first task.

We can see that Kiro is setting up the structure according to what’s stated on the external website terraform-best-practices.com, as requested. Nice.

Like with Amazon Q Developer, you can approve or let Kiro trust specific tools and commands:

After a short while the first task is complete!

I did note that Terraform AWS provider version 5 was installed. The latest one is major version 6.7.0, so I updated the Tech Steering specification and asked Kiro to refresh.

Kiro refreshed the context guidelines, found what to update, performed the changes and ran checks and pre-commit to verify it’s working as expected.

Then we move on to Task 2: Implement core Python service, and so on.

This is how the Summary of implementation looks like for task 7.

As you can see the main difference between the traditional vibe/CLI approach is that the spec-driven workflow keeps the steering and requirements up to date, to persist the context. This makes the process a lot more predictable, and possible to collaborate on in a team.

Keeping the specifications version controlled along with the application codebase makes it easy to track changes as changes are committed.

Depending on your team development workflow, each feature could be organized as a Spec, containing relevant user stories.

Kiro generated a comprehensive local testing suite. It turned out to be more complex than I think is necessary, with some tests being flaky. I asked Kiro to focus on testing the core functionality and remove brittle and complex tests. A reflection here is that I did not specify the desired test approach in detail in my steering context.

Here are the diagrams the AWS Diagrams MCP server helped create:

Deploying the final solution

I included the module definition in my existing Github Actions CI/CD Terraform codebase:

<textarea tabindex="-1" aria-hidden="true" readonly>module "weather_forecast_app" {
  source = "git::https://github.com/haakond/terraform-aws-weather-forecast.git?ref=COMMIT-SHA"
  project_name = "weather-forecast-app"
  environment = "prod"
  aws_region = "eu-west-1"
  weather_service_identification_domain = "youramazingwebsite.com"
}</textarea>

module "weather_forecast_app" {
  source = "git::https://github.com/haakond/terraform-aws-weather-forecast.git?ref=COMMIT-SHA"
  project_name = "weather-forecast-app"
  environment = "prod"
  aws_region = "eu-west-1"
  weather_service_identification_domain = "youramazingwebsite.com"
}

I experienced a few Terraform errors that Kiro wasn’t able to catch before terraform plan and apply. Issues:

CloudWatch Logs groups defined with the same name in two sub-modules
Missing API Gateway Account configuration for CloudWatch Logs
Missing deploy process for the React frontend to Amazon S3
- Since this is fairly static app I decided to keep it along with the infrastructure code, for simplicity.
- For production frontend applications I would set up a dedicated CI/CD pipeline to deploy only the frontend codebase.
An overly complex Reach frontend application, which I asked Kiro to simplify.

With some additional prompting assistance from Kiro I was able to resolve the issues and end up with a fully working deployment!

End result:

Workflow for adding a new feature

Here’s one possible suggested approach:

git clone into a new feature branch
Specify
- For a major new feature: create a new Specification (requirements, design, tasks)
- For a minor improvement: Incorporate into an existing Specification
Implement requirements
Create pull request
Review and merge to main

Learnings and key takeaways

If you do manual changes outside of Kiro, the specs (Requirements, Design, Tasks) will deviate and confuse Kiro. Stick to the Kiro workflow, and ask Kiro to refresh what you did. Kiro will backport into the specs.
- If during the preview period you keep hitting Kiro’s usage limit, a workaround can be to get Amazon Q Developer help you when troubleshooting, to save interaction tokens. Just make sure that you tell Kiro which areas has changed to get the specs up-to-date.
Separate product feature specifications from common steering context.
Kiro has a tendency to add more comprehensive and complex testing procedures and documentation than a human would appreciate. Be explicit about keeping things simple and focus on the core functionality.
To ensure the suggested tests are relevant and follows your company practice, add a detailed Agent Steering document for testing.
Organize specs by feature, to be able to work independently without conflicts or affecting other areas. This can also reduce the blast radius in case of unexpected changes, plus it reduces the context size for the agent.
Keep specs and user stories in version control along with your application. If you perform traditional manual changes, give Kiro a hint to have the design and requirements updated accordingly.

Reflections on context

Traditionally, the environment specific context are a company’s tech stack, policies and guidelines combined with the experience of experienced software engineers. This is information known and acquired in your setting and is normally not included in (Jira) User Stories. However, coding agents by default do not posess this context information. The closest thing may be the possibility to configure Amazon Q Developer with custom repositories, so that it can learn about company specific coding standards, libraries etc.

Spec driven development with Kiro now forces this context information to be defined as Agent Steering resources. Teams can organize workshops to document their guiding tenets, principles, organized in Git repo and iterated as the information evolves. Perhaps your team have something similar documented in a company wiki already?

I think we need to help AI coding companions build the same mental framework we’d give to a human colleague during onboarding and code review. Kiro solves this by checking in your specifications to Git. Consider creating a Kiro app bootstrap repository or include common steering as a Git submodule, package reference or similar.

Kiro pricing

When Kiro becomes Generally Available, there will be different tiers available to match your level of usage.

Vibe Requests cover any agentic operation in Kiro that does not involve execution of a Spec Task.

You start with Vibe requests to create requirements, design documents and tasks.

One Vibe Request typically equals one message or prompt, while one Spec Requests equals executing a single Spec Task.

For more information see https://kiro.dev/blog/understanding-kiro-pricing-specs-vibes-usage-tracking/.

Conclusion

My personal experience is that I really appreciate the spec-driven development process. Kiro is a game-changer which can significantly boost what builders are able to produce. There are no more valid excuses for avoiding sufficient test coverage or struggling with even a nice looking frontend app!

Kiro is not just a new tool, it is a new workflow; AI-assisted, spec-driven development which can incorporate mature engineering practices, and my initial evaluation leaves me thinking that AI is starting to grow up and become more professional. As a bonus, Kiro forces you to write better specifications, which can make it easier to establish a common alignment in a team, and while onboarding new team members.

This approach is a giant leap forward compared to coding assistants pre H2 2025. Project Rules and Customizations in Amazon Q Developer were a step in the right direction, but Kiro brings a sought-after structure and consistency that in my opinion not only is nice, but necessary, in a professional context. Yes, there are still some bugs and quirks (Kiro is at time of writing still in limited preview) but I am optimistic that this technology and the underlying Large Language Models will mature and produce results of increasingly higher quality and predictability over the coming months. Earlier, my experience was that code assistant suggestions could get me around ~70% up to speed, with ~30% traditional authoring for preciseness. Kiro boosts this to maybe ~85%+. I would claim that return on investment on the Kiro license can be achieved pretty fast.

I would still prefer a knowledgeable human in the loop reviewing Pull Requests and making the final decision for changes, that is until Agentic AI has matured a bit more. It will be exciting to see how this field evolves during the next couple of years.

I encourage you to try out spec-driven development with your colleagues and warmly welcome Kiro as your new team member.

Resources

The post How to use Kiro for AI assisted spec-driven development first appeared on Håkon Eriksen Drange.

Extensive reporting of Well-Architected Maturity

Håkon Eriksen Drange — Sun, 20 Jul 2025 07:59:45 +0000

Introduction
How to generate a Well-Architected Compliance report
What a Well-Architected Framework Compliance report looks like
How the reporting feature works
Conclusion
Feedback and contributions
Resources

Introduction

In the post How to measure Well-Architected maturity we explored how extensive insight into cloud infrastructure posture could help accelerate the Measure phase, leaving more time to Learn and discuss opportunities for Improvement.

Figure – Well-Architected Framework review cycle courtesy of AWS

A key factor was to make the data available while performing a review in the Well-Architected Tool. After discussing the solution with colleagues and customers I realized that the valuable data points weren’t exposed to their full potential within only the Notes field in the Well-Architected Tool. Key constraints are that the Notes field is limited to plain text and a maximum of 2000 characters. Valuable resource identifiers and tags had to be capped, which made it harder to easily identify the applicable resources. Could there be a better way?

I decided to develop an additional AWS Lambda function called well_architected_report_generator. The main purpose of this Lambda function is to collect all the available data points from various sources and generate a report in HTML format stored in Amazon Simple Storage Service (S3). When performing Well-Architected Framework Reviews, you are most likely already logged in to the AWS Console to access the Well-Architected Tool, so opening an additional tab with S3 could be useful. Thanks to Amazon Q Developer the HTML and CSS came out pretty good, too!

Current data sources:

AWS Config Conformance Packs
AWS Trusted Advisor checks (available checks depends on active AWS Support Plan)
Resource Tag: Name

How to generate a Well-Architected Compliance report

Deploy the Terraform module as also described in How to measure Well-Architected Maturity.

Please note that three new (optional) Terraform module variables have been introduced:

<textarea tabindex="-1" aria-hidden="true" readonly>variable "deploy_aws_config_recorder" {
  description = "Set to true to deploy an AWS Config Recorder. If you already have a customer managed AWS Config recorder in the desired region, set to false. AWS supports only one customer managed configuration recorder for each account for each AWS Region."
  type = bool
  default = true
}

variable "reports_bucket_name_prefix" {
  description = "Prefix for the S3 bucket name that stores Well-Architected compliance reports"
  type = string
  default = "well-architected-compliance-reports"
}

variable "reports_retention_days" {
  description = "Number of days to retain non-current versions of reports in the S3 bucket"
  type = number
  default = 90
}</textarea>

variable "deploy_aws_config_recorder" {
  description = "Set to true to deploy an AWS Config Recorder. If you already have a customer managed AWS Config recorder in the desired region, set to false. AWS supports only one customer managed configuration recorder for each account for each AWS Region."
  type = bool
  default = true
}

variable "reports_bucket_name_prefix" {
  description = "Prefix for the S3 bucket name that stores Well-Architected compliance reports"
  type = string
  default = "well-architected-compliance-reports"
}

variable "reports_retention_days" {
  description = "Number of days to retain non-current versions of reports in the S3 bucket"
  type = number
  default = 90
}

At the time of writing AWS only supports one customer managed AWS Config recorder in each region. If you already have one, set this value to false, for re-use.

If you would like to retain the reports for longer than the default value of 90 days you may set the desired value accordingly.

A minimal example module call if you already have a customer managed AWS Config recorder in your region and you prefer to retain the report files for 400 days may look like this:

<textarea tabindex="-1" aria-hidden="true" readonly>module "well_architected_config_conformance_pack" {
  source = "git::https://github.com/soprasteria/terraform-aws-wellarchitected-conformance.git?ref=&lt;DESIRED-COMMIT-SHA&gt;"
  deploy_aws_config_recorder = false
  reports_retention_days = 400
}</textarea>

module "well_architected_config_conformance_pack" {
  source = "git::https://github.com/soprasteria/terraform-aws-wellarchitected-conformance.git?ref=<DESIRED-COMMIT-SHA>"
  deploy_aws_config_recorder = false
  reports_retention_days = 400
}

Wait 24 hours for data to be collected and aggregated.

Then, in the AWS Console, go to AWS Lambda and locate the function well_architected_report_generator.

Create a new Test function with a JSON payload of workload_id for the Well-Architected Tool (not the complete ARN, just the last part) and dry_run. Example payload:

<textarea tabindex="-1" aria-hidden="true" readonly>{
  "workload_id": "141970ea95fd5b4329cyh05502659f39",
  "dry_run": 0
}</textarea>

{
  "workload_id": "141970ea95fd5b4329cyh05502659f39",
  "dry_run": 0
}

Hit Test. Execution may take a minute or two, depending on the amount of AWS infrastructure resources deployed in the AWS account.

The Cloud Watch Logs output will let you know which AWS Support plan is detected and where you can find the produced report.

Starting Well-Architected Report Generator
Collecting compliance data from AWS Config
Trusted Advisor compliance status mapping
AWS Business or Enterprise Support is/is not enabled
Successfully uploaded report to s3://well-architected-compliance-reports-123456789012/Reports/well_architected_compliance_report_%timestamp%.html

Navigate to Amazon S3, find the most recent report by sorting on the Last modified column, check the box to the left of Name and click Open to access the report in a new browser tab.

What a Well-Architected Framework Compliance report looks like

The produced report contains information about the time of generation, the AWS Account ID, region and detected AWS Support plan.

Example report 1, an AWS account with Basic Support:

Example report 2, an AWS account with Enterprise Support:

The report may contain a lot of detailed information, so an Executive Summary is provided as a high-level overview.

Example report 1:

Example report 2:

Next a Table of Contents is provided which includes an overview of the available Well-Architected Framework pillars and questions.

Each question includes relevant context information from the Well-Architected Framework and link to further guidance.

Here is the second question in the Security pillar: “How do you manage identifies for people and machines?”

For AWS Accounts with available AWS Premium Support, relevant Trusted Advisor check information is included.

Relevant AWS Config checks and their status are displayed, along with the detected Resource Type, Resource ID and Tag Name, if available, for easy identification and discussion.

For Reliability Pillar question number 9, Trusted Advisor checks indicates that RDS backups are enabled for all clusters, but there is at least one S3 bucket where replication is not configured.

The next section lists all detected AWS resources, including Resource ID, Status and Tag Name, as available (certain information has been obfuscated on purpose).

Moving to Cost Optimization pillar question number three: “How do you monitor cost and usage?” actually has no official Trusted Advisor checks associated, but the Terraform module fills the gap with custom AWS Config checks.

Based on this we can easily see that:

Cost Anomaly Detection is configured with a Cost monitor.
There is at least one AWS Budget configured with alert subscriptions.
There are no EC2 instances not in any Auto Scaling Groups.
The AWS account is a member of AWS Organizations and a Tag Policy is in effect.

How the reporting feature works

Let’s take a closer look into the underlying logic of the well_architected_report_generator Lambda function.

Diagram created by the help of AWS Diagram and Documentation MCP Servers

After the Terraform module is deployed AWS Config needs 24 hours to collect and aggregate all data points. Then when the Lambda function is triggered, the following happens:

The starting point is a Workload in the Well-Architected Tool. It’s not necessary to answer all questions first, you can do that with your team after the initial report has been generated. Every question in each pillar will then be mapped to relevant AWS Config and Trusted Advisor checks. Note: There is currently not a 100% 1-1 mapping with AWS Config and Trusted Advisor checks, but the availability may increase going forward as more custom checks are added to the Terraform module (contributions are welcome) and AWS Trusted Advisor.
Compliance status and information for all mapped checks are collected and aggregated. Resources in scope are fetched along with Tag Name, as available. Contextual information and further guidance is retrieved.
Scores and percentages are calculated.
Data and information are grouped by pillar.
The Executive Summary is generated.
The report is produced based on Python Jinja2 templating functionality and uploaded to a dedicated bucket on Amazon S3.

This functionality is part of the Terraform module in the file wa_report_generator.tf which includes the AWS Lambda function, a dedicated AWS KMS Key for encrypting the compliance reports (as resource and account information may be considered as sensitive) and a dedicated Amazon S3 bucket.

Conclusion

The solution now provides even more valuable insight to accelerate Well-Architected Framework Review conversations, providing more time to discuss opportunities for improvement. It fills some gaps if you don’t have AWS Premium Support available, and adds additional value otherwise. It can easily be deployed in existing Terraform pipelines. Please note that all resources will be cleaned up and deleted upon module call removal, including the reports, so make sure you remember REL09 and back up relevant reports accordingly.

Feedback and contributions

If you have any feedback please let me know through your preferred medium of contact.

If you would like to contribute with bugfixes, additional functionality or check coverage, pull requests are welcome!

Resources

AWS Config
AWS Config Conformance Packs
AWS Well-Architected Framework
AWS Well-Architected Tool
GitHub: Terraform module terraform-aws-wellarchitected-conformance
Development was accelerated by Amazon Q Developer
AWS Diagram MCP Server
AWS Documentation MCP Server

The post Extensive reporting of Well-Architected Maturity first appeared on Håkon Eriksen Drange.

How to measure Well-Architected maturity?

Håkon Eriksen Drange — Tue, 15 Apr 2025 08:14:40 +0000

TABLE OF CONTENTS

Introduction
Challenge
Measuring Well-Architected maturity with Terraform and AWS Config
- Functional flow of the solution
- Conceptual AWS architecture diagram
How to deploy and utilize
- Viewing measurement insights in AWS Console
- Well-Architected Tool integration
- Event JSON examples for dry_run/live mode
- Event JSON for cleaning notes fields for all questions
- Notice about compliance checks and automation
- Cost of AWS Config evaluations
How to remove and decommission after use
Behind the scenes
- AWS Config resources
- AWS Config Conformance Packs
Feedback and contributions
Resources

Introduction

“I always think that you should be asking yourself:

Are you Well-Architected?”

Dr. Werner Vogels, AWS re:Invent keynote 2018

Being Well-Architected means that you have taken care of the basics and your foundation is solid, so that you can move fast and focus on business requirements, with decreased risk of surprises. But how can we answer Werner’s question, with confidence? How can we measure Well-Architected maturity in a tangible manner? And what is Well-Architected enough, in your project phaze?

As the first part of the journey I would suggest to plan and conduct a Well-Architected Framework Review; to have a conversation about your solution architecture and how the different best practices from AWS could apply in your context (Measure).

In most review conversations I observe teams are spending a substantial amount of time trying to understand if the designed or provisioned resources are meeting the AWS recommended best practice configurations. “Did we set up alerting for this scenario?”, “Did we configure encryption at rest for the database cluster?”, “Did anyone get any alerts about spiking costs?”, “Is our documentation still up to date?” and so on.

During a conversation, spending less efforts on Measure could provide us with more time to Learn about best practices and align on opportunities for Improvement. Personally, I don’t believe that automation and AI capabilities will fully replace the WAFR lifecycle, but they may help accelerate them, reducing Mean-Time To Deployed Improvement for your users (MTTDI) [yes, I just invented that term].

Figure – Well-Architected Framework review cycle courtesy of AWS

Challenge

You could use a variety of options for measuring if cloud resources are meeting AWS best practices. Most commonly:

AWS Security Hub with the AWS Foundational Security Best Practices control reference (highly recommended).
AWS Trusted Advisor (full set of checks requires Business or Enterprise Support from AWS).
3rd party open-source tools such as Prowler and Steampipe.
3rd party SaaS vendors offering similar functionality in APM/Observability services.

Some of these options may not be available during a Well-Architected Framework Review due to company policies on changes potentially affecting the entire AWS Organization, cost development, security validations or procurement.

But, what if AWS native technology, provisioned for a limited time period, would be acceptable? In this article I will share an possible approach to measure Well-Architected maturity in the form of AWS Config Conformance packs.

Measuring Well-Architected maturity with Terraform and AWS Config

Recently I have been working on a Terraform module which can be utilized in scenarios with constraints as described above. A particular reflection I’ve made is that most tools focus primarily on security and reliability. Some dedicated offerings focus solely on Cloud Financial Management and Cost Optimization (also called FinOps), but finding one complete COTS Solution To Rule Them All (that doesn’t charge a premium) is unlikely.

If we wrap up our sleeves and develop our own solution supporting our own custom logic we could also cover other aspects such as Cost Optimization.

This Terraform module deploys AWS Config Conformance Packs mapped to pillars in the Well-Architected Framework.

For relevant pillars in the AWS Well-Architected Framework, each best practice that is specific enough to be detected will report to be COMPLIANT or NON_COMPLIANT. Some best practices are harder to measure, or up to subjective consideration if a team is happy with how things are, or if the team considers there is room for improvement:

How a team evaluates culture and priorities.
How satisfied a team is with insight into their workload(s) or business continuity and disaster recovery planning.
How to practice cloud financial management.

Best practices in Operational Excellence are not straight forward to detect, as implementation of observability may have subjective opinion on room for improvement or may be performed with 3rd party tools. The main outcome of this module is to accelerate the Well-Architected Framework Review conversation, not to replace it with automation. Our hope is to shift the focus from “how did we configure this?” to “this is where we are today, what could we do to improve?”, thus freeing up valuable time for busy teams.

In addition, the Notes field in the Well-Architected Tool can be populated directly with AWS Config resource compliance check results, leaving you with more insight to discuss improvement actions.

Functional flow of the solution

Figure – Flow sequence accelerating WAFR conversations

Conceptual AWS architecture diagram

AWS Config Configuration Recorder • Records configuration changes for resources in your local AWS account (no impact or dependencies on AWS Organizations Config recorder(s)) • Set to record either daily or continuously (configurable) • Stores configuration snapshots in a dedicated Amazon S3 bucket
Amazon S3 Bucket • Stores AWS Config configuration snapshots • Stores CloudFormation templates for conformance packs • Encrypted with a dedicated KMS key
AWS Config Conformance Packs • Well-Architected-Security • Well-Architected-Reliability • Well-Architected-Cost-Optimization • Well-Architected-IAM (optional, subset of Security checks)
Custom Lambda Functions • Cost Optimization checks: • Account structure implementation • AWS Budgets configuration • AWS Cost Anomaly Detection • Organization information in cost and usage • EC2 instances without Auto Scaling Groups
Well-Architected Tool Updater Lambda function • Retrieves compliance data from AWS Config • Maps compliance results to specific Well-Architected Framework best practices • Updates Notes fields in Well-Architected Tool

How to deploy and utilize

At least two days before your planned review, deploy the module as suggested in examples/main.tf and described below. Compliance checks will update on a daily basis, to optimize costs for AWS Config Evaluations.
Right before the review, trigger the Lambda function well_architected_tool_updater to update the Well-Architected Tool workload notes sections based on AWS Config Conformance packs compliance status.
Run the review, look to the data in the notes field for discussion. No checked/answered questions will be modified, that would be up to subjective evaluation.

provider "aws" {
  region = "eu-west-1" # Change to your preferred region
}

module "well_architected_conformance" {
  source = "git::https://github.com/soprasteria/terraform-aws-wellarchitected-conformance.git?ref=c006f439fc07d2e898cc7f67c5e7bcad1dcbd2e8"

  # AWS Config recording configuration
  recording_frequency = "DAILY" # Use DAILY to reduce costs

  # Deploy conformance packs
  deploy_security_conformance_pack = true
  deploy_reliability_conformance_pack = true
  deploy_cost_optimization_conformance_pack = true
  deploy_iam_conformance_pack = true
}

Viewing measurement insights in AWS Console

Navigating to AWS Config – Conformance packs will present a dashboard with packs for the Security, Reliability and Cost Optimization Pillars by default, plus IAM for Identity and Access Management, if enabled.

You can view the compliance score trend for each pillar/pack:

You can also view the compliance status for each check, prefixed with the related best practice question, mapped to the AWS Well-Architected Framework whitepaper.

Well-Architected Tool integration

This module can also automatically update Well-Architected Tool workloads with compliance data from the AWS Config Conformance Packs.

The Lambda function well_architected_tool_updater will:

Process each conformance pack (Security, Reliability, Cost Optimization).
Loop through all rules in sequence (SEC01, SEC02, REL01, REL02, COST01, etc.).
For each rule, list the resource type, resource ID, and compliance status in the Notes field of the corresponding best practice question of your Well-Architected Tool workload.
- The notes field has a limitation of maximum 2084 characters. When more resources are discovered than there is room for, resources will be summarized.
Overwrite old data if triggered more than once.
If you would like to erase all contents in all notes field, set the clean_notes input parameter to 1.

The source code for the Lambda function is located in the src/wa_tool_updater directory.

To trigger the Well-Architected Tool updater, go to Well-Architected Tool and extract the Workload ID (not the full resource ARN).

Then go to AWS Lambda and find the function well_architected_tool_updater. Create test event JSON definition as follows (Console or CLI):

Event JSON examples for dry_run/live mode

Extract the Well-Architected Tool Workload ID from Properties – ARN. This example with dry_run = 1 will find relevant compliance data and log to CloudWatch Logs. No changes or updates will be performed.

{
  "workload_id": "141970ea95fd5b4329cea05202659f39",
  "dry_run": 1,
  "clean_notes": 0
}

Flipping dry_run = 0 will perform updates of the notes field. No checked/answered questions will be modified.

{
  "workload_id": "141970ea95fd5b4329cea05202659f39",
  "dry_run": 0,
  "clean_notes": 0
}

Event JSON for cleaning notes fields for all questions

If you end up with a lot of mess and would like a fresh start, setting clean_notes to 1 will clean the notes field for all questions and return. No further changes to checked/answered questions or compliance data updates will be performed.

{
  "workload_id": "141970ea95fd5b4329cea05202659f39",
  "dry_run": 1,
  "clean_notes": 1
}

Expected output is as follows. Full log output is available in Cloudwatch Logs.

Back in Well-Architected Tool, the notes field will now be updated with detected compliance for SEC 4. How do you detect and investigate security events?

Notice about compliance checks and automation

Check data is based on all resources in the current AWS account. Tagging based filtering is currently not supported. Be aware if you have multiple workloads in the same AWS account.

Cost of AWS Config evaluations

According to the AWS Config pricing page; “With AWS Config, you are charged based on the number of configuration items recorded, the number of active AWS Config rule evaluations, and the number of conformance pack evaluations in your account. A configuration item is a record of the configuration state of a resource in your AWS account. An AWS Config rule evaluation is a compliance state evaluation of a resource by an AWS Config rule in your AWS account. A conformance pack evaluation is the evaluation of a resource by an AWS Config rule within the conformance pack”.

AWS Config supports Continuous recording and Daily recording. You can choose between Daily or Continuous by setting the desired value for the variable recording_frequency, which defaults to DAILY.

How to remove and decommission after use

Some might see this solution as valuable long-term, others might have other tools coming in which overlaps.

As this Terraform module deploys an S3 bucket for storing Config evaluations, the bucket must be emptied before it can be deleted.

Empty the bucket in the AWS Console.
Remove the Terraform module call declaration from your code base.
Trigger your CI/CD pipeline.

Behind the scenes

Some Terraform snippets on how to deploy an AWS Config Conformance Pack

How to write a custom AWS Config check.

AWS Config resources

To avoid dependencies or conflicts with existing AWS Organization based AWS Config, this module deploys a dedicated AWS Config Recorder, which has to be started after provisioning.

# Excerpts for illustration, not complete example, see main.tf

# AWS Config Delivery Channel to S3
resource "aws_config_delivery_channel" "well_architected" {
  name = "well_architected_config_delivery_channel"
  s3_bucket_name = module.aws_config_well_architected_recorder_s3_bucket.s3_bucket_id
  depends_on = [aws_config_configuration_recorder.well_architected]
}

# AWS Config Configuration Recorder with recording_frequency set by input variable
resource "aws_config_configuration_recorder" "well_architected" {
  name = "well-architected"
  role_arn = aws_iam_role.config_role.arn

  recording_group {
    all_supported = true
    include_global_resource_types = true
  }

  recording_mode {
    recording_frequency = var.recording_frequency
  }
}

# AWS Config retention configuration: Number of days AWS Config stores your historical information.
resource "aws_config_retention_configuration" "example" {
  retention_period_in_days = 400
}

# Manages status (recording / stopped) of an AWS Config Configuration Recorder.
resource "aws_config_configuration_recorder_status" "well_architected" {
  name = aws_config_configuration_recorder.well_architected.name
  is_enabled = true
  depends_on = [aws_config_delivery_channel.well_architected]
}

AWS Config Conformance Packs

Security, Reliability and IAM conformance packs are based on AWS’ library of Conformance Pack Sample Templates for AWS Config (in Cloudformation format):

The underlying checks are AWS Config Managed Rules and cannot be edited. The Cloudformation templates are imported in Terraform as data objects. ConfigRuleNames are replaced to suit the particular Well-Architected Framework Pillar and best practice.

# Excerpts for illustration, not complete example
locals {
  url_template_body_wa_security_pillar = "https://raw.githubusercontent.com/awslabs/aws-config-rules/refs/heads/master/aws-config-conformance-packs/Operational-Best-Practices-for-AWS-Well-Architected-Security-Pillar.yaml"
}

data "http" "template_body_wa_security_pillar" {
  url = local.url_template_body_wa_security_pillar
}

data "util_replace" "transformed_wa_security_pillar" {
  content = data.http.template_body_wa_security_pillar.response_body
  replacements = {
    "account-part-of-organizations" : "SEC01-securely-operate_bp_account-part-of-organizations",
    "ec2-instance-managed-by-systems-manager" : "SEC01-securely-operate_bp_ec2-instance-managed-by-systems-manager",
    "codebuild-project-envvar-awscred-check" : "SEC01-securely-operate_bp_codebuild-project-envvar-awscred-check",
    "mfa-enabled-for-iam-console-access" : "SEC02-identities_bp_mfa-enabled-for-iam-console-access"
     # .. and so on
    }
}

# Render templates to file on S3 to avoid template_body file limitation of 51,200 bytes
resource "aws_s3_object" "cloudformation_wa_config_security_template" {
  bucket = module.aws_config_well_architected_recorder_s3_bucket.s3_bucket_id
  key = "Cloudformation/wa-config-security.yaml"
  content = data.util_replace.transformed_wa_security_pillar.replaced
  content_type = "application/yaml"
}

# Takes the source Cloudformation file from S3, generates an AWS Config Conformance pack which behind the scenes creates an AWS managed Cloudformation stack. 
resource "aws_config_conformance_pack" "well_architected_conformance_pack_security" {
  count = var.deploy_security_conformance_pack ? 1 : 0
  name = "Well-Architected-Security"
  template_s3_uri = "s3://${module.aws_config_well_architected_recorder_s3_bucket.s3_bucket_id}/${aws_s3_object.cloudformation_wa_config_security_template.key}"
  depends_on = [aws_config_configuration_recorder.well_architected]
}

The Cost Optimization Conformance Pack is built from scratch. Custom Lambda Rules may be implemented like this:

# AWS Lambda function based on module from terraform-aws-modules
module "lambda_function_wa_conformance_cost_03_aws_budgets" {
  source = "git::https://github.com/terraform-aws-modules/terraform-aws-lambda.git?ref=f7866811bc1429ce224bf6a35448cb44aa5155e7"
  trigger_on_package_timestamp = false
  function_name = "WA-COST03-BP05-AWS-Budgets"
  description = "AWS Config Custom Rule which checks for AWS Budgets setup according to WAF COST03-BP05."
  handler = "index.lambda_handler"
  runtime = var.lambda_python_runtime
  source_path = "../../local-modules/wa-config-conformance/src/cost03_aws_budgets/index.py"
  attach_policy_statements = true
  timeout = var.lambda_timeout
  cloudwatch_logs_retention_in_days = var.lambda_cloudwatch_logs_retention_in_days
  policy_statements = {
    statement = {
      effect = "Allow"
      actions = [
        "budgets:DescribeBudgets",
        "budgets:ViewBudget",
        "config:PutEvaluations"
      ]
      resources = ["*"]
    }
  }

  tags = {
    Name = "Well-Architected-Conformance-COST03-BP05-AWS-Budgets"
  }
}

resource "aws_config_config_rule" "cost_01_aws_budgets" {
  name = "cost01-cloud-financial-management_bp_aws-budgets"
  description = "Checks for AWS Budgets setup according to WAF COST01-BP05 Report and notify on cost optimization."

  source {
    owner = "CUSTOM_LAMBDA"
    source_identifier = module.lambda_function_wa_conformance_cost_03_aws_budgets.lambda_function_arn

    source_detail {
      message_type = "ScheduledNotification"
      maximum_execution_frequency = var.scheduled_config_custom_lambda_periodic_trigger_interval
    }
  }

  depends_on = [module.lambda_function_wa_conformance_cost_03_aws_budgets]
}

# Lambda permissions for all AWS Config Custom Lambda Rules
resource "aws_lambda_permission" "config_permissions" {
  for_each = toset([
    module.lambda_function_wa_conformance_cost_02_account_structure_implemented.lambda_function_name,
    module.lambda_function_wa_conformance_cost_03_aws_budgets.lambda_function_name,
    module.lambda_function_wa_conformance_cost_03_aws_cost_anomaly_detection.lambda_function_name,
    module.lambda_function_wa_conformance_cost_03_add_organization_information_to_cost_and_usage.lambda_function_name,
    module.lambda_function_wa_conformance_cost_04_ec2_instances_without_auto_scaling.lambda_function_name
  ])

  statement_id = "AllowConfigInvoke"
  action = "lambda:InvokeFunction"
  function_name = each.value
  principal = "config.amazonaws.com"
  source_account = local.aws_account_id
}

Feedback and contributions

If you have any feedback, please let me know through your preferred medium of contact.

If you would like to contribute with additional functionality and check coverage, pull requests are welcome!

Resources

The post How to measure Well-Architected maturity? first appeared on Håkon Eriksen Drange.

AWS re:Invent re:Cap talk – Simplifying developer experience with new features in AWS Step Functions

Håkon Eriksen Drange — Fri, 17 Jan 2025 07:43:58 +0000

Monday January 13th I was invited by AWS User Group Oslo to participate in the traditional AWS re:Invent re:Cap meetup. My re:Cap contribution was a reflection on recent and valuable features in AWS Step Functions which can make the developer experience simpler and more efficient, reducing the time from idea to business value.

Main topics of my re:Cap talk

How to manage AWS Step Functions configuration with Infrastructure-as-Code
Replacing application (Lambda) code with Step Functions workflow configuration
Breaking apart a “Lambda-lith”
Step Functions intrinsic functions
New support for JSONata, in addition to JSONPath
Walkthrough of JSONata native functionality
New support for Variables
Step Functions Distributed Map state

“Customers choose Step Functions to build complex workflows that involve multiple services such as AWS Lambda, AWS Fargate, Amazon Bedrock, and HTTP API integrations. Within these workflows, you build states to interface with these various services, passing input data and receiving responses as output. While you can use Lambda functions for date, time, and number manipulations beyond Step Functions’ intrinsic capabilities, these methods struggle with increasing complexity, leading to payload restrictions, data conversion burdens, and more state changes. This affects the overall cost of the solution. You use variables and JSONata to address this.“

AWS Compute Blog: Simplifying developer experience with variables and JSONata in AWS Step Functions

Source material references this re:Cap talk was based upon are listed below.

Event highlights

Gunnar Grosch from AWS kicked off the event with highlights of new announcements from the pre:Invent and actual re:Invent period with a deeper dive on Amazon Aurora DSQL.

Then it was my turn before Colin from Capra Consulting shared his perception about the state of Platform Engineering.

The meetup wrapped up with an interesting panel discussion with Martin from Sopra Steria, Gunnar from AWS, Anders from Webstep and Erlend from Capra.

Many thanks to the organizers for a great event.

Slides from my re:Cap talk

Simplifying developer experience with new features in AWS Step Functions Download

Reference material

The post AWS re:Invent re:Cap talk – Simplifying developer experience with new features in AWS Step Functions first appeared on Håkon Eriksen Drange.

Increase system reliability with Immutable Infrastructure – Move fast and avoid surprises

Håkon Eriksen Drange — Fri, 09 Aug 2024 17:52:53 +0000

Table of contents

The largest outage in history of IT (so far)
Configuration drift
The concept of Immutable Infrastructure
Immutable Infrastructure in the AWS Cloud
- Virtual machine based workloads on EC2
- Container based workloads
- Immutability for Docker and local testing
- Immutability for workloads provisioned with AWS Elastic Container Service (ECS based on EC2 or Fargate)
- Immutability for workloads provisioned with AWS Elastic Kubernetes Service
- About Docker labels/image tags and deployment
- Immutability for serverless workloads provisioned with AWS Lambda
- Data handling in the immutable infrastructure model
Conclusion
References

The largest outage in history of IT (so far)

On 19 July 2024, the cybersecurity company CrowdStrike distributed a faulty update to its Falcon Sensor security software that caused widespread problems with Microsoft Windows computers running the software. As a result, roughly 8.5 million systems crashed and were unable to properly restart in what has been called the largest outage in the history of information technology and “historic in scale”.

The outage disrupted daily life, businesses, and governments around the world. Many industries were affected; airlines, airports, banks, hotels, hospitals, manufacturing, stock markets, broadcasting, gas stations, retail stores, emergency services and governmental websites. The worldwide financial damage has been estimated to be at least $10 billion.

*What happened? *

The CrowdStrike Falcon software suite consists, highly simplified, of an software application (agent), which is versioned, and Rapid Response Content configuration updates, the latter not being versioned, to facilitate rapid deployment. However, the outcome of this change was not quality assured properly before widespread deployment. In their Preliminary Post Incident Review, CrowdStrike indicated they promised to improve their quality assurance process and provide customers with “greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed.”

I would like to highlight that there are certain nuances to this case (rapid distribution of security detection/mitigation mechanisms, client machines vs. servers etc.), but my personal stance is that any system that is providing mission critical services should have a level of quality assurance that matches it’s Service Level Agreement/Objective and risk tolerance.

Outages like this are not something new. It can happen to any system where changes are applied live or in-place, without being tested properly in advance. Service disruptions can also happen during regular operating system patching procedures, if/when a bug or a security issue is introduced, or with any application or server process, depending on how privileged the process is.

Configuration drift

Long running systems will over the course of time experience configuration drift. A clean server started from a golden image immediately has a high degree of certainty of it’s current state, but as time goes by, certainty decreases. In a traditional (legacy) deployment model, software packages are updated, cache and log files are stored on disk. Perhaps not all log file locations are configured for rotation, perhaps an application stores temp files at a non-standard location and not all temp files are cleaned up. Over time disk space utilization increases, and at some point in the critical partitions, such as /boot may become full without it being noticed. The server’s state has now deviated significantly from its initial state, and it may be hard to reproduce that same state.

A software update that passed QA in staging is not guaranteed be successfully applied in production because of lacking disk space or some trash that has been lying around, accumulating over time, or custom, manual configurations which was supposed to “improve” something in the live environment.

Let’s say you start comparing a green Granny Smith against a green Granny Smith, but over the course of time the green Granny Smith may have evolved to a Red Delicious apple, and perhaps to an orange Asian pear!

For production systems, especially mission-critical ones, risk management becomes increasingly important. An operating model based on long running servers is prone to introduce risk and uncertainty as time goes by. Staging and production environments can over the course of time drift and not be identical anymore. You end up with a Snowflake Server.

Can the new state be predicted? Can the previously known state be reproduced? If not, you may have a challenge with rollbacks and disaster recovery.

It is a good idea to virtually burn down your servers at regular intervals. A server should be like a phoenix, regularly rising from the ashes.

https://martinfowler.com/bliki/PhoenixServer.html

The concept of Immutable Infrastructure

The Cambridge dictionary definition of Immutability is:

The state of not changing, or being unable to be changed

https://dictionary.cambridge.org/dictionary/english/immutability

With the advent of cloud and more capabilities for automation at our disposal, an alternative pattern called Immutable Infrastructure has gained popularity. The concept is simply about starting from a clean, well-known slate, every time a change is performed.

The concept of the Immutable Server, or disposable servers, was introduced around 2012 by actors such as Thoughtworks, Netflix and Google. Instead of using configuration management to try to keep systems in compliance, they advocated for using configuration management to create base images for servers that could be torn down and rebuilt at will.

An Immutable Server is the logical conclusion of this approach, a server that once deployed, is never modified or changed. No software or operating system updates, security patches, application releases or configuration changes are being performed in-place on live production servers. “Treat your servers like cattle, not like pets” gained traction. If there was a problem with a live server, it would be terminated and replaced by a new one, from a known state.

Not even new application releases/software artifacts, are deployed to existing servers. The running servers are replaced with new instances that has software artifacts built-in. With load balancing, automatic health checks and blue/green or canary capabilities, deploying new servers or rolling back to previous versions can be done without end user impact (if done correctly).

This concept is also described in the AWS Well-Architected Framework – Reliability Pillar – REL08-BP04 Deploy using immutable infrastructure.

Implementing in-place changes to running infrastructure resources, the common approach, is actually stated as an anti-pattern.

Benefits of establishing this best practice:

_ Increased consistency across environments: Since there are no differences in infrastructure resources across environments, consistency is increased and testing is simplified._
_ Reduction in configuration drifts: By replacing infrastructure resources with a known and version-controlled configuration, the infrastructure is set to a known, tested, and trusted state, avoiding configuration drifts._
_ Reliable atomic deployments: Deployments either complete successfully or nothing changes, increasing consistency and reliability in the deployment process._
_ Simplified deployments: Deployments are simplified because they don’t need to support upgrades. Upgrades are just new deployments._
_ Safer deployments with fast rollback and recovery processes: Deployments are safer because the previous working version is not changed. You can roll back to it if errors are detected._
_ Enhanced security posture: By not allowing changes to infrastructure, remote access mechanisms (such as SSH) can be disabled. This reduces the attack vector, improving your organization’s security posture._

Immutable Infrastructure in the AWS Cloud

This practice can be achieved regardless of compute option, but to reduce time to market and operational overhead it mandates a high degree of automation with CI/CD tools such as AWS CodePipeline and GitHub Actions.

Virtual machine based workloads on EC2

In this scenario an automated routine is established which produces virtual machine images called EC2 Amazon Machine Images (AMI). This “golden image” includes the respective version of application source code and it’s dependencies such as operating system services, at the expected versions. AMIs can be produced without environment specific configuration built-in, to be fetched from AWS Systems Manager Parameter Store and/or AWS Secrets Manager, so that the same image can be deployed and verified in dev/test, staging and production environments.

Hashicorp Packer has been available for quite some time. AWS also provides the native service EC2 Image Builder for this purpose.

Amazon provides managed AMIs for both Linux (Amazon Linux based on Fedora) and Windows workloads that are tailored for security and performance in AWS.

With Immutable Infrastructure, Windows Updates and Linux unattended-upgrades are disabled. Every time AWS releases a new officially supported base AMI version, or ad-hoc updates are required, the EC2 Image Builder Pipeline is triggered, which produces a new artifact and validates each change. Every new application version fetches the latest quality assured base AMI and produces a self-contained application AMI.

You can find a Terraform example with inline comments and explanations of EC2 Image Builder resources below .

# EC2 Image Builder component that installs Git and Nginx, clones a sample app repo from GitHub and starts the Nginx web server
resource "aws_imagebuilder_component" "hello_world" {
  name = "hello-world-component"
  platform = "Linux"
  version = "1.0.0"
  description = "Hello World application component"

  data {
    name = "hello-world-app-script"
    type = "AWS_LAMBDA"
    content = <<-EOF
      # Install Git and Nginx
      yum install -y git nginx

      # Clone sample Hello World app from GitHub
      mkdir app-repo
      git clone https://github.com/aws-samples/aws-codepipeline-s3-codedeploy-linux app-repo

      # Copy cloned files to Nginx public HTML directory
      cp -r app-repo/* /usr/share/nginx/html/

      # Start Nginx
      systemctl start nginx
    EOF
  }
}

# Recipe named nginx-recipe which includes the nginx-component.
# parent_image is set to Amazon Linux 2 AMI version 2024.7.20. This can be parameterized. 
resource "aws_imagebuilder_image_recipe" "nginx_recipe" {
  name = "nginx-recipe"
  parent_image = "arn:aws:imagebuilder:${var.region}:aws:image/amazon-linux-2-x86/2024.7.20"
  version = "1.0.0"

  component {
    component_arn = aws_imagebuilder_component.nginx.arn
  }
}

# Defines an infrastructure configuration, including the instance profile, security group, and subnet (intentionally not included)
resource "aws_imagebuilder_infrastructure_configuration" "nginx_infra" {
  name = "nginx-infra"
  instance_profile_name = aws_iam_instance_profile.image_builder_instance_profile.name
  security_group_ids = [aws_security_group.image_builder_sg.id]
  subnet_id = aws_subnet.image_builder_subnet.id
  terminate_instance_on_failure = true
}

# Pipeline output should be an Amazon Machine Image (AMI) with a name based on the build date.
resource "aws_imagebuilder_distribution_configuration" "nginx_distribution" {
  name = "nginx-distribution"

  distribution {
    ami_distribution_configuration {
      name = "nginx-ami-{{ imagebuilder:buildDate }}"
    }
    region = var.region
  }
}

# Defines the Image Builder pipeline, ties together the recipe, infrastructure configuration, and distribution configuration.
resource "aws_imagebuilder_image_pipeline" "nginx_pipeline" {
  name = "nginx-pipeline"
  image_recipe_arn = aws_imagebuilder_image_recipe.nginx_recipe.arn
  infrastructure_configuration_arn = aws_imagebuilder_infrastructure_configuration.nginx_infra.arn
  distribution_configuration_arn = aws_imagebuilder_distribution_configuration.nginx_distribution.arn
}

Container based workloads

In your CI/CD tool of choice, container images are produced and pushed to a container repository such as AWS Elastic Container Registry (ECR). Even though immutability is one of the foundations behind containers, it is not restricted, but based on how the container configuration is specified and launched. Container root filesystems are usually writable by default.

In many cases software packages are being updated on container launch, but this is an anti-pattern in immutability. Update installed packages and install Apache could yield different results if executed at different points in time, as new upstream software package updates are made available. We need to ensure that we test the exact same artifact in staging that is being deployed to production, so the recommended approach is to update all packages at container build time and then launch it with the root storage partition in read only mode.

The container’s root filesystem should be treated as a ‘golden image’ by using Docker run’s --read-only option. This prevents any writes to the container’s root filesystem at container runtime and enforces the principle of immutable infrastructure.

CIS Docker Benchmark control 5.12 – Datadog

Immutability for Docker and local testing

Adding a read-only flag at the container’s runtime enforces the container’s root filesystem being mounted as read only. With the –tmpfs option it is possible to mount a temporary file system for non-persistent data/cache.

docker run <Run arguments> --read-only <Container Image Name or ID> <Command>

# Example with the --tmpfs option to mount a temporary file system for non-persistent data/cache
docker run --interactive --tty --read-only --tmpfs "/run" --tmpfs "/tmp" ubuntu /bin/bash

Immutability for workloads provisioned with AWS Elastic Container Service (ECS based on EC2 or Fargate)

Configure the Amazon ECS Task Definition file to set parameter readonlyRootFilesystemin section Storage and logging to true, as the default value is false.

Example Terraform resource definition, see line 27:

resource "aws_ecs_task_definition" "hardened_task_definition" {
  family = "hardened-task-definition"
  network_mode = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu = 256
  memory = 512

  volume {
    name = "efs-volume"
    efs_volume_configuration {
      file_system_id = "fs-0123456789abcdef" # Replace with your EFS file system ID
      root_directory = "/tmp"
    }
  }

  container_definitions = jsonencode([
    {
      name = "hardened-container"
      image = "nginx:latest"
      essential = true
      portMappings = [
        {
          containerPort = 8080
          hostPort = 8080
        }
      ]
      readonlyRootFilesystem = true
      volumesFrom = [
        {
          sourceContainer = "efs-volume-container"
        }
      ]
    },
    {
      name = "efs-volume-container"
      image = "amazon/amazon-efs-utils:latest"
      essential = true
      volumeMounts = [
        {
          name = "efs-volume"
          mountPath = "/tmp"
        }
      ]
    }
  ])
}

With this configuration, the /tmp directory inside the hardened-container will be mounted to the specified EFS file system, allowing temporary files to be stored on the persistent EFS file system instead of the read-only root filesystem.

Immutability for workloads provisioned with AWS Elastic Kubernetes Service

This also applies to Kubernetes in general. In the manifest file, specify securityContext: readOnlyRootFilesystem: true.

Example Kubernetes manifest below which demonstrates read only configuration on lines 11-12:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  containers:
    - image: nginx
      name: hardened-container
      securityContext:
        readOnlyRootFilesystem: true
      volumeMounts:
        - name: cache-volume
          mountPath: /var/cache/nginx
        - name: runtime-volume
          mountPath: /var/run
        - name: efs-volume
          mountPath: /tmp
  volumes:
    - name: cache-volume
      emptyDir: {}
    - name: runtime-volume
      emptyDir: {}
    - name: efs-volume
      nfs:
        server: efs-server.default.svc.cluster.local
        path: "/efs-share"

After applying this updated manifest, the Nginx container will have an Amazon EFS volume mounted at /tmp, allowing it to use the persistent storage provided by Amazon EFS for temporary files while still maintaining a read-only root filesystem.

Leverage Kubernetes’ Deployment strategies for blue/green, canaryetc. as best suits your use-case.

About Docker labels/image tags and deployment

When new Docker images are produced and pushed to a container repository, a common approach is to add the label/tag “latest” to it (similar to HEAD in Git), and this is also referenced in CI/CD pipelines. However, with this approach it is challenging to answer exactly which Docker image was in production at any given time, since the reference to the actual image is lost.

To follow through on immutability it is recommended to set a unique tag on every new image. You can reference the actual image tag checksum or some other application specific tag like hello-world-v1.2.3 for GitOps and CI/CD deployment. If you also add a tag with the Git commit ID, debugging becomes much easier.

It also becomes more transparent when dealing with rollbacks when you know hello-world-v1.2.3 was live in production, hello-world-v1.2.4 failed the canary health checks, the automatic procedure rolled back to hello-world-v1.2.3, and you can quickly look up the git commit of hello-world-v1.2.4 for the change.

In Amazon Elastic Container Registry you can prevent image tags from being overwritten by enabling the property Tag immutability.

Immutability for serverless workloads provisioned with AWS Lambda

AWS Lambda is immutable by design. Lambda creates a new version of your function each time that you publish the function. It’s code, runtime, architecture, memory, layers, and most other configuration settings will remain unchanged.

Versions can be used to control function deployment. You can publish a new version of a function for beta testing, or canary testing for a small amount of users instead of deploying the change to all production users at once by using a specific version reference (Qualified ARN instead of $LATEST), in AWS API Gateway or other services that are routing traffic to your Lambda functions.

In the example below we define three aliases: staging, canary-prod and prod, which refers to relevant Lambda function versions. A Lambda routing configuration is defined which directs 90% of the traffic to alias prod (v41), and 10% to alias canary-prod (v42).

Procedure

Test version 42 in staging.
Roll out version 42 to 10% of all production users.
CloudWatch Metrics for Lambda functions are observed and grouped by a combination of alias and executed version.
If application monitoring and health checks are sustained over a course of eg. 30 minutes with no increases in error rates, update the alias for prod to increase function_version from 41 to 42 and reset routing_config. If not, roll back.
If health checks are still OK, end. If not, roll back.

Terraform example of AWS Lambda aliases and routing configuration for canary deployment:

resource "aws_lambda_alias" "staging" {
    name = "staging"
    function_name = "arn:aws:lambda:aws-region:123456789012:function:helloworld"
    function_version = "42"
    description = "Canary alias for production environment"
}

resource "aws_lambda_alias" "canary_prod" {
    name = "canary-prod"
    function_name = "arn:aws:lambda:aws-region:123456789012:function:helloworld"
    function_version = "42"
    description = "Canary alias for production environment"
  }

resource "aws_lambda_alias" "prod" {
  name = "prod"
  function_name = "arn:aws:lambda:aws-region:123456789012:function:helloworld"
  function_version = "41"
  description = "Alias for main production audience"
}

resource "aws_lambda_update_alias" "prod_routing_with_canary" {
  name = aws_lambda_alias.prod.name
  function_name = aws_lambda_alias.prod.function_name
  function_version = aws_lambda_alias.prod.function_version

  routing_config {
    additional_version_weights = {
      aws_lambda_alias.canary_prod.function_version = 0.1
    }
  }
}

Data handling in the immutable infrastructure model

As the compute resources themselves can come and go, data that needs to be persisted, including session state, has to be moved off of the compute tier. Depending on the type of data there are different AWS services at your disposal:

Conclusion

By adhering to the principle of never changing a running system, a higher degree of predictability and reliability can be achieved.

Many view rollbacks as a pain. Most people avoid spending time on rollback and disaster recovery testing. “We don’t do rollbacks, we prefer to try to fix the problem and roll forward”. This is a warning signal of manual procedures and lacking automation.

With the immutable infrastructure approach rollbacks becomes a natural part of change management. Database changes can be managed with the Expand-Contract pattern. Any failing automated checks with increased error rates should automatically roll back to the previously known working version, also supported by the database schema. By adopting canary releases and/or blue-green deployments rollbacks can be performed fast and with reduced end user impact.

A bonus with the Immutable Infrastructure approach is that rollbacks will become trivial. By releasing small changes frequently a failed new release shouldn’t be a big issue. Instead of being up at night or feeling the pressure of having to fight fires to resolve issues which are impacting end users, teams can investigate in peace and quiet, produce a new artifact version with the bug-fix and ship it quickly through fully automated and quality assured CI/CD pipelines, to master the art of Continuous Delivery and realize business value faster.

References

The post Increase system reliability with Immutable Infrastructure – Move fast and avoid surprises first appeared on Håkon Eriksen Drange.

Bye bye Bastion!

Håkon Eriksen Drange — Wed, 03 Jul 2024 17:27:38 +0000

Introduction
Exploring alternative workflows
- Deploy sample infrastructure
- Well-Architected Virtual Private Cloud (VPC)
- RDS Aurora MySQL Multi-AZ cluster
- EC2 Amazon Linux instance and security group
- AWS Cloud9 SSM Managed Instance
- Alternative 1: AWS Systems Manager – Session Manager
- Prerequisites
- Connecting to an EC2 instance in a private subnet from the AWS Console
- Connecting to an EC2 instance in a private subnet from your local workstation with the AWS CLI and AWS IAM Identity Center
- Connecting to an RDS cluster in a private database subnet with local port forwarding
- Alternative 2: AWS CloudShell VPC Environment
- Alternative 3: AWS Cloud9 IDE
Conclusion and feature comparison
References

Introduction

The Bastion Host, or Jump Host concept, has historically been a traditional pattern for providing system administrators with external access to internal compute resources on distinct networks. An actor connects to a dedicated host in a DMZ by most commonly Secure Shell (SSH) or Remote Desktop Protocol (RDP), and from there gain access to compute resources on internal networks to perform system maintenance, apply patches, check content in a database , update schemas etc.

These Bastion Hosts must behighly secured to withstand attacks. Measures include hardened operating systems and server configurations, removing non-required services and libraries, firewalling and audit logging, but the reality is that many instances do not meet the recommended standards, either by lack of knowledge or configuration mistakes. One of the most common attack vectors is simply not locking down the firewall rules/security group rules to only permit access on relevant ports from trusted IP ranges, leaving SSH/RDP open to anyone (0.0.0.0/0) instead of your corporate gateway or VPN.

As mentioned in my post Protect your webapps from malicious traffic with AWS Web Application Firewall, _ 7 percent of EC2 instances , 3 percent of Azure VMs , and 13 percent of Google Cloud VMs are publicly exposed to the internet. Among instances that are publicly exposed, HTTP and HTTPS are the most commonly exposed ports, and are not considered risky in general. After these, SSH and RDP remote access protocols are common._

On July 2nd 2024 the critical vulnerability CVE-2024-6387, labeled regreSSHion, was announced, where an unauthenticated remote code execution in OpenSSH’s server (sshd) could grant full root access. With a vulnerability score of 9.8/10 this is one of the most serious bugs in OpenSSH in years.

The first thing to do is to stop exposing SSH/RDP, and then find alternative, more modern workflows for accessing cloud resources. Even if access is whitelisted from trusted IP address ranges, it’s a bad practice in 2024 to have direct access into production environments.

As stated in the AWS Well-Architected Framework – Security Pillar:

Use automation to perform deployment, configuration, maintenance, and investigative tasks wherever possible. Consider manual access to compute resources in cases of emergency procedures or in safe (sandbox) environments, when automation is not available.

Common anti-patterns

Interactive access to Amazon EC2 instances with protocols such as SSH or RDP.

Maintaining individual user logins such as /etc/passwd or Windows local users.

Sharing a password or private key to access an instance among multiple users.

Manually installing software and creating or updating configuration files.

Manually updating or patching software.

Logging into an instance to troubleshoot problems.

Removing the use of Secure Shell (SSH) and Remote Desktop Protocol (RDP) for interactive access reduces the scope of access to your compute resources. This takes away a common path for unauthorized actions.

SEC06-BP03 Reduce manual management and interactive access

For reference, the CIS AWS Foundations Benchmark has multiple controls when it comes to detecting public exposure of SSH/RDP. AWS Security Hub AWS Config and AWS Trusted Advisor can help you here.

[EC2.53] EC2 security groups should not allow ingress from 0.0.0.0/0 to remote server administration ports
- This control checks whether an Amazon EC2 security group allows ingress from 0.0.0.0/0 to remote server administration ports (ports 22 and 3389). The control fails if the security group allows ingress from 0.0.0.0/0 to port 22 or 3389.
[EC2.13] Security groups should not allow ingress from 0.0.0.0/0 or ::/0 to port 22
- This control checks whether an Amazon EC2 security group allows ingress from 0.0.0.0/0 or ::/0 to port 22. The control fails if the security group allows ingress from 0.0.0.0/0 or ::/0 to port 22.
[EC2.21] Network ACLs should not allow ingress from 0.0.0.0/0 to port 22 or port 3389
- This control checks whether a network access control list (network ACL) allows unrestricted access to the default TCP ports for SSH/RDP ingress traffic. The control fails if the network ACL inbound entry allows a source CIDR block of ‘0.0.0.0/0’ or ‘::/0’ for TCP ports 22 or 3389.

Exploring alternative workflows

It’s encouraged to pivot from a classical systems administration approach and substitute interactive access with AWS Systems Manager capabilities.

Look into how you can automate runbooks and trigger maintenance tasks with AWS Systems Manager Automation documents.

Don’t perform changes in live systems, deploy EC2 compute resources using the immutable infrastructure pattern, as for container based workloads. Or, even better, containerize to AWS ECS Fargate/EKS.

If your use-case dictates interactive access; disable security group ingress rules for port 22/tcp (SSH) or port 3389/tcp (RDP) and leverage AWS SSM – Session Manager agent based access to EC2. You can also configure activity logging to Amazon CloudWatch Logs for a full audit trail. We will explore this workflow in the coming chapters.

Deploy sample infrastructure

For testing the described alternatives I have developed a sample Terraform module which deploys the following resources.

Well-Architected Virtual Private Cloud (VPC)

Public subnets
Private subnets with VPC endpoints
Database subnets
NAT Gateways
VPC endpoints

locals {
  azs = slice(data.aws_availability_zones.available.names, 0, 2)
  aws_account_id = data.aws_caller_identity.current.account_id
}

module "vpc" {
  source = "git::https://github.com/terraform-aws-modules/terraform-aws-vpc.git?ref=25322b6b6be69db6cca7f167d7b0e5327156a595"

  name = var.name_prefix
  cidr = var.vpc_cidr
  azs = local.azs
  private_subnets = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k)]
  public_subnets = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k + 4)]
  database_subnets = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k + 8)]

  create_database_subnet_group = true
  create_database_subnet_route_table = true
  create_database_internet_gateway_route = false

  manage_default_network_acl = true
  manage_default_route_table = true
  manage_default_security_group = true

  enable_dns_hostnames = true
  enable_dns_support = true
  enable_nat_gateway = true
  single_nat_gateway = false
  one_nat_gateway_per_az = true

  enable_flow_log = true
  create_flow_log_cloudwatch_log_group = true
  create_flow_log_cloudwatch_iam_role = true
  flow_log_max_aggregation_interval = 60

  vpc_tags = {
    Name = var.name_prefix
  }
}

module "vpc_endpoints" {
  source = "git::https://github.com/terraform-aws-modules/terraform-aws-vpc.git//modules/vpc-endpoints?ref=4a2809c673afa13097af98c2e3c553da8db766a9"

  vpc_id = module.vpc.vpc_id

  create_security_group = true
  security_group_name_prefix = "${var.name_prefix}-vpc-endpoints-"
  security_group_description = "VPC endpoint security group"
  security_group_rules = {
    ingress_https = {
      description = "HTTPS from VPC"
      cidr_blocks = [module.vpc.vpc_cidr_block]
    }
  }

  endpoints = {
    dynamodb = {
      service = "dynamodb"
      service_type = "Gateway"
      route_table_ids = flatten([module.vpc.intra_route_table_ids, module.vpc.private_route_table_ids, module.vpc.public_route_table_ids])
      policy = data.aws_iam_policy_document.dynamodb_endpoint_policy.json
      tags = { Name = "dynamodb-vpc-endpoint" }
    },
    ecs = {
      service = "ecs"
      private_dns_enabled = true
      subnet_ids = module.vpc.private_subnets
    },
    ecs_telemetry = {
      create = false
      service = "ecs-telemetry"
      private_dns_enabled = true
      subnet_ids = module.vpc.private_subnets
    },
    ecr_api = {
      service = "ecr.api"
      private_dns_enabled = true
      subnet_ids = module.vpc.private_subnets
      policy = data.aws_iam_policy_document.generic_endpoint_policy.json
    },
    ecr_dkr = {
      service = "ecr.dkr"
      private_dns_enabled = true
      subnet_ids = module.vpc.private_subnets
      policy = data.aws_iam_policy_document.generic_endpoint_policy.json
    },
    rds = {
      service = "rds"
      private_dns_enabled = true
      subnet_ids = module.vpc.private_subnets
      security_group_ids = [module.db.security_group_id]
    },
    kms = {
      service = "kms"
      private_dns_enabled = true
      subnet_ids = module.vpc.database_subnets
    },
    ssm = {
      service = "ssm"
      private_dns_enabled = true
      subnet_ids = module.vpc.private_subnets
    },
    ssmmessages = {
      service = "ssmmessages"
      private_dns_enabled = true
      subnet_ids = module.vpc.private_subnets
    },
    ec2messages = {
      service = "ec2messages"
      private_dns_enabled = true
      subnet_ids = module.vpc.private_subnets
    }
  }

  tags = {
    Name = var.name_prefix
  }
}

RDS Aurora MySQL Multi-AZ cluster

module "db" {
  source = "git::https://github.com/terraform-aws-modules/terraform-aws-rds-aurora.git?ref=7d46e900b31322fd7a0ab0d7f67006ba4836c995"

  name = "${var.name_prefix}-rds"
  engine = "aurora-mysql"
  engine_version = "8.0"
  master_username = "root"
  instances = {
    1 = {
      instance_class = "db.t3.medium"
    }
    2 = {
      instance_class = "db.t3.medium"
    }
  }
  vpc_id = module.vpc.vpc_id
  db_subnet_group_name = module.vpc.database_subnet_group_name
  security_group_rules = {
    ingress = {
      source_security_group_id = aws_security_group.private_access.id
    }
    ingress = {
      source_security_group_id = data.aws_security_group.cloud9_security_group.id
    }
    kms_vpc_endpoint = {
      type = "egress"
      from_port = 443
      to_port = 443
      source_security_group_id = module.vpc_endpoints.security_group_id
    }
  }

  tags = {
    Name = var.name_prefix
    Environment = "dev"
    Classification = "internal"
  }

  manage_master_user_password_rotation = true
  master_user_password_rotation_schedule_expression = "rate(7 days)"
}

EC2 Amazon Linux instance and security group

The Security Group for RDS is provisioned within the module. One placeholder security group labeled “private_access” is defined for EC2 and CloudShell purposes which only permits egress traffic. It is referenced in the RDS cluster security group to permit incoming connections on port 3306 for MySQL. This is called security group referencing and allows for dynamic configurations, instead of specifying static CIDR ranges, which often are too permissive.

data "aws_ami" "amazon_linux_23" {
  most_recent = true
  owners = ["amazon"]

  filter {
    name = "name"
    values = ["al2023-ami-2023*-x86_64"]
  }
}

module "ec2_instance" {
  source = "git::https://github.com/terraform-aws-modules/terraform-aws-ec2-instance.git?ref=4f8387d0925510a83ee3cb88c541beb77ce4bad6"

  name = "${var.name_prefix}-ec2"
  ami = data.aws_ami.amazon_linux_23.id
  create_iam_instance_profile = true
  iam_role_description = "IAM role for EC2 instance and SSM access"
  iam_role_policies = {
    AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  }
  instance_type = "t2.micro"
  vpc_security_group_ids = [aws_security_group.private_access.id]
  subnet_id = element(module.vpc.private_subnets, 0)

  # Enforces IMDSv2
  metadata_options = {
    "http_endpoint" : "enabled",
    "http_put_response_hop_limit" : 1,
    "http_tokens" : "required"
  }

  tags = {
    Name = "${var.name_prefix}-ec2"
    Environment = "dev"
  }
}

resource "aws_security_group" "private_access" {
  #checkov:skip=CKV2_AWS_5: Placeholder security group, to be assigned to applicable resources, but beyond scope of this module.
  name_prefix = "${var.name_prefix}-private-access"
  description = "Security group for private access from local resources. Permits egress traffic."
  vpc_id = module.vpc.vpc_id

  egress {
    description = "Permit egress TCP"
    from_port = 0
    to_port = 65535
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    description = "Permit egress UDP"
    from_port = 0
    to_port = 65535
    protocol = "udp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    description = "Permit egress ICMP"
    from_port = -1
    to_port = -1
    protocol = "icmp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.name_prefix}-sg-private-access"
  }
}

AWS Cloud9 SSM Managed Instance

The data property for the security group makes it possible to identity the security group provisioned by AWS Cloud9. This is added in an ingress rule for the RDS cluster, as previously defined.

resource "aws_cloud9_environment_ec2" "cloud9_ssm_instance" {
  name = "${var.name_prefix}-cloud9"
  instance_type = "t2.micro"
  automatic_stop_time_minutes = 30
  image_id = "amazonlinux-2023-x86_64"
  connection_type = "CONNECT_SSM"
  subnet_id = element(module.vpc.private_subnets, 0)
  owner_arn = length(var.cloud9_instance_owner_arn) > 0 ? var.cloud9_instance_owner_arn : null
}

data "aws_security_group" "cloud9_security_group" {
  filter {
    name = "tag:aws:cloud9:environment"
    values = [
      aws_cloud9_environment_ec2.cloud9_ssm_instance.id
    ]
  }
}

For a fully working code repository along with setup instructions, see https://github.com/haakond/terraform-aws-bastion-host-alternatives/blob/main/examples/main.tf and https://github.com/haakond/terraform-aws-bastion-host-alternatives/blob/main/README.md.

Alternative 1: AWS Systems Manager – Session Manager

With AWS Systems Manager – Session Manager, you can manage your Amazon Elastic Compute Cloud (Amazon EC2) instances, edge devices, on-premises servers, and virtual machines (VMs). Port forwarding is also supported to connect to remote hosts in private subnets.

You can use either an interactive one-click browser-based shell or the AWS Command Line Interface (AWS CLI). Session Manager provides secure and auditable node management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys. Session Manager supports Linux, Windows and macOS and session activity can be logged with AWS CloudTrail and Amazon CloudWatch Logs.

Session Manager can be configured to log entered commands and their output during a session, which can be used for generating reports or audit situations.

Note: AWS Systems Manager also provides an option with EC2 Instance Connect Endpoints, but this is based on SSH.

The coming two examples are based on this access pattern.

Prerequisites

Supported operating system
AWS Systems Manager SSM agent installed
- List of AMIs with the SSM Agent preinstalled
Connectivity to endpoints ec2messages, ssm and ssmmessages in the current region
IAM service role permissions AmazonSSMManagedInstanceCore or equivalent
Optional: Install the Session Manager plugin for the AWS CLI

Connecting to an EC2 instance in a private subnet from the AWS Console

There are two optional starting points in the AWS Console, either from the AWS Systems Manager – Session Manager or directly from the EC2 instances list. The EC2 approach is usually the fastest and most convenient. With the prerequisites in order, find the instance you want to connect to, hit Connect.

Ensure the tab with the option Session Manager is chosen and click Connect again.

I am now logged in and authenticated with my Federated AWS IAM Identity Center method and we have full traceability.

Connecting to an EC2 instance in a private subnet from your local workstation with the AWS CLI and AWS IAM Identity Center

In this example Microsoft Entra ID is the identity provider and federationis configured with AWS IAM Identity Center for modern user management. Do yourself a favor and get rid of those IAM users with static access keys.

To authorize your workstation based on Linux, macOS or Windows Subsystem for Linux, ensure you have the latest version of the AWS CLI installed.

Run aws configure sso and follow the instructions to obtain a valid session based on a browser where you’re currently logged in. For a full step by step guide see https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html#sso-configure-profile-token-auto-sso.

You should now be logged in and have chosen the relevant AWS account and role to assume.

To confirm I run the following AWS cli command to get a list of EC2 instances including the Name tag value. One instance is returned.

aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId, Tags[?Key==Name].Value[]]' --output=json

[
    [
        [
            "i-0b448a908727eec7d",
            [
                "bastion-alternative-demo-ec2"
            ]
        ]
    ]
]

Connecting to the instance is as simple as:

aws ssm start-session --target i-0b448a908727eec7d

Type exitto terminate the session. There you go, CLI or Console based access with a security group with no inbound rules.

Connecting to an RDS cluster in a private database subnet with local port forwarding

As with the previous example, establish a session with aws configure sso and identify the EC2 instance you would like to use as proxy. In this example we use the same one. To find the RDS cluster writer endpoint name you can issue the following command:

aws rds describe-db-clusters \
  --query 'DBClusters[?starts_with(DBClusterIdentifier, `bastion-alternative-demo`)].DBClusterIdentifier' \
  --output text | xargs -I {} aws rds describe-db-cluster-endpoints \
  --db-cluster-identifier {} \
  --query 'DBClusterEndpoints[?EndpointType==`WRITER`].Endpoint' \
  --output text

Alternatively, in the AWS Console navigate to RDS and copy the Writer Endpoint name, FQDN.

Execute the following command to start a port forwarding session which should provide MySQL connectivity to port 3306 on your local machine:

aws ssm start-session --target i-0b448a908727eec7d --document-name AWS-StartPortForwardingSessionToRemoteHost --parameters '{"portNumber":["3306"],"localPortNumber":["3306"],"host":["bastion-alternative-demo-rds.cluster-cmikipz5ncly.eu-west-1.rds.amazonaws.com"]}'

The main difference is that this time the aws ssm start-session command will trigger an AWS Systems Managed Document called “AWS-StartPortForwardingSessionToRemoteHost” and we supply desired parameters. For demonstration the Terraform module has provisioned an RDS Aurora MySQL cluster in private database subnets.

The command outputs “Starting session with SessionId (..) Port 3306 opened, Waiting for connections.

In another terminal I run mysql -y root -p -h 127.0.0.1 --port 3306 and take the opportunity to create a new privileged mysql user called chuck_norris.

Verified working as expected.

If you prefer to use a GUI client like MySQL Workbench or HeidiSQLconfigure it to connect on localhost port 3306. If you run a development database server locally you probably would like to configure port forwarding to a different port.

Alternative 2: AWS CloudShell VPC Environment

AWS CloudShell is a browser-based shell that is pre-authenticated with your console credentials which makes it easy to securely manage, explore and interact with AWS resources. Common development tools related to AWS are also pre-installed.

AWS announcedVPC Environment support for AWS CloudShell on June 26th 2024. This enables the possibility to use CloudShell securely within the same subnet as other resources in your VPCs without the need for additional network configuration. Before this there was no means to control the network flow.

One caveat is that AWS CloudShell VPC environment does not support persistent storage, as the regular CloudShell feature has. Storage is ephemeral and data and home directories are deleted when an active environment ends, so you have to ensure data you care about is saved in Amazon S3 or another relevant persistent store. In my opinion, from a security perspective, auto-cleanup is a positive thing.

Open Cloudshell from the main AWS Services search box or the logo icon:

We choose the pre-provisioned VPC, subnet and security group:

Let’s verify outbound connectivity:

The /home partition has about 12GB free space. If you need more scratch space, look into mounting Amazon Elastic File System or dump stuff on Amazon S3. Keep in mind that the CloudShell volume is ephemeral and data will be gone after your session ends.

Let’s also check that Cloudshell’s Elastic Network Interface is in the expected private subnet. Thank you Amazon Q for the creative query suggestion.

aws ec2 describe-network-interfaces \
  --filters "Name=private-ip-address,Values=$(ip addr show ens6 | grep -oE '((1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])' | head -n 1)" \
  --query 'NetworkInterfaces[*].{SubnetId:SubnetId}' \
  --output text | xargs -I {} aws ec2 describe-subnets --subnet-ids {} \
  --query 'Subnets[*].{SubnetId:SubnetId,Ipv4CidrBlock:CidrBlock,TagName:Tags[?Key==`Name`].Value|[0]}' --output table

Cloudshell has many development tools pre-installed, including a mysql client. To verify that the endpoint hostname resolves to a private ip address in the private subnet, install dig from bind-utils:

sudo yum install bind-utils -y
dig +short <hostname>

We successfully managed to connect to the RDS Aurora MySQL cluster and created a new database.

Alternative 3: AWS Cloud9 IDE

AWS Cloud9 is a cloud based integrated development environment (IDE) that can be accessed through the AWS Console. It could be an alternative if you prefer a lightweight client/workstation environment, where your development environment will be the configured the same across client devices.

Cloud9 is a fully managed service based on EC2 and EBS for persistent data storage. Instance hibernation takes place after a period if inactivity (configurable, from 30 minutes to x hours) to save costs when not in use. A Cloud9 environmentcan be deployed into both public and private subnets, in modes EC2(recommended) or SSH(discouraged). The EC2 mode supports the “no-ingress instance” pattern, without the need to open any inbound ports, by leveraging AWS Systems Manager. This is the procedure we will explore further.

A sample Cloud9 instance has been provisioned as part of the Terraform sample module. Navigate to AWS Cloud9 in the AWS Console, locate “bastion-alternative-demo-cloud9” and click Open.

The EC2 environment type comes with AWS managed temporary credentials activated by default, which manages the AWS access credentials on the users behalf, with certain restrictions. To ensure you get all privileges available to the IAM policies for your active role session, disable this in the AWS Cloud9 main window, Preferences, AWS Settings, Credentials.

Open a Terminal tab and issue aws configure sso as in the previous example, and set the SSO session name “default”.

aws s3 ls verifies the AWS session. Technically, this could have been executed from “anywhere”, but the main benefit are:

The Cloud9 environment maintains the configuration regardless of client device.
Other resources on private subnets may be configured to be available.

Verify database connectivity:

Conclusion and feature comparison

In this blog post we explored alternative workflows which can replace the concept of a traditional Bastion Host accessed by SSH or RDP. AWS provides customers with alternatives so that you can choose the one that best fits your use-case.

Supports port forwarding and local GUI clients.

Full integration with Cloudtrail and CloudWatch Logs for auditing and activity tracking | All prerequisites for SSM Managed Instances may be seen as a hurdle, but can be solved with IaC. | No additional charges for accessing Amazon EC2 instances.

The advanced on-premises instance tier is required for using Session Manager to interactively access on-premises instances. |
| AWS CloudShell VPC Environments | Available anywhere in the AWS Console.

Does not require extensive configuration.

Easy to use for quick commands or lookups.

Ephemeral storage, auto-cleanup after session inactivity.

| Ephemeral storage, auto-cleanup after session inactivity.

Due to possible session timeout issues consider other options for long running commands, database imports/exports etc.

Audit and activity logging capabilities not matching SSM Session Manager. | No additional charges, minimum fees nor commitments. You only pay for other AWS resources you use with CloudShell to create and run your applications. |
| AWS Cloud9 | Appealing if you’re also working on code development (application/IaC).

Same IDE experience regardless of client device.

Terraform/Cloudformation deployments can be triggered triggered from the same terminal.

Data is persisted on EBS until environment termination. | Session/permission management can be cumbersome.

Audit and activity logging capabilities not matching SSM Session Manager. | No additional charges, minimum fees nor commitments for the service itself. You pay only for the compute (EC2) and storage resources (EBS) for the environment.

Example:

t2.micro Linux instance at $0.0116/hour x 90 total hours used per month = $1.05

$0.10 per GB-month of provisioned storage x 10-GB storage volume = $1.00

Total monthly fees: $2.05 |

My personal preference is to pivot to immutability with containers and automation, but AWS Systems Manager – Session Manager would be the most viable alternative workflow for any remaining EC2 based workloads.

Instead of querying the database directly for troubleshooting or support requests, develop a support microservice or dashboard for the purpose. It could be as simple as SELECT * from relevant tables which returns relevant data in JSON format, with standard employee authentication and authorization mechanisms built in.

Full Terraform sample code is available at https://github.com/haakond/terraform-aws-bastion-host-alternatives, feel free to grab anything you need.

References

The post Bye bye Bastion! first appeared on Håkon Eriksen Drange.