DEV Community: Serhii Vasylenko

Layers Made It Universal. Harnesses Made It Run

Serhii Vasylenko — Fri, 24 Apr 2026 21:15:56 +0000

A continuation of Flip the Axis: A Layer-Based Approach to Multi-Service Migrations.

TL;DR

You can't script your way across a fleet of snowflake repositories. Neither can you just ask an AI agent to "migrate this service" and hope. What worked for the eight-quarter migration was a harness -- a prompt pipeline in which each layer was a sequence of ordered steps: some calling scripts for deterministic changes, some using AI to discover and adapt, and some validating the results. The harness chained their outputs, ran each layer across 21 repos at once, and landed merge requests when it was done.

Here's what that looked like in practice -- including the parts we got wrong.

From methodology to machinery

The previous article ended on a line worth repeating: the methodology enables the tooling, not the other way around. This post is where that line becomes machinery.

The project: migrating 21 services from ECS to EKS -- the final wave of an eight-quarter effort. Four engineers, targeting roughly ten repos per engineer per day on a given layer. The services were snowflakes: each with its own code style, CI configuration, logging framework, naming conventions, and infrastructure setup. The layers defined what to do -- add an OIDC provider, swap the logging appender, rewrite the CI pipeline, set up a piece of infra -- and the action was identical across services. But how to execute each layer varied across repositories. Same change, different wiring, 21 times over -- toil that doesn't yield to a single script. The layer approach made that pace imaginable. The harness made it real -- though the most important piece wasn't what we built first.

How a layer runs

Every layer ran through the same pipeline -- a workflow of ordered prompts that mixed step types.

Take logging. One service uses logback.xml and is Java, another uses a custom logging setup and is NodeJS, and another has a custom appender in a shared library. You can't script that discovery -- but you can script adding the dependency once the agent finds the right config. So the workflow does both: an AI step to discover and adapt, a script step for the deterministic edit, and a validation step to check the result.

Some steps called Go tools for deterministic changes -- known target, computable edit, no reasoning required. Others used AI to discover and adapt: Terraform is a good example -- we could not realistically script the changes, but we could give the LLM the modules to use for adding new configuration. Others validated what the earlier ones produced. The pipeline chained their outputs, for example:

What a script produced in step 3 informed what the agent analyzed in step 4.
What the agent implemented in step 5 was what the validation prompt checked in step 7.

Why "ask Claude to migrate this service" fails

You cannot ask an AI agent to "migrate this service to EKS." The task is too broad. The context is too large. The agent will hallucinate a plausible-looking solution, skip steps it decided weren't important, or produce something that looks right and isn't.

[!NOTE]
The failure mode isn't that the AI is dumb. It's that the task has no structure, so the agent invents its own -- and its structure drifts between runs.

You get 21 repositories migrated in 21 different ways, with 21 different sets of mistakes to audit. That's worse than having done nothing.

The fix isn't a better prompt. The fix is everything around the prompt.

Harness engineering, named a little late

A few months ago, the term harness engineering started showing up -- a zero-volume search term until early 2026. The idea: design the constraints and scaffolding around an LLM that make it reliable. Not whether to use AI, but to answer: what do you do after it's part of your toolchain?

So far, the conversation is mostly about coding agents. We got here through infrastructure migration -- and structured infra work is where the pattern fits most naturally. The changes follow patterns. The validation criteria are concrete. And the same change repeats across dozens of repos -- which is exactly what a harness is built for.

The answer turns out to be unsexy: a pipeline that runs prompts in a fixed order, enforces what the agent can touch at each step, and validates its own work before declaring done.

I didn't have that term in October 2025 when we built our own version of one. It's easier to name a thing after you've already built it wrong twice. We just kept running into the same failure -- the agent doing too much, too fast, with not enough guardrails -- and kept adding constraints until it stopped failing. Mutation boundaries. Ordered steps. Explicit reasoning gates. Validation passes.

What was new -- for us -- was treating the harness as the primary engineering artifact. Not the prompt. Not the model. The pipeline around them.

The shift worth taking from it is this -- stop optimizing the prompt, start engineering the pipeline that runs it.

What our harness looked like

A workflow is a sequence of numbered markdown files -- each one a step prompt. The agent runs them in order inside an isolated Claude Code session. Context compounds: what step 2 discovered informs what step 5 decides.

The shape is always the same: context → discovery → analysis → planning → implementation → validation → ship → report.

Not every step needs the agent to reason. Some prompts call a Go tool or script for a deterministic change -- same edit, known target, no drift. Others need AI to discover, analyze, and adapt. The workflow mixes both and outputs the chain forward.

Six things make this structure trustworthy:

Mutation boundaries. Every step is tagged READ ONLY, MAKES CHANGES, or PLANNING ONLY. The agent knows what it's allowed to do at each point. No surprise writes during discovery.
Context anchoring. Step 0 explains why, not just what. "Here is why we move this piece of infrastructure and need it to be X and Z in a new destination." "Here is why templates can't be applied blindly." The agent gets the intent before it touches any code.
Domain knowledge in the prompts. Hard-won lessons encoded directly: "BOM manages versions -- you still must declare STS explicitly." "Check actual pipeline behavior, not just config files." These are the footnotes a senior engineer would leave for a junior one. The agent gets them every time.

The first three give the agent the right starting position. The rest keep it from wandering.

Early exits. If the dependency already exists, skip to the report. If a values file for a Helm chart is already configured, skip to the report. Not every repo needs every step.
Explicit reasoning gates. Complex steps require reasoning, so the agent has to reason through the problem before acting -- no freestyle implementation -- and this must be declared explicitly.
The implement/validate split. For critical layers, two separate workflows ran in sequence: one to implement, one to validate. The validation workflow reviewed the implementation against defined criteria rather than rubber-stamping its own work. This did more for reliability than any other constraint we added.

None of these is about making the agent smarter. They're about making it predictable. Every constraint trades a degree of freedom for a degree of reliability. That's the whole game.

The instrument

Workflows describe what the agent should do per repo. We still needed an orchestrator to run them -- and to run them across a batch of repos at once.

So we built a parallel execution wrapper on top of the Claude Agent SDK. Think of it as a prompt pipeline runner. Point it at a workflow and a list of repositories, and for each repo it clones into its own git worktree, spins up an isolated Claude Code session, runs the workflow's step files in order, enforces the mutation boundaries from the previous section, and lands a per-repo report when it's done. One engineer monitors the batch and reviews the results.

Each session adapts to its repository while following the same workflow. The agent in repo A figures out one logging setup; the agent in repo B figures out a completely different one. Both produce the same kind of report. The engineer opens ten reports instead of writing ten PRs.

This is the automated flip-the-axis model. One layer, one workflow, N repos, one human in the loop. All twelve migration layers ran through this pipeline.

Prompt pipeline + parallel sessions + per-repo reports + human review at the end.

Prompts as production code

The non-obvious part: all of this lived in a shared repository. Workflows, runbooks, Go tools, scripts, per-service metadata -- all version-controlled in one place. The project headquarters.

This wasn't convenient. It was a knowledge-sharing mechanism.

When an engineer discovered a prompt needed more specificity -- say, the IAM workflow missed an edge case with cross-account trust policies -- they updated the prompt and committed it. On the next pull, every teammate got the improvement. Same for validation scripts, runbook instructions, and service metadata.

The lesson: treat AI prompts and workflows like production code. Review changes. Iterate as a team. The prompts you write in week 1 will not be the prompts you need in week 6 -- and every improvement should propagate to everyone automatically.

If your team is adopting AI for infra work and the prompts live in private gists, you've already lost the compounding advantage.

Honest assessment

Cross-referencing was the hardest problem we didn't fully solve. The Helm values layer required correlating data from Terraform configs, ECS task definitions, and application codebases into a single file. AI agents working within a single repository can't hold that full picture. Human judgment and manual cross-checking stayed essential here, and I don't think that's going to change for this class of problem anytime soon.

The reliability gradient was predictable. Within each workflow, the deterministic steps -- the ones calling scripts -- never drifted. The AI steps were reliable when isolated to a single file following a repeatable pattern. Reliability dropped when a step required cross-repository context or judgment calls about tradeoffs. Those stayed with humans.

We validated -- but not enough. The implement/validate split worked wherever we applied it -- it was the highest-leverage pattern in the whole setup. But we didn't apply it to every layer, and we should have. The same workflows we used for implementation could have run in verification mode at near-zero additional cost. Our QA and PREPROD environments caught what universal validation would have. They shouldn't have had to.

If I were starting this project again tomorrow, the first thing I'd build is a validation workflow for every implementation workflow, from day one. Not as an afterthought. As a matching pair.

What this means

Layers made the work universal. Harnesses made the AI trustworthy enough to run it -- even across 21 repos, each wired differently. Together they closed out an eight-quarter migration -- four engineers, not forty.

The next time someone frames it as "just write a script" versus "just use AI" -- it's the wrong question. Build the harness that runs both.

Further reading:

Harness Design for Long-Running Application Development -- Anthropic's engineering team on the same concept applied to agentic coding.
Harness Engineering -- OpenAI's take on the same discipline.
It's a Skill Issue: Harness Engineering for Coding Agents -- HumanLayer on harness configuration for coding agents.

Flip the Axis: A Layer-Based Approach to Multi-Service Migrations

Serhii Vasylenko — Fri, 24 Apr 2026 21:15:41 +0000

TL;DR

When you're migrating many services through the same steps, parallelize by step, not by service. Sweep one type of change across all services, then the next – it compounds learning, catches inconsistencies early, and makes automation viable. But recognize which services don't fit the pattern: architecturally unique services should still be migrated serially.

The Problem

You've probably seen the shape of this problem before. You're planning next quarter's migration – could be Kubernetes, a new database engine, a cloud provider switch, a major framework version bump. You count the services. You count the engineers. The math doesn't work.

Here's what it looked like for us: 2025 Q4 planning, 21 services still running on ECS (Amazon's container orchestration service) that needed to move to EKS (their managed Kubernetes platform). A headcount cut left us with 4 engineers. Each service migration had been taking 3-4 weeks of effort. The project had already been running since January 2024 – nearly two years – and the serial execution model from the previous quarter had required 8 engineers for 19 services. We had half the people, more services, and a timeline that was starting to feel permanent.

Nobody was telling us it had to be done next quarter. Our management said, "It's okay if we can't, don't worry."

But you know what happens to migrations that stretch for many months. They lose momentum. Engineers rotate off. Institutional knowledge erodes. The remaining services – always the hardest ones – sit in a permanent "next quarter" backlog.

You don't have a staffing problem. You have an execution model problem.

Why serial breaks down

The default migration approach is serial: one engineer owns a service end-to-end and walks it through every step – networking, permissions, environment adjustments, certificates, CI/CD, DNS, cleanup. This works fine for a couple of services. This breaks down at scale.

The engineer context-switches across completely different types of work – networking, then application configuration, then debugging a permissions issue – and never builds deep fluency in any of them. Services are unique snowflakes – each with its own code style, dependency patterns, and configuration quirks. Serial migration means absorbing that uniqueness for every service, at every step.

Even worse – learning stays siloed. An engineer who figured out a networking edge case in week 2 can't help the engineer who hits the same issue in week 6 – by then, they've moved on. Everyone is deep in a different service, at a different stage. The team can't effectively pair, review, or unblock each other.

The Insight

When I looked at what we'd actually done in Q3 – service by service, step by step – the pattern was obvious in hindsight: we were doing the same work over and over again. Networking, permissions, application setup – identical across services.

It only looked unique because we were thinking one service at a time.

What if we flipped the axis? Instead of completing all steps for one service before moving to the next, complete one step across all services before moving to the next step.

That's the core of the layer-based approach. A layer is one type of infrastructure or configuration change, applied to every service in the migration scope. You sweep through all services at one layer, validate, then move to the next layer.

Why this works

Repetition builds expertise. By the third service in a layer, you've seen the pattern. By the tenth, you're fast.
Cross-service checks catch errors early. When you're applying the same change to 20 services in a row, inconsistencies become obvious.
Learning compounds across the team. Everyone works the same layer simultaneously – discoveries spread instantly instead of weeks later.
Automation becomes viable. Identical changes across services are exactly what tooling excels at – predictable patterns with minor per-service variations.

Defining Your Layers

The number of layers depends on your migration. Ours had 14. Yours might have 8 or 20.

Here are the categories we found useful, grouped by concern:

Discovery: mapping downstream dependencies – services, databases, endpoints, protocols
Connectivity: networking between environments, firewall configurations
Identity: permissions, service accounts, trust policies, OIDC configuration
App level security: certificates, TLS termination, WAF rules
Application: runtime configuration, environment variables, secrets, logging adjustments
Delivery: CI/CD pipelines, ingress and routing, traffic management for gradual rollout

Your categories will differ. The names don't matter – the decomposition does.

How to decompose your own migration

Start from a single service migration you've already done. List every change you made, in order. Group changes by type, not by when they happened. Each group is a candidate layer.

Then validate: can this layer be applied independently of the next one? Can you validate it before moving on? If yes, it's a good layer boundary. If two changes are tightly coupled and can't be validated separately, merge them into one layer.

One rule we learned the hard way: one layer per pull request. Early on, some PRs combined changes from multiple layers – networking and permissions in the same commit. Validation got complex, rollbacks got messy. Keep them separate.

Execution Model

A layer sweep works like this: the team takes on a layer, splits the service list among themselves, and each engineer applies that layer to their assigned services. Everyone works the same type of change simultaneously.

One engineer can realistically sweep a single layer across 6-8 services in a day. That number surprises people – until they know the tooling. We paired the layered methodology with AI-assisted automation that handled the repetitive configuration work across services. But the important thing is: the layer-based structure is what makes that automation possible.

When every service needs the same type of change with minor variations, you can build prompts, scripts, and validation checks that apply across the board. Serial, per-service work is too varied to automate effectively. The AI tooling story – what worked, what failed, and where human judgment was irreplaceable – is the subject of the next post in this series.

During execution, the team meets briefly to sync on edge cases – because the work is homogeneous, an edge case in one service is immediately relevant to every other service going through the same layer.

Progress tracking

A simple table – one column per layer, one row per service – serves as the source of truth. The team updates it in real time. Status per cell: not started, in progress, done, blocked, not applicable. This sounds basic, but it's surprisingly effective. You can see at a glance where the project stands, which layers are complete, and where blockers are clustering.

Services that don't fit

Not every service fits the pattern. In our case, 4 out of 21 services were architecturally complex enough that the layered approach didn't help – they required deep, per-service analysis that negated the speed advantage.

We recognized this early and migrated them serially, with dedicated engineers working in parallel with the layer sweeps. Trying to force these into the pattern would have slowed everything down.

The lesson: the layer-based approach is a force multiplier for homogeneous work. When a service is genuinely unique, serial migration is the right tool. Budget for both.

Coordination That Matches the Work

The coordination model that works during one phase can hurt you in the next.

During layers: synchronize

When the whole team works the same layer, synchronous coordination is natural and cheap. Team syncs are short because everyone has context on the same type of work. An edge case discovered by one engineer is immediately useful to the others. Knowledge transfer happens without any deliberate mechanism – the work itself is identical.

During traffic switching: structure async handoffs

When the project moves from layer execution to per-service traffic switching, the work diverges. Each service has its own timeline, its own blockers, its own owning team with a different schedule. Synchronous coordination becomes expensive – the team is now working on different problems.

This is where a handoff log pays for itself. A shared document – not "made progress on Service X" but the actual PR link, the specific blocker, the decision to skip WAF configuration for this service, and why. What made it work: specificity over summary, explicit ownership, and early surfacing of blockers.

We heavily used this approach during the last phase of migration – the traffic switch – when two team members went to our SF hub to be on site with service owners, and two stayed in Berlin. But this isn't a timezone trick – it works for co-located teams just as well. Fewer meetings, more focused execution, and a written record that prevents "I thought you were handling that" conversations.

The lesson: match the coordination model to the shape of the work. When work is homogeneous, synchronize. When it diverges, structure async handoffs and get out of each other's way.

The Traffic Switch Cadence

Layer execution is predictable. Traffic switching is where the surprises live.

We used a graduated weekly cadence: Monday preflight (verify hostnames, certificates, ingress, autoscaling, dashboards – deploy one instance), Tuesday scale up and shift 1% of traffic, Wednesday observe and fix, Thursday shift to 50%, Friday observe and fix, following Monday shift to 100%.

The observation days weren't idle – they were when most debug work happens. Issues that don't surface at 1% show up at 50%. Fixes discovered for one service often apply to others in the same batch.

Batch your traffic switches. Running multiple services through this cadence simultaneously amortizes the coordination overhead – the preflight checklist, once built, applies to every service.

When This Works (and When It Doesn't)

The layered approach is not universal. It works well under specific conditions.

Use layers when:

The migration is decomposable into independent, repeatable steps
The same type of change applies across many targets with minor per-service variations
Changes can be batch-validated – all services at one layer before moving on
The team is small relative to the workload and needs a force multiplier

Use serial when:

Services are architecturally unique and complex, and require deep, per-service analysis
The number of targets is small enough that coordination overhead outweighs the parallelization benefit

This is not an either/or decision. In our migration, the layered approach covered 17 of 21 services. The remaining 4 were migrated serially. Recognizing which services don't fit the pattern early is just as important as the pattern itself.

What We'd Do Differently

Start the handoff log from day one. We introduced it when the team split across workstreams during traffic switching. In retrospect, the discipline of specificity and explicit ownership helps even when everyone is in the same room working the same layer.

Run validation sweeps after each layer, not at the end. We deferred some validation to later phases, when we did traffic switch on preproduction environments, which made fixing errors more expensive and created pressure during the most time-sensitive window.

Define service owner readiness criteria upfront. Some services reached the traffic switch phase with owners who weren't fully briefed, dashboards that weren't adjusted, etc. Clear criteria before the switch phase would have eliminated friction during the highest-pressure window.

Plan for the energy arc. An intensive, multi-month migration grinds people down. Build rotation points into the plan. Bring fresh perspective at deliberate moments – especially before the production switch phase.

Track decisions explicitly, separate from action items. Some decisions logged in the handoff document were missed because they were buried among task updates. A dedicated "decisions" section prevents teams from diverging without realizing it.

Key Takeaways

Flip the axis. When many services go through the same steps, parallelize by step, not by service. The efficiency gain comes from repetition, shared learning, and automation – not from working harder.
Define your layers by decomposing a single service migration. Group changes by type, validate that layers can be applied independently, and enforce one layer per merge request.
Match coordination to the shape of the work. Synchronize when work is homogeneous. Structure async handoffs when it diverges.
Recognize what doesn't fit the pattern. Some services are genuinely unique. Budget for serial migration alongside the layer sweeps.
The traffic switch is its own phase. Layer execution is predictable. Traffic switching is where surprises live. Treat it with a graduated cadence and observation days.
The methodology enables the tooling, not the other way around. We paired layer-based execution with AI-assisted automation – and that's what made one engineer sweeping 6-8 services in a day realistic. But the automation only worked because the layers created predictable, repeatable patterns. That story is next.

If this feels like a problem you've hit – or you're about to – I'd like to hear your approach. Same constraint, different solution? A migration where layers didn't work? Drop a comment.

Full Attendance, Zero Presence

Serhii Vasylenko — Thu, 23 Apr 2026 21:33:08 +0000

Five minutes after the all-hands, someone asks me in the hallway, "What did he mean by that?" Then another person: "So what are we actually supposed to do?" I didn't have a great answer either. We'd all sat through twenty minutes of a strategy change presentation. We all nodded. None of us was in the conversation.

This isn't a one-off. I see this pattern everywhere – across teams, across functions, across seniority levels. People show up to meetings. They sit. They nod. They leave without having processed what was said. Full attendance, zero presence.

Most advice on listening tells you to add behaviors – nod more, paraphrase, ask clarifying questions. I think the fix is the opposite. Presence isn't something you perform. It's what remains when you stop doing everything else.

This isn't a listening guide. What follows are patterns – ways presence breaks down in rooms I've been in, including when I was the one not present. Once you start recognizing them, they're hard to unsee.

How presence breaks down

I see this play out in a few distinct ways. Sometimes it's obvious – a room full of silent nods. Sometimes it's subtler.

Silent nods, two flavors

This is the one that does the most damage, and it's invisible in the moment.

Flavor one: don't understand, won't say so. That all-hands I described at the top – this was exactly it. A room full of people nodding along to something they hadn't actually processed. Nobody asked a clarifying question. Nobody said, "I don't follow." The meeting felt smooth. The hallway afterward told the real story.

Flavor two: understand fine, disagree, won't surface it. A manager assigned a direction. The team silently accepted. Later, over coffee, one of the engineers told me directly: he disagreed, saw no value in it, and was going to do the minimum to get it off his plate.

The disagreement never entered the room where it could have changed something. It lived in the hallway, in the coffee chat, in the Slack DM. By the time it surfaced – slow execution, low quality, quiet attrition – nobody connected it back to that meeting where everyone nodded.

Both flavors look identical in the room. Silent nods. But one is a comprehension gap, and the other is a trust gap. A speaker who's actually paying attention to the room – not just broadcasting – can sometimes tell the difference. Yet most speakers don't, because they're focused on getting through their material. They're not listening either.

Solving before hearing

Engineers are trained to decompose problems. It's the job. But that instinct kicks in too early in conversations. Someone starts describing a situation. Ten seconds in, your brain is already pattern-matching and mapping it to something you've seen before, composing a solution.

The problem: you're now building your response in parallel with the other person still talking. You're not tracking their reasoning – you're constructing your own. By the time they finish, you've got an answer to a question they didn't ask, because you stopped listening after the first thirty seconds.

Listening for your turn

This one I catch myself doing. Someone is talking, and I'm holding my argument in my head, waiting for a gap. The physical tell is real – I lean forward, I nod faster. Not because I agree, but because I want them to finish so I can say my thing.

People who are vocal and not shy do this without thinking. They fire their thoughts at every opening. The quieter person's point never fully lands because the room has already moved on to the next response.

The worst part: you don't realize you're doing it until someone's point slides past you and you can't recall what they actually said. And when it happens to me – when I'm the one not being heard – I push through. Force the idea through anyway. It works, but it's not solving the listening problem. It's bulldozing past it.

It happens in 1:1s too

You might think this is a group-meeting problem – too many people, too many laptops, too much noise. But it happens in the most intimate format we have: the 1:1.

I've caught myself doing it. Someone is talking to me one-on-one, and my eyes drift to the screen. I'm scanning something – a notification, a message, a tab I forgot to close. Tracking something that isn't the person in front of me. Not because I don't care. Because the pull of the inbox is stronger than any of us wants to admit.

This is the version of the problem that's hardest to name, because it feels personal.

What being present actually looks like

Closing the lid

I was in a meeting with my team. Everyone had laptops open – the usual. Someone was presenting an idea, and the room had that familiar energy: half-listening, half-typing.

I closed my laptop lid. Started looking at the speaker. Actually tracking what they were saying.

I saw something shift in their eyes – a kind of recognition that someone was paying attention. Then one of my teammates closed their lid too. Then another. Not everyone, but enough. The conversation changed. Suddenly, there were real questions. Pushback. Ideas building on each other. Before the lids closed, it was a broadcast. After, it was a dialogue.

That small act – closing a laptop – did more for the conversation than any "great question" could have. Because the question only works if you've been listening long enough to know what to ask.

Dropping performance

Here's the thing about active listening that most advice gets wrong: it's not a technique. If you try to perform it – the nodding, the eye contact, the clarification question every twenty seconds – people notice. They feel the gap between your body language and your attention.

It only works when you actually stop thinking about everything else. You look at the person. Not staring – eye contact from time to time. You let them flesh out their thoughts. You can be completely silent. That's fine. Nobody needs you to nod on a timer.

The real skill is subtraction, not addition. Remove the distractions. Remove the urge to respond. Remove the performance. What's left is just… focus. You're in this conversation and nowhere else. That's simpler than any framework, and harder than all of them.

Holding back

Before a recent vacation, I was handing over a project I'd been leading – research done, core team assembled, draft plan written. My teammate needed to pick it up and run with it.

In our 1:1 handover meeting, she started asking questions and sharing her own vision for how to approach it. I remember the tension in my own head – I knew the answers. I had my understanding, my vision, my plan. Every instinct said: jump in, push the knowledge, save time.

I didn't say a word. I just listened. Looked at her screen when she showed me things. Nodded – naturally, not performatively.

She covered the whole story. Answered the majority of her own questions just by working through the plan and my notes. And then she proposed a couple of insights I had missed in the planning – things she added as follow-up topics for the project team.

I'm fairly sure that wouldn't have happened if I'd started in expert mode. The handover would have been faster, but worse. She would have received my understanding instead of building her own. And those insights she caught? Buried under my monologue.

Pulling the rope

Here's how I think about it. When someone is explaining a hard problem – the kind that doesn't have clean edges – they're pulling a heavy rock up from the bottom on a rope.

You can stand there and watch them pull. Or you can step closer, grab the rope, and help.

Active listening is the grabbing. Resist the urge to decompose immediately. Follow the other person's thread before you start your own. That's the rope-pulling: making the effort to understand what they're actually saying before you decide whether you agree.

Key takeaways

The next time you're in a room, you'll probably catch one of these – in yourself or someone else. That's the point. Once you see the pattern, you can't run it on autopilot anymore. You don't need a technique for what happens next. You just stop doing the thing you caught yourself doing.

Presence is not attendance. Being in the room and being in the conversation are different things.
Engineers default to problem-solving mode, which means they start constructing responses before they've finished hearing the problem. Resist the decomposition reflex.
Silent nods hide two different problems: people who don't understand and people who disagree. Both look the same. Both break things differently.
Listening is labor, not passivity. You're helping the speaker articulate something hard, not waiting for them to finish so you can talk.
Listening isn't a skill to add to your toolkit. It's setting down reflexes you already have – the solving, the responding, the performing.

A Deep Dive Into Terraform Static Code Analysis Tools: Features and Comparisons

Serhii Vasylenko — Tue, 16 Apr 2024 23:32:03 +0000

Many teams employ Terraform by HashiCorp to efficiently manage their infrastructure, leveraging its ability to automate the lifecycle of complex environments. Yet, integrating security scanning into Terraform pipelines often remains overlooked, exposing these environments to potential security risks and compliance issues.

This article explores several prominent static code analyzers that support Terraform code and focus on its security scanning. This comparison will guide teams in choosing the right tool to enhance their security measures within Terraform workflows, ensuring safer and more compliant infrastructure management.

Here are the tools we'll be reviewing: KICS, tfsec, Trivy, Terrascan, Checkov, and Semgrep OSS.

While many of these tools also support other platforms and technologies, this review will concentrate exclusively on their functionality with Terraform.

Why use Static Code Analysis for Terraform

Static code analysis tools are necessary to enhance the security of Terraform-managed infrastructures. Unlike linters, these tools focus not on syntax errors or coding style but delve deeply into the code to identify security vulnerabilities and potential compliance issues without running the actual code. This proactive approach to security helps safeguard the infrastructure from potential threats before deployment.

Key Benefits

Early Detection: Identifies security vulnerabilities and misconfigurations early in development, preventing them from reaching production.
Compliance Assurance: Ensures Terraform code complies with industry standards and internal security policies.
Automated Security Integration: Seamlessly integrates with CI/CD pipelines, automating security checks to maintain a continuous focus on security.
Actionable Insights: Delivers detailed vulnerability reports, facilitating swift and effective resolution.
Scalability: Effectively handles increasing project complexity and size, maintaining rigorous security standards without additional manual effort.

Expected Features

Policy Coverage: The tool should offer comprehensive scanning capabilities to detect security vulnerabilities specific to Infrastructure as Code.
Customizable Security Policies: It must allow users to define and adjust security policies and severity levels to align with specific project needs or compliance requirements.
Seamless Integration: The analyzer should integrate effortlessly with existing CI/CD tools and version control systems, facilitating a smooth workflow.
Detailed Reporting: Clear and actionable reports are crucial. The tool should prioritize issues based on severity and provide practical steps for remediation.
Scanning Customization: Users should be able to tailor the scanning process to focus on particular aspects of the codebase, enabling targeted and efficient security assessments.

With a clear understanding of the necessary features in a static code analyzer, which tools on the market best fulfill these criteria?

Let's take a closer look at some leading options!

Meet the Static Code Analyzers for Terraform

Following on what makes a static code analyzer robust, let's dive into some open-source tools that exemplify these essential features.

I picked six tools for my review. I know there are more on the market, but I focused on open-source, free-to-use tools and those that provide at least >100 out-of-the-box scanning policies for Terraform.

KICS (stands for "Keeping Infrastructure as Code Secure"):
Owner/Maintainer: Checkmarx
Age: First released on GitHub on November 30th, 2020
License: Apache License 2.0

tfsec
Owner/Maintainer: Aqua Security (acquired in 2021)
Age: First released on GitHub on March 5th, 2019
License: MIT License
tfsec project is no longer actively maintained in favor of the Trivy tool. But because many people still use it and it's quite famous, I added tfsec to this comparison.
However, I recommend against using it for new projects.

Trivy
Owner/Maintainer: Aqua Security
Age: First released on GitHub on May 7th, 2019
License: Apache License 2.0
backward-compatible with tfsec

Terrascan
Owner/Maintainer: Tenable (acquired in 2022)
Age: First release on GitHub on November 28th, 2017
License: Apache License 2.0

Checkov
Owner/Maintainer: Prisma Cloud by Palo Alto Networks (acquired in 2021)
Age: First released on GitHub on March 31st, 2021
License: Apache License 2.0

Semgrep OSS
Owner/Maintainer: Semgrep
Age: First release on GitHub on February 6th, 2020
License: GNU Lesser General Public License v2.1

These tools are essential in enhancing Terraform's security posture and reflect a strong collaboration between open-source communities and enterprise backing. This blend ensures that the tools are not only accessible but also robustly maintained and up-to-date.

Let’s explore how these tools stack up regarding features and usability.

Comparing Out-of-the-Box Policies and Terraform Providers

Understanding the number and variety of default policies each tool offers is crucial for those just beginning to explore security automation for Terraform.

The extent of out-of-the-box policies can significantly ease the integration process of static analysis by providing immediate and comprehensive insights into potential security and compliance issues. Similarly, the number of supported Terraform Providers also plays a critical role.

In this chapter, we delve into these foundational features across observed tools, helping you pinpoint which one could best satisfy your requirements for robust, ready-to-use security scanning.

Tool	Policies	Supported Terraform Providers
KICKS	663	aws, azure, gcp, kubernetes, alicloud, databricks, github, nifcloud
tfsec	154	aws, azure, gcp, digitalocean, kubernetes, cloudstack, github, openstack, oracle
Trivy	322	aws, azure, gcp, digitalocean, cloudstack, github, oracle, openstack
Terrascan	790	aws, azure, gcp, digitalocean, kubernetes, docker, github
Checkov	2110	aws, azure, gcp, digitalocean, kubernetes, github, gitlab, ibm, linode, openstack, alicloud
Semgrep OSS	362	aws, azure, gcp

As you can see, all tools support the "Big Three" cloud service Terraform providers—AWS, Azure, and GCP—for managing resources on these popular platforms.

With over 2000 out-of-the-box policies, Checkov significantly stands out from the competition. This tool also leads in the total number of supported Terraform providers.

While the default policies provide a strong foundation for security scanning, the ability to tailor these policies is just as crucial. Next, we'll explore how each tool accommodates custom policy capabilities, allowing you to fine-tune the policies to fit your project's specific requirements.

Custom Policy Capabilities

Default policies serve as the foundation, but the nuances of each project demand the extension of this base.

Here, we delve into how each tool enables you to add custom policies, thus enhancing and refining the provided defaults.

While all six tools support adding custom policies to their default set, they differ in terminology: 'policy' is the common term, whereas KICS refers to them as 'queries,' and Semgrep calls them 'rules.'

Regarding policy syntax:

OPA Rego syntax is used by KICS, Trivy, tfsec, and Terrascan. It's a powerful language widely adopted in the industry, though there's a learning curve that could pay dividends for future projects.

YAML syntax is used by Checkov and Semgrep. This offers a familiar and straightforward start, with Checkov also allowing policies to be written in Python, albeit with some constraints. With YAML, the ease of use is balanced against the limitations set by the tool's capabilities.

Understanding these differences will guide you to a tool that matches your security requirements, your team's expertise, and the scope of your infrastructure projects.

To illustrate, here is an example of a KICS Rego policy checking for default RDS instance ports:

package Cx

import data.generic.common as common_lib
import data.generic.terraform as tf_lib

CxPolicy[result] {
    db := input.document[i].resource.aws_db_instance[name]

    enginePort := common_lib.engines[e]

    db.engine == e
    db.port == enginePort

    result := {
        "documentId": input.document[i].id,
        "resourceType": "aws_db_instance",
        "resourceName": tf_lib.get_resource_name(db, name),
        "searchKey": sprintf("aws_db_instance[%s].port", [name]),
        "issueType": "IncorrectValue",
        "keyExpectedValue": sprintf("aws_db_instance[%s].port should not be set to %d", [name, enginePort]),
        "keyActualValue": sprintf("aws_db_instance[%s].port is set to %d", [name, enginePort]),
        "searchLine": common_lib.build_search_line(["resource", "aws_db_instance", name, "port"], []),
    }
}

And an example of a Checkov YAML policy forbidding specific EC2 instance types:

---
metadata:
 name: "Org's compute instances should not be p5.48xlarge or p4d.24xlarge"
 id: "ACME_AWS_FORBIDDEN_EC2_TYPES"
 category: "NETWORKING"
definition:
 or:
 - cond_type: "attribute"
   resource_types:
    - "aws_instance"
   attribute: "instance_type"
   operator: "not_equals"
   value: "p5.48xlarge"
 - cond_type: "attribute"
   resource_types:
   - "aws_instance"
   attribute: "instance_type"
   operator: "not_equals"
   value: "p4d.24xlarge"

With the ability to tailor policies to our specific needs, we'll next explore each tool's capacity to integrate broadly, determining how well they play with the rest of our tech stack.

Integration Capabilities

Integration capabilities are the cornerstone of efficient DevOps practices.

This section will evaluate how each static code analyzer enhances your tech stack through seamless integration with other systems and technologies.

We will assess each tool against four key integration points that are vital for development workflows:

Docker Image: Ensures easy deployment across any container-supported environment.
IDE Plugins: Facilitates real-time feedback and improves code quality directly within the developer's workspace.
CI/CD Systems: Supports direct integration through plugins or extensions, eliminating the need for manual downloads or CLI setups.
Pre-commit Hook: Provides an early security checkpoint by scanning code before it is committed, catching errors at the initial stages.

Tool	Docker Image	IDE Plugins	CI/CD Systems	Hook
KICKS	✅	VSCode	Github Actions, Terraform Cloud, Codefresh	✅
tfsec	✅	VSCode, JetBrains, Vim	Github Actions	❌
Trivy	✅	VSCode, JetBrains, Vim	Azure DevOps, GitHub Actions, Buildkite, Dagger, Semaphore, CircleCI, Concourse CI	❌
Terrascan	✅	VSCode	GitHub Actions, Atlantis	✅
Checkov	✅	VSCode, JetBrains	GitHub Actions	✅
Semgrep OSS	✅	VSCode, JetBrains, Emacs, Vim	GitLab	✅

In addition to the table above, here are a few noteworthy features of some tools:

Checkov supports OpenAI integration to suggest remediations. But be careful because AI tends to hallucinate.
KICS supports applying auto-remediation for some of its out-of-the-box policies. This also applies to custom policies, where you can define remediations and apply them automatically.
Terrascan is the only one that provides the VSCode extension to create and test custom policies written in Rego.

Having covered the integration capabilities, let’s now focus on the output formats each tool provides.

Output Formats Provided

Output formats extend the utility of static code analysis, facilitating integration with the CI/CD feedback loop and enabling its use as an artifact in subsequent CI jobs.

This chapter examines the variety of formats each tool supports for this purpose.

Each tool offers a range of output formats tailored to different needs.

For GitLab users: For teams leveraging GitLab's security scanning, KICS, Checkov, and Semgrep OSS are equipped with compatible output formats, facilitating smooth GitLab integration.

For GitHub users: SARIF's adoption as an industry standard, particularly by GitHub for code scanning, makes it a must-have. All tools assessed offer SARIF support, ensuring interoperability and broad utility.

JUnit Reports: The availability of JUnit output is crucial for capturing test results in a format recognizable by various CI systems. Trivy, Terrascan, Checkov, and Semgrep OSS support this, enabling clear visualization of test outcomes and enhancing the feedback loop within CI pipelines.

Beyond these, each tool supports additional formats, enriching their application and versatility. Here's the full breakdown of the output formats, complementing the standard CLI output:

Tool	Supported Output Formats
KICKS	ASFF, CSV, Code Climate, CycloneDX, GItLab SAST, HTML, JSON, JUnit, PDF, SARIF, SonarQube
tfsec	Checkstyle, CSV, HTML, JSON, JUnit, Markdown, SARIF
Trivy	ASFF, Cosign, CycloneDX, JSON, SARIF, SPDX
Terrascan	JSON, JUnit, SARIF, XML, YAML
Checkov	CSV, CycloneDX, GItLab SAST, JSON, JUnit, SARIF, SPDX
Semgrep OSS	Emacs, GitLab SAST, JSON, JUnit, SARIF, Vim

Moving from output formats to operational adaptability, let's investigate the customization options for scanner settings. This important feature allows each tool to align with varied project demands.

Customizing Scanner Settings

This chapter moves beyond the default scanner settings and delves into scanner settings' customizability, ensuring that tools can be calibrated for any development environment or security requirement.

I will evaluate each tool against criteria that define a tool's adaptability and user-friendliness:

Targeted Scans: Select specific directories for scanning or exclusion to focus on pertinent areas and skip irrelevant ones.
In-Code Ignore Policies: Enable ignore directives within code to skip checks when exceptions apply selectively.
Severity Thresholds: Set reporting to include only findings above a chosen severity level, concentrating on the most impactful issues.
Configuration File: Employ configuration files for consistency and collaboration, enabling a 'configuration as code' approach.
TF Variables Interpolation: Interpret and evaluate Terraform variables for an accurate security assessment of IaC.
Module Scanning: For complete coverage, scans should include both local and remote (public/private) Terraform modules.

Based on these criteria, the following table offers a comparative view of how each tool performs, giving you a clear snapshot of their customization capabilities:

Tool	Targeted Scans	Ignore Policies	Min Severity	Config File	Variables Interpolation	Module Scanning
KICKS	✅	✅	✅	✅	✅	⁉️️
tfsec	✅	✅	✅	✅	✅	✅
Trivy	✅	✅	✅	✅	✅	✅
Terrascan	✅	✅	✅	✅	✅	✅
Checkov	✅	✅	✅	✅	✅	✅
Semgrep OSS	✅	✅	✅	❌	✅	❌

Most reviewed tools meet nearly all the criteria set for scanner setting customization, demonstrating their flexibility and advanced capabilities. However, there are notable features worth considering:

KICS: Provides limited module scanning capabilities, restricted to some public modules from the Terraform registry, and does not cover local or private custom modules.
Terrascan & Trivy: Both feature server modes that centralize vulnerability databases. This centralization facilitates a unified approach to applying policies and configurations, enhancing consistency and efficiency for teams and reducing the management overhead of diverse policies across multiple projects.
Semgrep: It doesn't support scanner configuration files; instead, it uses the "config" word to call the rule sets and accepts such configs. Notably, it also does not support the scanning of Terraform modules at all.

Terraform Security Scanning: The Big Picture and Top Pick

Here's a comprehensive comparison summary to guide your selection of the most suitable Terraform static code analyzer:

Tool	Default Policies	Custom Policies	Integration	Output Formats	Customization
KICKS	663	OPA Rego	✅Docker, ✅IDE, ✅CI/CD, ✅Git Hook	ASFF, CSV, Code Climate, CycloneDX, GItLab SAST, HTML, JSON, JUnit, PDF, SARIF, SonarQube	✅Targeted Scans, ✅Ignore Policies, ✅Min Severity, ✅Config File, ✅Variables Interpolation, ❌Module Scanning
tfsec	154	OPA Rego	✅Docker, ✅IDE, ✅CI/CD, ❌Git Hook	Checkstyle, CSV, HTML, JSON, JUnit, Markdown, SARIF	✅Targeted Scans, ✅Ignore Policies, ✅Min Severity, ✅Config File, ✅Variables Interpolation, ✅Module Scanning
Trivy	322	OPA Rego	✅Docker, ✅IDE, ✅CI/CD, ❌Git Hook	ASFF, Cosign, CycloneDX, JSON, SARIF, SPDX	✅Targeted Scans, ✅Ignore Policies, ✅Min Severity, ✅Config File, ✅Variables Interpolation, ✅Module Scanning
Terrascan	790	OPA Rego	✅Docker, ✅IDE, ✅CI/CD, ✅Git Hook	JSON, JUnit, SARIF, XML, YAML	✅Targeted Scans, ✅Ignore Policies, ✅Min Severity, ✅Config File, ✅Variables Interpolation, ✅Module Scanning
Checkov	2110	YAML, Python	✅Docker, ✅IDE, ✅CI/CD, ✅Git Hook	CSV, CycloneDX, GItLab SAST, JSON, JUnit, SARIF, SPDX	✅Targeted Scans, ✅Ignore Policies, ✅Min Severity, ✅Config File, ✅Variables Interpolation, ✅Module Scanning
Semgrep OSS	362	YAML	✅Docker, ✅IDE, ✅CI/CD, ✅Git Hook	Emacs, GitLab SAST, JSON, JUnit, SARIF, Vim	✅Targeted Scans, ✅Ignore Policies, ✅Min Severity, ❌Config File, ✅Variables Interpolation, ❌Module Scanning

Integrating a Terraform security scanning into your development pipeline is a proven strategy to boost your security posture. These tools detect potential vulnerabilities early and enforce best practices and compliance standards, representing a proactive approach to infrastructure security.

For teams not yet utilizing these tools, Checkov is my top recommendation:

Biggest number of default policies and supported Terraform providers for a quick start.
Custom policy support in YAML and Python for flexible policy creation.
Wide integration options with Docker, IDEs, CI/CD systems, and Git Hooks for a smooth workflow.

Please share your favorite tool in the comments below! Also, let me know if I missed a cool product that should have been included in the review.

Mastering AWS API Gateway V2 HTTP and AWS Lambda With Terraform

Serhii Vasylenko — Thu, 11 Jan 2024 15:18:00 +0000

With a solid foundation in AWS API Gateway and Lambda for serverless architecture, my recent deep dive into these cloud computing services felt like uncovering new layers in familiar territory. This article aims to be a comprehensive guide for developers and DevOps professionals looking to master serverless solutions using AWS and Terraform.

The article provides an in-depth guide to combining AWS API Gateway V2 HTTP API (yes, this is the official name of that service 😄) and AWS Lambda services to implement a simple, robust, and cost-effective serverless back-end using Terraform.

The journey was enlightening and engaging, especially as I were transforming these services into Infrastructure as Code. Through this article, I aim to share those moments of insight and the practical, hands-on tips that emerged from weaving these AWS services into a seamless, serverless architecture.

Navigating the System Design: HTTP API Gateway and Lambda in Action

Beginning our journey, we examine the complexities of serverless architecture, focusing on HTTP API and Lambda. A comprehensive system diagram will guide us as we analyze each component’s function and their collaborative roles in the larger infrastructure.

In our architecture, the HTTP API delegates access control to the Lambda function called “Authorizer”. This function stands as the gatekeeper, ensuring that only legitimate requests pass through to the underlying business logic.

The HTTP API can have multiple routes (e.g., “/calendar,” “/meters,” and so on) and use different Authorizers per route or a single one for all of them. Clients that send their requests to the API must include specific identification information in their request header or query string. In this project, I go with a single authorizer to keep it simple.

Upon receiving a request, the API service forwards a payload to the Authorizer containing metadata about the request, such as headers and query string components. The Authorizer processes this metadata (headers, in my case) to determine the request’s legitimacy.

The decision, Allow or Deny, is passed back to the API, and if allowed, the API service then forwards the original request to the back-end, which, in this case, is implemented by additional Lambda functions. Otherwise, the client gets a response with a 403 status code, and the original request is not passed to the back-end.

Behind The Decision: Why Such a Setup?

Choosing the right architectural setup is critical in balancing simplicity, cost-efficiency, and security. In this section, we uncover why integrating AWS HTTP API Gateway with Lambda Authorizer is a compelling choice, offering a streamlined approach without compromising security.

Cost-Effectiveness: Balancing Performance and Price

The AWS HTTP API is noteworthy for its streamlined and simple design compared to other API Gateway options. That translates directly into cost savings for businesses. Its efficiency makes it an ideal choice for cost-effective serverless computing, especially for those looking to optimize their cloud infrastructure with Terraform automation. Here is a more detailed comparison of different API Gateway options — Cost optimization.

Security with Lambda Authorizer. This option means a Lambda function used for authorization, which is lean and efficient. It generally requires a bare minimum of resources. It executes quickly, particularly when configured with the ARM-based environment and 128M RAM allocation, costing $0,0000017 per second of running time, with $0.20 per 1M requests per month.

💰 This pricing and performance combination are well-suited for rapid, lightweight authorizations. Together with AWS Lambda as a back-end, it makes a cost-effective solution. For example, if we add a few more Lambdas to back-end and assume that our setup receives 10000 requests per month, it would cost around $0.6 per month. Here is the link to detailed calculations — AWS Pricing Calculator.

Simplicity in Configuration: The Power of Header-Based Authorization

A header-based authentication method facilitates straightforward client-server communication, often requiring less coding and resources to implement compared to more complex schemes.

Although HTTP API offers stronger JWT-based authorization and mutual TLS authentication, header-based authorization remains a suitable choice for simpler applications that prioritize ease and quickness. By the way, there is also an option for IAM-based authorization whose core idea is the “private API” or internal usage of the API (e.g., solely inside the VPC, no internet), but with “IAM Anywhere,” this can be expanded to practically anywhere. 😁

This architecture suits applications requiring rapid development and deployment without complex authorization mechanisms. It’s ideal for small to medium-sized serverless applications or specific use cases in larger systems where quick, cost-effective, and secure access to APIs is a priority.

💡 Imagine a retail company wanting to manage its inventory efficiently. By leveraging AWS API Gateway and Lambda, they can develop a system where each item’s RFID tags are scanned and processed through an API endpoint. When a product is moved or sold, its status is updated in real-time in the database, facilitated by Lambda functions. This serverless architecture ensures high availability and scalability and significantly reduces operational costs, a crucial factor for the highly competitive retail industry. This example showcases how our serverless setup can be effectively utilized in retail for streamlined inventory tracking and management.

Exploring AWS Lambda: Features and Integration

Diving into AWS Lambda, this section explores its features and indispensable role within the serverless infrastructure. We will unravel the complexities of Lambda functions and examine the practicalities of deploying and managing these functions within the project.

AWS Lambda Runtime and Deployment Model

🚀 Choosing the AWS Lambda runtime arm64, combined with the OS-only runtime based on Amazon Linux 2023, strategically boosts cost efficiency and performance. This choice aligns with the best practices for serverless computing in AWS, offering an optimal solution for those seeking to leverage AWS services for scalable cloud solutions.

Particularly effective for Go-based functions, this runtime configuration is lean yet powerful. For applications in other languages, delving into language-specific runtimes based on AL 2023 can also leverage the latest efficiencies of AWS-managed operating systems.

I also welcome you to read this benchmarking analysis to get more insights about the ARM-based environment for AWS Lambda — Comparing AWS Lambda Arm vs. x86 Performance, Cost, and Analysis.

The .zip deployment model is chosen for its simplicity, avoiding additional management of the image registry (ECR) and Docker images. Also, AWS automatically patches .zip functions for the latest runtime security and bug fixes.

Efficient Terraform Coding for AWS Lambda

In our architecture, AWS Lambda functions serve dual purposes — as an authentication gatekeeper and a robust back-end for business logic. Despite varying code across functions, their configurations share much of similarities.

By adhering to the DRY (Don't Repeat Yourself) principle, I have crafted a Terraform module to streamline the management of Lambda functions and their dependencies. This approach ensures maintainable and scalable infrastructure. The module's structure is as follows:

aws_lambda_function — to describe the core configuration of the function
aws_iam_role + aws_iam_role_policy + aws_iam_policy_document — to manage the access from Lambda to other resources (e.g., SSM Parameter Store)
aws_cloudwatch_log_group — to keep the execution logs
aws_ssm_parameter — to store sensitive information (e.g., secrets) and other configurations that we should keep separate from the source code.

This Terraform module implements a project-specific use case for Lambda functions. However, if you're seeking for a generic all-in-one module for AWS Lambda, I recommend checking out this one — Terraform AWS Lambda Module by Anton Babenko.

To efficiently develop Terraform code for Lambda functions, use the following techniques:

Use local values, expressions, and variables to implement consistent naming across different resources logically grouped by a module or project;
Use function environment variables to connect the code with SSM Parameter Store parameters or Secrets Manager secrets to protect sensitive data like tokens or credentials;
Use for_each meta-argument and for expression to reduce the amount of code and automate the configuration for resources of the same type (e.g., ssm_parameter) or code blocks within a resource.

Below is a practical example illustrating these Terraform strategies in action:

locals {  
  full_function_name = "${var.project_name}-${var.function_name}"  
}  
resource "aws_lambda_function" "this" {  
  function_name = local.full_function_name  
  role          = aws_iam_role.this.arn  
  architectures = ["arm64"]  
  filename      = var.deployment_file  
  package_type  = "Zip"  
  runtime       = "provided.al2023"  
  handler       = "bootstrap.handler"  
  timeout       = var.function_timeout  
  environment {  
    variables = { for item in var.function_ssm_parameter_names : upper(replace(item, "-", "_")) => aws_ssm_parameter.function_ssm_parameters[item].name }  
  }  
}  

resource "aws_ssm_parameter" "function_ssm_parameters" {  
  for_each = var.function_ssm_parameter_names  
  name     = "/projects/${var.project_name}/lambda/${var.function_name}/${each.value}"  
  type     = "SecureString"  
  key_id   = data.aws_kms_alias.ssm.arn  
  value    = "1"  
  lifecycle {  
    ignore_changes = [  
      value,  
    ]  
  }  
}  

resource "aws_cloudwatch_log_group" "this" {  
  name              = "/aws/lambda/${local.full_function_name}"  
  log_group_class   = "STANDARD"  
  retention_in_days = 7  
}

The complete terraform module code is available in the project repository.

In this Terraform code, I deliberately hardcoded specific arguments for an optimal Lambda runtime configuration, ensuring efficiency and performance.

Then variables and local values, set only once, implement a naming convention for all resource arguments, making it easy to understand the infrastructure and change the naming and attributes later.

Lambda's environment variables and corresponding SSM parameters coexist effectively with the help of for_each and for. I used the for_each meta-argument to dynamically create SSM Parameter resources and the for expression to configure environment variables in AWS Lambda. This also means that if the function_ssm_parameter_names variable value is not provided, then Terraform does not create either SSM parameter resources or the environment code block inside the Lambda resource because the default value of that variable is an empty set.

By the way, I have another blog post that explains several techniques to enhance your Terraform proficiency — check it out!

Invoking Lambda: Permissions and Resource-Based Policies

Configured with just a few input variables, the Terraform module efficiently outputs the aws_lambda_function resource. This streamlined output is then adeptly used to facilitate subsequent configurations within the HTTP API.

module "lambda_api_gw_authorizer" {  
  source          = "./modules/lambda"  
  deployment_file = "../backend/lambda-apigw-authorizer/deployment.zip"  
  function_name   = "api-gateway-authorizer"  
  project_name    = local.project_name  
  function_ssm_parameters = [  
    "authorization-token"  
  ]  
}

module "lambda_calendar_backend" {  
  source          = "./modules/lambda"  
  deployment_file = "../backend/lambda-calendar-backend/deployment.zip"  
  function_name   = "calendar-backend"  
  project_name    = local.project_name  
  function_ssm_parameters = [  
    "google-api-oauth-token",  
    "google-api-credentials"  
  ]  
}

As an example of module output usage, here is the configuration of aws_lambda_permissions resource that I use outside the AWS Lambda module to allow the HTTP API service to invoke the function used as Authorizer:

resource "aws_lambda_permission" "allow_api_gw_invoke_authorizer" {  
  statement_id  = "allowInvokeFromAPIGatewayAuthorizer"  
  action        = "lambda:InvokeFunction"  
  function_name = module.lambda_api_gw_authorizer.lambda.function_name  
  principal     = "apigateway.amazonaws.com"  
  source_arn    = "${aws_apigatewayv2_api.this.execution_arn}/authorizers/${aws_apigatewayv2_authorizer.header_based_authorizer.id}"  
}

The Lambda resource-based policy combines the trust and permission policies, and provides a simple yet efficient way to grant other AWS services or principals the ability to invoke Lambda functions. It is important to note that for an API to invoke a function, Lambda requires its execution ARN, not the resource ARN.

As a side note, check out this AWS Lambda Operator Guide, which offers specialized advice on developing, securing, and monitoring applications based on AWS Lambda.

Let's switch to the HTTP API part to see how it looks and learn how it integrates Lambda functions.

Deep Dive into HTTP API Gateway

Now, we focus on the HTTP API Gateway, delving into its essential concepts, seamless integration with AWS Lambda, and using Terraform efficiently for streamlined configuration.

But before we do that, and since we have partially covered the Terraform code already, I'd like to illustrate the logical connection between three main components of the project's Terraform codebase: AWS Lambda, HTTP API, and API Routes.

I will explain the API Route module in detail a bit later, but for now, for the broader context, here is what happens inside Terraform:
AWS HTTP API code logically represents the "global" (within a project) set of resources and uses the function created by the Lambda Terraform module for the Authorizer configuration. Meanwhile, the API Route Terraform module configures specific routes for the HTTP API (hence, requires some info from it) with integration to back-ends implemented by Lambdas (hence, requires some info from them, too).

Back to HTTP API overview. The following components of the HTTP API constitute its backbone:

Route — a combination of the HTTP method (e.g., GET or POST) with the API route (e.g., /meters). For example: "POST /meters". Routes can optionally use Authorizers — a mechanism to control access to the HTTP API.
Integration — the technical and logical connection between the Route and one of the supported back-end resources. For example, with AWS Lambda integration, API Gateway sends the entire request as input to a back-end Lambda function and then transforms the Lambda function output to a front-end HTTP response.
Stage and Deployment — A stage serves as a designated reference to a deployment, essentially capturing a snapshot of the API at a certain point. It's employed to control and optimize a specific deployment version. For instance, stage configurations can be adjusted to tailor request throttling, set up logging, or establish stage variables to be used by API (if needed).

Implementing AWS API Gateway V2 HTTP API with Terraform

Below, I detail the Terraform resources essential for implementing the HTTP API, ensuring a transparent and effective setup:

aws_apigatewayv2_api — the HTTP API itself;
aws_apigatewayv2_route — the Route for the API that must specify the integration target (e.g., Lambda) and, optionally, the Authorizer;
aws_apigatewayv2_authorizer — the Authorizer to use for Routes;
aws_apigatewayv2_integration — the resource that specifies the back-end where the API sends the requests (e.g., AWS Lambda);
aws_lambda_permission — the resource-based policy for AWS Lambda to allow the invocations from the API;
aws_apigatewayv2_stage — the name of the Stage that references the Deployment.

Applying Terraform for HTTP API Gateway and Lambda Authorizer

The HTTP API is the simplest in the API Gateway family (so far), so its Terraform resource has relatively few configuration options, most of which can be left at their default values.

As for the Authorizer, it can have two options for letting API know its decision: simple response and IAM policy.

The simple response just returns a Boolean value to indicate whether the API should allow the request (True) or forbid it (False).

The IAM policy option is customizable and allows crafting custom policy statements that allow granular access to explicitly provided resources.

In this project, I follow the way of simplicity and use the "simple response", so the response from Lambda Authorizer to HTTP API looks as follows:

{
  "isAuthorized": true/false
}

Let's review the HTTP API resource along with the API Authorizer that I used for all routes:

resource "aws_apigatewayv2_api" "this" {  
  name          = local.project_name  
  protocol_type = "HTTP"  
}

resource "aws_apigatewayv2_authorizer" "header_based_authorizer" {  
  api_id                            = aws_apigatewayv2_api.this.id  
  authorizer_type                   = "REQUEST"  
  name                              = "header-based-authorizer"  
  authorizer_payload_format_version = "2.0"  
  authorizer_uri                    = module.lambda_api_gw_authorizer.lambda.invoke_arn  
  enable_simple_responses           = true  
  identity_sources                  = ["$request.header.authorization"] 
  authorizer_result_ttl_in_seconds  = 3600  
}  

resource "aws_lambda_permission" "allow_api_gw_invoke_authorizer" {  
  statement_id  = "allowInvokeFromAPIGatewayAuthorizer"  
  action        = "lambda:InvokeFunction"  
  function_name = module.lambda_api_gw_authorizer.lambda.function_name  
  principal     = "apigateway.amazonaws.com"  
  source_arn    = "${aws_apigatewayv2_api.this.execution_arn}/authorizers/${aws_apigatewayv2_authorizer.header_based_authorizer.id}"  
}

The complete code is available in the project repository.

Consider the following key points when Terraforming this part.

identity_sources argument of the aws_apigatewayv2_authorizer resource: This is where I defined what exactly the Authorizer should validate. I used the header named authorization so the Authorizer Lambda function would check its value to decide whether to authorize the request.\
💡 Check out other options available to use as the identity source — Identity sources.

authorizer_uri argument of the aws_apigatewayv2_authorizer resource: It is the invocation ARN of the Lambda function used as Authorizer (not the Lambda's resource ARN).

authorizer_result_ttl_in_seconds argument of the aws_apigatewayv2_authorizer resource: This allows to skip the Authorizer invocation for the given time if a client provided the same identity source values (e.g., authorization header).

AWS API Gateway HTTP can employ the identity sources as the cache key to preserve the authorization results for a while. Should a client provide identical parameters in identity sources within the preset TTL duration, API Gateway will retrieve the result from the cached authorizer instead of calling upon it again. This helps save a lot on AWS Lambda Authorizer invocations and works great with simple scenarios. However, it might be cumbersome if you need severral custom authorization responses per function or if you use custom IAM policies instead of the "simple response" option.

source_arn argument of aws_lambda_permission: Similar to the authorizer_uri argument, this one expects the execution ARN of the HTTP API followed by the Authorizer identifier.

Now, let's see how Routes are codified with Terraform.

Applying Terraform for HTTP API Routes

💡 Because an API typically has multiple routes, creating another Terraform module that implements the configurable HTTP API Gateway route is beneficial. Hence, the aws_apigatewayv2_route, aws_apigatewayv2_integration, and aws_lambda_permission resources would constitute such a module.

This Terraform module implements a specific use case for HTTP API Gateway. However, if you're seeking for a generic all-in-one module for API Gateway, I recommend checking out this one — Terraform AWS API Gateway Module by Anton Babenko.

resource "aws_apigatewayv2_route" "this" {  
  api_id             = var.api_id  
  route_key          = var.route_key  
  authorization_type = "CUSTOM"  
  authorizer_id      = var.authorizer_id  
  target             = "integrations/${aws_apigatewayv2_integration.this.id}"  
}  

resource "aws_apigatewayv2_integration" "this" {  
  api_id                 = var.api_id  
  integration_type       = "AWS_PROXY"  
  connection_type        = "INTERNET"  
  integration_uri        = var.lambda_invocation_arn  
  payload_format_version = "2.0"  
}  

resource "aws_lambda_permission" "this" {  
  statement_id  = "allowInvokeFromAPIGatewayRoute"  
  action        = "lambda:InvokeFunction"  
  function_name = var.lambda_function_name  
  principal     = "apigateway.amazonaws.com"  
  source_arn    = "${var.api_gw_execution_arn}/*/*/*/*"  
}

First, I want to highlight several key aspects for understanding the resources' arguments within that module.

The target argument of the aws_apigatewayv2_route resource implies that the integration ID should be prefixed with the "integrations/" keyword.

While the connection_type argument of the aws_apigatewayv2_integration resource specifies "INTERNET", it does not mean that the Lambda function must have the publicly available URL. This value must be used unless you work with a VPC endpoint for API Gateway for internal usage.

For the source_arn argument in the aws_lambda_permission resource, similar to earlier, it requires the execution ARN of the API. However, this time, it is the integration of the HTTP API Route with Lambda. And the ARN format of this one is different and a bit tricky:

arn:partition:execute-api:region:account-id:api-id/stage/http-method/resource-path

The arn:partition:execute-api:region:account-id:api-id part constitutes the execution ARN of the HTTP API itself, so for the sake of simplicity, I decided to go with wildcards after it.\
For your convenience, here is the detailed specification of API Gateway ARNs.

The HTTP API Route module expects several input variables:

authorizer_id — the identifier of the Authorizer to use on this route;
route_key — the route key for the route, e.g., GET /foo/bar;
api_id — the identifier of HTTP API created earlier;
lambda_invocation_arn — the Invocation ARN of the Lambda function;
lambda_function_name — the name of the Lambda function to integrate with the route;
api_gw_execution_arn — the Execution ARN of the HTTP API that invokes a Lambda function.

Let's take a closer look on API Gateway V2 HTTP API route.

A route consists of an HTTP method and a resource path with an optional variable. Based on the pre-defined convention, it uses a simplified routing configuration and methods request model (comparable to other APIs).

While I was working with the HTTP API, I found this simplified approach to be great because it allows easy access to the request context from AWS Lambda functions, for example:

A path variable in a route, e.g., GET /calendars/{calendar-name}, would be available for the integrated AWS Lambda by its name inside the pathParameter JSON field, e.g., pathParamters.calendar-name, of the event object sent by API to Lambda. In other words, you do not need to explicitly set the mapping between the path variable and its representation to the back-end.
A request query string is parsed into separate parameter-value pairs and available in the queryStringParameters field of the event object sent by API to Lambda. Again, without the explicit mapping configuration.

Here, you can read more about the Route specification of HTTP API and how to transform requests and responses from the API side if you need to adjust something:

Now back to Terraform. Below is the code snippet that illustrates the call of the API Route Terraform module:

module "route_calendars" {  
  source                = "./modules/api-gateway-route"  
  api_id                = aws_apigatewayv2_api.this.id  
  route_key             = "GET /calendars/{calendar-name}"  
  api_gw_execution_arn  = aws_apigatewayv2_api.this.execution_arn  
  lambda_invocation_arn = module.lambda_calendar_backend.lambda.invoke_arn  
  lambda_function_name  = module.lambda_calendar_backend.lambda.function_name  
  authorizer_id         = aws_apigatewayv2_authorizer.header_based_authorizer.id  
}

This module logically relies on both the HTTP API and Lambda resources to configure their integration by implementing the Route.

Enhancing Security and Monitoring of AWS API Gateway V2 HTTP API

Several additional options are available to monitor and protect the HTTP API: logs, metrics, and throttling.

Overview of HTTP API monitoring and protection options

Logging, metrics, and throttling are configured on the Stage level but allow configuration granularity for the Routes.

For logs, you can configure the CloudWatch log group, the log format (JSON, CLF, XML, CSV), and content filters. The logging variables allow you to customize the information that appears in logs. I will provide an example of such a configuration later in the article.

By default, API Gateway sends only API and stage-level metrics to CloudWatch in one-minute periods. However, you can enable detailed metrics and additionally collect the per-route metrics.

To safeguard your HTTP API from excessive requests, you can employ throttling settings, which allow you to set limits per individual route as well as for all routes collectively.

Configuring monitoring and protection for HTTP API with Terraform

Now, let's see how Terraform helps configure the protection and monitoring for HTTP API.

As mentioned earlier, API Gateway applies these configurations at the Stage level, which is why the aws_apigatewayv2_stage resource encapsulates them all.

resource "aws_apigatewayv2_stage" "default" {  
  api_id      = aws_apigatewayv2_api.this.id  
  name        = "$default"  
  auto_deploy = true  
  description = "Default stage (i.e., Production mode)"  
  default_route_settings {  
    throttling_burst_limit = 1  
    throttling_rate_limit  = 1  
  }  
  access_log_settings {  
    destination_arn = aws_cloudwatch_log_group.api_gateway_logs_inkyframe.arn  
    format = jsonencode({  
      authorizerError           = "$context.authorizer.error",  
      identitySourceIP          = "$context.identity.sourceIp",  
      integrationError          = "$context.integration.error",  
      integrationErrorMessage   = "$context.integration.errorMessage"  
      integrationLatency        = "$context.integration.latency",  
      integrationRequestId      = "$context.integration.requestId",  
      integrationStatus         = "$context.integration.integrationStatus",  
      integrationStatusCode     = "$context.integration.status",  
      requestErrorMessage       = "$context.error.message",  
      requestErrorMessageString = "$context.error.messageString",  
      requestId                 = "$context.requestId",  
      routeKey                  = "$context.routeKey",  
    })   
  }  
}

Here, I applied the default throttling settings: for my project, 1 request per second was enough at that point.

🤔 There is a nuance, though, that makes Terraforming API Gateway a little inconvenient — the IAM role that allows API to write logs must be defined on a region level. Therefore, if you maintain several Terraform projects for the same AWS account, you might need to have the following configuration stand separately to avoid conflicts or misunderstandings:

resource "aws_api_gateway_account" "this" {  
  cloudwatch_role_arn = aws_iam_role.api_gateway_cloudwatch_logs.arn  
}  

resource "aws_iam_role" "api_gateway_cloudwatch_logs" {  
  name = "api-gateway-cloudwatch-logs"  
  assume_role_policy = jsonencode({  
    Version = "2012-10-17"  
    Statement = [  
      {  
        Effect = "Allow"  
        Principal = {  
          Service = "apigateway.amazonaws.com"  
        }  
        Action = "sts:AssumeRole"  
      }  
    ]  
  })  
  managed_policy_arns = ["arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs"]  
}  

resource "aws_cloudwatch_log_group" "api_gateway_logs_inkyframe" {  
  name              = "/aws/apigateway/inkyframe"  
  log_group_class   = "STANDARD"  
  retention_in_days = 7  
}

And one more thing about HTTP API deployments and stages. I use the special $default keyword to have a single stage (hence, the default one), and I also used automatic deployments: with any change made to API configuration, AWS will automatically generate a new Deployment and bound it with the Stage. If you prefer controlling deployments manually, there is a special resource exists that implements this — aws_apigatewayv2_deployment

resource "aws_apigatewayv2_deployment" "example" {
  api_id      = aws_apigatewayv2_api.example.id
  description = "Example deployment"

  triggers = {
    redeployment = sha1(join(",", tolist([
      jsonencode(aws_apigatewayv2_integration.example),
      jsonencode(aws_apigatewayv2_route.example),
    ])))
  }

  lifecycle {
    create_before_destroy = true
  }
}

In that case, the aws_apigatewayv2_stage resource requires the deployment_id argument to link itself with a particular Deployment and, therefore, represent the state of the API configuration.

Also, API Gateway requires at least one configured API Route before the deployment is initiated/created. However, these resources do not explicitly depend on each other via attribute references. To avoid the race condition in Terraform, you need to reference the Route resource in the aws_apigatewayv2_deployment resource via the triggers argument (as shown above) or via the depends_on meta-argument. Otherwise, Terraform will try to apply changes to both resources simultaneously.

Afterword: Simplifying Serverless Architectures

In wrapping up our exploration of AWS HTTP API Gateway, AWS Lambda, and Terraform, we've delved into how these powerful tools work in tandem to streamline and enhance serverless architectures. This article aimed to combine my experience with new knowledge and demystify the complexities of used services, showcasing their capabilities in creating efficient, cost-effective solutions for modern cloud-based applications.

We focused on practical implementation and the tangible benefits of combining these technologies. By leveraging Terraform, we've seen how infrastructure management can be simplified, allowing for clearer, more maintainable code. The combination of AWS Lambda and HTTP API Gateway has demonstrated the efficiency of serverless computing, offering scalability and performance without the burden of extensive configuration and management.

This exploration underlines the importance of choosing the right tools and strategies in cloud computing. It reminds developers and architects that creating robust and efficient serverless systems is within reach with a thoughtful approach and the right set of tools. As the cloud landscape continues to evolve, staying informed and adaptable is key to harnessing the full potential of these technologies. 💚

Hello terraform_data! Goodbye Null Resource!

Serhii Vasylenko — Wed, 19 Apr 2023 00:17:47 +0000

Terraform version 1.4 was recently released and brought a range of new features, including improved run output in Terraform Cloud, the ability to use OPA policy results in the CLI, and a built-in alternative to the null resource — terraform_data.

In this blog post, I want to demonstrate and explain the terraform_data resource that serves two purposes:

firstly, it allows arbitrary values to be stored and used afterward to implement lifecycle triggers of other resources
secondly, it can be used to trigger provisioners when there isn't a more appropriate managed resource available.

For those of you, who are familiar with the null provider, the terraform_data resource might look very similar. And you're right!\
Rather than being a separate provider, the terraform_data managed resource now offers the same capabilities as an integrated feature. Pretty cool! \
While the null provider is still available and there are no statements about its deprecation thus far (as of April 2023, v3.2.1), the terraform_data is the native replacement of the null_resource, and the latter might soon become deprecated.

The terraform_data resource has two optional arguments, input and triggers_replace, and its configuration looks as follows:

The input (optional) stores the value that is passed to the resource
The triggers_replace (optional) is a value that triggers resource replacement when changes.

There are two attributes, in addition to the arguments, which are stored in the state: id and output after the resource is created. Let's take a look:

The output attribute is computed based on the value of the input
The id is just a unique value of the resource instance in the state (as for any other resource).

Use case for terraform_data with replace_triggered_by

Let's take a look at the first use case for the terraform_data resource. It is the ability to trigger resource replacement based on the value of the input argument.

A bit of context here: the replace_triggered_by argument of the resource lifecycle meta-argument allows you to trigger resource replacement based on another referenced resource or its attribute.

If you are not yet familiar with the replace_triggered_by, you can check another blog post that explains it.

The replace_triggered_by is a powerful feature, but here is the thing about it: only a resource or its attribute must be specified, and you cannot use a variable or a local value for replace_triggered_by.

But with terraform_data, you can indirectly initiate another resource replacement by using either a variable or an expression within a local value for the input argument.

Let me give you an example here. Consider the following code:

By modifying the revision variable, the next Terraform plan will suggest a replacement action against aws_instance.webserver:

Use case for terraform_data with provisioner

Before we start: HashiCorp suggests (and I also support that) avoiding provisioner usage unless you have no other options left. One of the reasons — additional, implicit, and unobvious dependency that appears in the codebase — the binary, which is called inside the provisioner block, must be present on the machine. \
But let's be real, the provisioner feature is still kicking, and there's always that one unique project that needs it.

Here is the code snippet that demonstrates the usage of the provisioner within the terraform_data resource:

In this example, the following happens:

When resources are created the first time, the provisioner inside terraform_data runs
Sequential plan/apply will trigger another execution of the provisioner only when the private IP of the instance (aws_instance.webserver.private_ip) changes because that will trigger terraform_data recreation. At the same time, no changes to the internal IP mean no provisioner execution.

With its ability to store and use values for lifecycle triggers and provisioners, terraform_data is a powerful tool that can enhance your Terraform configuration.

Although the null provider still has its place in the Terraform ecosystem, terraform_data is its evolution, and its integration as a feature is certainly something to be excited about.

Why not give it a try in your next project and see how it can simplify your infrastructure as code workflows? Keep on coding! 🙌

Amazon VPC Lattice — Build Applications, Not Networks

Serhii Vasylenko — Fri, 17 Mar 2023 23:33:13 +0000

Last year's re:Invent brought a lot of amazing updates to the big family of AWS services. In this blog post, I would like to explain one of such new offerings — Amazon VPC Lattice — an exciting new service that simplifies the networking layer for developers and cloud administrators.

What is Lattice

So what exactly is Amazon VPC Lattice? It is an application layer networking service that enables consistent and secure service-to-service communication without the need for prior networking expertise. With VPC Lattice, you can easily configure network access, traffic management, and network monitoring, making service-to-service communication seamless across VPCs and accounts, irrespective of the underlying compute type.

How it helps

VPC Lattice helps address several use cases, including connecting services at scale, implementing granular access permissions, advanced traffic controls, and observing service-to-service interactions. The service offers connectivity over HTTP/HTTPS and gRPC protocols through a dedicated data plane within VPC. Administrators can use AWS Resource Access Manager (AWS RAM) to control which accounts and VPCs can establish communication through a service network.

What's more, VPC Lattice is designed to be non-invasive and work alongside existing architecture patterns, allowing development teams across your organization to onboard their services incrementally.

How it works

VPC Lattice introduces four key components: Service, Service Directory, Service Network, and Auth Policy. These components simplify how users enable connectivity and apply standard policies to a collection of services. Service networks can be shared across accounts with AWS RAM and associated with VPCs to allow connectivity to a group of services.

Here is the diagram that illustrates the use of Amazon VPC Lattice and the Service Network Manager to create a service network, define policies, and share cross-account access.

The Service Network Manager subset at the top consists of four icons representing the process flow:

1️⃣ The first step involves creating a service network by choosing a name and authentication type.

2️⃣ The second step consists in defining access and monitoring by setting and managing access policies and selecting log destinations.

3️⃣ The third step involves associating clients and services, allowing resources in associated VPCs to access the benefits associated with the service network.

4️⃣The fourth step consists in adding specific assistance or service networks to AWS RAM shares to facilitate cross-account access.

The Service Owner subset at the bottom consists of three steps:

1️⃣ The first step involves creating a service by identifying the benefit and defining access and monitoring.

2️⃣ The second step consists in defining routing by adding listeners and rules that point to the target groups that store the service.

3️⃣ The third step consists in selecting the networks from the service that receives traffic.

Pricing

There are three factors that compose the price of VPC Lattice:

number of provisioned services
traffic volume to and from each service
number of requests each service receives

You can see the detailed and actual price list on the services' pricing page — link, but here is an example:

You provision a service network in the US East (N. Virginia) Region and associate 100 services to it. In a month, each service processes 100 GB of data at 200,000 requests per hour. In this example, we calculate your charges as follows (all prices shown in USD):

Monthly hourly charges:
You pay $0.025 per hour for each service in US East (N. Virginia)
We assume that a month equals 730 hours (8,760 hours in a year/12 months = 730 hours per month)
100 services * $0.025 per hour * 730 hours = $1,825.00 per month

Win-win for Ops and Developers

Overall, VPC Lattice bridges the gap between developers and cloud administrators by providing role-specific features and capabilities. Developers can focus on building applications, not networks, while cloud and network administrators can increase their organization's security posture by enabling authentication, authorization, and encryption consistently across mixed computing environments.

Currently, Amazon VPC Lattice is in Preview in the US West (Oregon) region. I'm excited to see how VPC Lattice will shape the future of networking and make it even easier for developers to build complex applications. 🚀

Some additional resources to learn more about Lattice:

Presentation at re:Invent 2022

A blog post at AWS with examples Introducing VPC Lattice – Simplify Networking for Service-to-Service Communication

Five Practical Ways To Get The Verified EC2 AMI

Serhii Vasylenko — Tue, 26 Jul 2022 15:53:00 +0000

EC2 AMI catalog consists of more than 160k public AMIs — a mix of Images created by users, published by vendors, and provided by AWS.

So how to ensure that an AMI comes from the verified vendor or that is an official AMI published by AWS?

How to find the trusted AMI among them all when you’re about to launch an EC2 Instance?

On AWS, it’s typical that something can be made or done in several ways — that’s awesome. Some of them work better than others, some methods are official, and some you can use just for fun (check that).

In this article, I will describe five ways of getting the official and verified AMI for your next EC2 Instance launch.

Use EC2 Launch Wizard and AMI Catalog to get the official AMI

When launching an EC2 Instance from a Management Console, you can apply the “Verified Provider” filter for the Community AMIs tab to ensure you get an AMI from a verified provider.

The “Verified provider” label means an AMI is owned by an Amazon verified account.

In the following example, I want to make sure that the Ubuntu 20.04 AMI comes from the verified source:

In the past, you had to compare the AMI Owner ID with the publicly shared list of verified Owner IDs for every region.

Not rocket science, but it takes time. So now it’s much more straightforward, thanks to the “Verified Provider” label.

This feature also works great when you are creating a Launch Template. For example, if you want to create a fleet of macOS Instances with AutoScaling.

The Launch Template creation wizard seamlessly guides you from itself to the AMI Catalog (where you can search and pick the AMI) and back again.

Look for verified AMIs on the AMI page

Another interface in the Management Console acts as the AMI browser. It does not have any fancy name except for the “AMIs page”, but you probably already know about it: it looks like a list of AMIs, and you can see it when you click on the “AMIs” menu item on the left side of the EC2 page menu.

The AMI page allows you to leverage the API filters to narrow down the search, and the “Owner alias” filter is the one you need to ensure that an AMI comes from a trusted owner.

Here is how it looks for my search of the official Amazon Linux 2 AMI:

AMIs shared by verified sources have amazon (for AWS) or aws-marketplace (for AWS partners) as the value for the Owner alias filter.

Find the EC2 AMI with Terraform

Finding the official AMI with Terraform is also simple — the aws_ami data source does the job.

For example, here is how you can find the same Amazon Linux 2 AMI by specifying the amazon as the value for the owner argument of the data source:

Compare that with the filters on the AMI page — it looks similar, right? This is because of how Terraform works: it translates your code into API calls and sends them to AWS API endpoints.

If you’re very new to Terraform, I suggest reading this article to understand the basic concepts: Terraform explained in English

Find the official AWS AMI using Describe Images CLI

Sometimes you might need to get the AMI from CLI to pass it along as an argument downstream of the pipeline.

This can be done with the ec2 describe-images command of the AWS CLI

The API filters I mentioned before also work here — use them to narrow your search.

Find the trusted AWS AMI with SSM

Another way that involves AWS CLI is the ssm get-parameter command:

It reveals one helpful feature of the Systems Manager — the Public parameters.

Systems Manager Public parameters are how AWS distributes some widely used artifacts related to their services.

For example, you can find official AMIs for many distributives there: Amazon Linux, Windows, macOS, Bottlerocket, Ubuntu, Debian, and FreeBSD.

Read more at the Finding public parameters documentation page if you want to know more.

Are all verified AMIs good?

The “Verified provider” badge can be earned by a third party only when an AMI developer is registered as a Seller on the AWS Marketplace.

Becoming a Seller there is not trivial and requires some conditions to be met, and the registration process itself also implies submitting the tax and banking information.

Additionally, there are specific policies and review processes apply to all AMIs submitted to the Marketplace.

So it is okay to trust the third-party vendors with the “Verified” badge on a certain level. However, it is also always good to have additional scans and validation of the software you use. 🪲 😉

Serhii VasylenkoFollow

New Lifecycle Options and Refactoring Capabilities in Terraform 1.1 and 1.2

Serhii Vasylenko — Thu, 05 May 2022 13:14:48 +0000

In this blog, I would like to tell you about new cool features that Terraform 1.1 and 1.2 bring. It feels like Terraform has doubled its speed of delivering the new features after they released the 1.0. 🤩

All code examples are based on the AWS Terraform provider ⛅️💛

It’s been only a few months since Terraform 1.1 was released with the moved block that empowers the code refactoring.

Now Terraform 1.2 is almost ready (as I am writing this blog in early May 2022) to bring three new efficient controls to the resource lifecycle.

These are three new expressions: precondition, postcondition, and replace_triggered_by.

Terraform Code Refactoring With the Moved Block

Starting from the 1.1 version, Terraform users can use the moved block to describe the changes in resource or module addresses (or resources inside a module) in the form of code.

Once that is described, Terraform performs the movement of the resource within the state during the first apply.

In other words, what this feature gives you, is the ability to document your terraform state mv actions, so you and other project or module users don’t need to perform them manually.

As your code evolves, a resource or module can have several moved blocks associated with it, and Terraform will thoroughly reproduce the whole history of its movement within a state (i.e., renaming).

Let’s review some examples that illustrate how it works.

Moving a resource

In a module, I have a bucket policy that has a generic, meaningless name. It is used in a module that creates a CloudFront distribution with an S3 bucket.

It’s pretty OK to name a resource like that if you have only a single instance of that kind in your code.

Later, when I need to add another policy to the module, I don’t want to name it “that”. Instead, I want my policies to have meaningful names now.

For example, I could rename the old policy with the terraform state mv command, but other users of my module would not know about that.

That is where the moved block turns out to be helpful: I can document the name change, and later, everyone else who uses my module will get the same renaming.

Terraform follows the instructions inside the module block to plan and apply changes. Although the resource address update is not counted as a change in the Plan output, Terraform will perform that update during apply.

Moving a module

The same approach can be applied to a module.

Here, I use two modules to create static hosting for a website with a custom TLS certificate: a combo of the CloudFront + S3 + ACM services.

Again, if I need to add another couple of the CDN+Certificate modules, I would like to have meaningful names in my code so clearly distinguish one from another.

Therefore, I would add two moved blocks — one per module call.

And by the way, since I renamed the module (from cert to example_com_cert), I need to update all references to that module’s outputs in the code too.

However, there is one nuance: when you rename a module and declare that in the moved block, you need to run the terraform init before applying the change because Terraform must initialize the module with the new name first.

There are some more advanced actions you can make with the moved block:

Implement count and for_each meta-arguments to resources and modules
Break one module into multiple Check the following detailed guide from HashiCorp that explains how to do that — Refactoring

Introducing the moved blocks into your codebase defacto starts the refactoring process for your module users. But the finale of that refactoring happens when you ultimately remove these blocks.

Therefore, here is some advice on how to manage that:

💡 Keep the moved blocks in your code for long. For example, when removing a moved block from the code, Terraform does not treat the new object name as a renaming anymore. Instead, Terraform will plan to delete the resource or module with the old name instead of renaming it.
💡 Keep the complete chains of object renaming (sequence of moves). The whole history of object movement ensures that users with different module versions will get a consistent and predictable behavior of the refactoring.

Lifecycle expressions: precondition, postcondition, and replace_triggered_by

Terraform 1.2 fundamentally improves the lifecycle meta-argument by adding three new configuration options with rich capabilities.

Precondition and Postcondition

When you need to make sure that specific condition is met before or after you create a resource, you can use postcondition and precondition blocks.

The condition here — is some data or information about a resource you need to confirm to apply the code.

Here are a few examples of such conditions:

Validate some attributes of the Data Source that you cannot check using filters or other available arguments;
Confirm the Resource argument that can compound several variables (e.g., list);

💡 Precondition works as an expectation or a guess about some external (but within a module) value that a resource depends on.
💡 Postcondition works as the assurance that a resource fulfils a specific condition so other resources may rely on that. If postcondition fails for a resource, this prevents changes to all other resources that depend on it.

Let’s review this new feature with an example of postcondition usage.

Consider the following case: our module receives AMI ID as the input variable, and that AMI should be used in the Launch Template then; we also have the requirement for the EC2 instance created from that Launch Template — its root EBS size must be equal or bigger than 600 GB.

We cannot validate the EBS size using the variable that accepts the AMI ID. But we can write a postcondition for the Data Source that gets the information about the AMI and reference that Data Source in the Launch Template resource afterward.

The condition argument within the block accepts any of Terraform’s built-in functions or language operators.

The special self object is available only for the postcondition block because it assumes that validation can be performed after the object is created and its attributes are known.

Later, if a module user specifies the AMI with an EBS size lesser than 600 GB, Terraform will fail to create the Launch Template because it depends on the Data Source that did not pass the postcondition check.

Terraform tries to evaluate the condition expressions as soonest: sometimes Terraform can check the value during the planning phase, but sometimes that can be done only after the resource is created if the value is unknown.

Validating module output with precondition

The precondition block is also available for the module outputs.

Just like the variable validation block assures that module input meets certain expectations, the precondition is intended to ensure that a module produces the valid output.

Here is an example: a module that creates an ACM certificate must prevent the usage of a specific domain name in the certificate’s Common Name or its SANs.

In this case, instead of validating several input variables, we can write the validation only once for the output.

Trigger resource replacement with replace_triggered_by

Sometimes it’s needed to specify the dependency in the way that recreates a resource when another resource or its attribute changes.

This is useful when two (or more) resources do not have any explicit dependency.

Consider the following case: you have two EC2 instances, A and B, and need to recreate the B instance if the private IP of instance A is changed.

This is extremely useful when you’re dealing with logical abstractions over the set of resources.

Replacement is triggered when:

💡 Any of the resources referenced in replace_triggered_by are updated
💡 Any value is set to the resource attribute that is referenced in replace_triggered_by

Getting started with Terraform 1.1 and 1.2

If you’re still using older Terraform versions, these new features might be a good motivation for you to upgrade!

Before upgrading, be sure to read the upgrade notes for the specific version at the releases page.

Also, an excellent tool can help with fast switching between different Terraform versions while you’re experimenting — tfswitch.

And if you liked that article, you might also like the others wrote about Terraform: Terraform Proficiency

Serhii VasylenkoFollow

Some Techniques to Enhance Your Terraform Proficiency

Serhii Vasylenko — Tue, 18 Jan 2022 09:05:09 +0000

Terraform built-in functionality is very feature-rich: functions, expressions, and meta-arguments provide many ways to shape the code and fit it to a particular use case.

I want to share a few valuable practices to boost your Terraform expertise in this blog.

Two notes:

1️⃣ Some code examples in this article will work with Terraform version 0.15 and onwards. But if you’re still using 0.14 or lower, here’s another motivation for you to upgrade

2️⃣ I'll be using AWS Terraform provider code examples throughout the article but mentioned Terraform functions and expressions are provider-agnostic and will work the same with other providers

Conditional resources creation

Let’s start from the most popular one (although, still may be new for somebody): whether to create a resource depending on some fact, e.g., the value of a variable. Terraform meta-argument count helps to describe that kind of logic.

Here is how it may look like:

data "aws_ssm_parameter" "ami_id" {
  count = var.ami_channel == "" ? 0 : 1

  name = local.ami_channels[var.ami_channel]
}

The notation var.ami_channel == "" ? 0 : 1 is called conditional expression and means the following: if my variable is empty (var.ami_channel == "" — hence, true) then set the count to 0, otherwise set to 1.

condition ? true_val : false_val

In this illustration, I want to get the AMI ID from the SSM Parameter only if the AMI channel (e.g., beta or alpha) is specified. Otherwise, providing that the ami_channel variable is an empty string by default (""), the data source should not be created.

When following this method, keep in mind that the resource address will contain the index identifier. So when I need to use the value of the SSM parameter from our example, I need to reference it the following way:

ami_id = data.aws_ssm_parameter.ami_id[0].value

The count meta-argument can also be used to create a whole module conditionally:

module "bucket" {
  count = var.create_bucket == true ? 1 : 0
  source = "./modules/s3_bucket"

  name = "my-unique-bucket"
  ...
}

The var.create_bucket == true ? 1 : 0 expression can be written even shorter: var.create_bucket ? 1 : 0 because the create_bucket variable has boolean type, apparently.

But what if you need to produce more than one instance of a resource or module? And still be able to avoid their creation.

Another meta-argument — for_each — will do the trick.

For example, this is how it looks for a module:

module "bucket" {
  for_each = var.bucket_names == [] ? [] : var.bucket_names
  source = "./modules/s3_bucket"

  name = "${each.key}"
  enable_encryption = true
  ...
}

In this illustration, I also used a conditional expression that makes Terraform iterate through the set of values of var.bucket_names if it’s not empty and create several modules. Otherwise, do not iterate at all and do not create anything.

The same can be done for the resources. For example, when you need to create an arbitrary number of security group rules, e.g., to allowlist some IPs for your bastion host:

resource "aws_security_group_rule" "allowlist" {
  for_each = var.cidr_blocks == [] ? [] : var.cidr_blocks
  type = "ingress"
  from_port = 22
  to_port = 22
  protocol = "tcp"
  cidr_blocks = [each.value]
  security_group_id = aws_security_group.bastion.id
}

And just like with the count meta-argument, with the for_each, resource addresses will have the identifier named by the values provided to for_each. For example, here is how I would reference a resource created in the module with for_each described earlier:

bucket_name = module.bucket["photos"].name

Conditional resource arguments (attributes) setting

Now let’s go deeper and see how resource arguments can be conditionally set (or not).

First, let’s review the conditional argument value setting with the null data type:

resource "aws_launch_template" "this" {
  name = "my-launch-template"
  ...
  key_name = var.use_default_keypair ? var.keypair_name : null
  ...

Here I want to skip the usage of the EC2 Key Pair for the Launch Template in some instances and Terraform allows me to write the conditional expression that will set the null value for the argument. It means the absence or omission and Terraform would behave the same as if you did not specify the argument at all.

Dynamic blocks are another case where this technique suits best. Take a look at the following piece of CloudFront resource code where I want to either describe the configuration for the custom error response or omit that completely:

resource "aws_cloudfront_distribution" "cdn" {
  enabled = true
  ...
  dynamic "custom_error_response" {
    for_each = var.custom_error_response == null ? [] : [var.custom_error_response]
    iterator = cer
    content {
      error_code = lookup(cer.value, "error_code", null)
      error_caching_min_ttl = lookup(cer.value, "error_caching_min_ttl", null)
      response_code = lookup(cer.value, "response_code", null)
      response_page_path = lookup(cer.value, "response_page_path", null)
    }
  }
  ...
}

The custom_error_response variable is null by default, but it has the object type, and users can assign the variable with the required nested specifications if needed. And when they do it, Terraform will add the custom_error_response block to the resource configuration. Otherwise, it will be omitted entirely.

Convert types with ease

Ok, let’s move to the less conditional things now 😅

Terraform has several type conversion functions: tobool(), tolist(),tomap(), tonumber(), toset(), and tostring(). Their purpose is to convert the input values to the compatible types.

For example, suppose I need to pass the set to the for_each (it accepts only sets and maps types of value), but I got the list as an input; let’s say I got it as an output from another module. In such a case, I would do something like this:

for_each = toset(var.remote_access_ports)

However, I can make my code cleaner and avoid the explicit conversion — I just need to define the value type in the configuration block of the my_list variable. Terraform will do the conversion automatically when the value is assigned.

variable "remote_access_ports" {
  description = "Ports for remote access"
  type = set(string)
}

While Terraform can do a lot of implicit conversions for you, explicit type conversions are practical during values normalization or when you need to calculate some complex value for a variable. For example, the Local Values, known as locals, are the most suitable place for doing that.

By the way, although there is a tolist() function, there is no such thing as the tostring() function. But what if you need to convert the list to string in Terraform?

The one() function can help here: it takes a list, set, or tuple value with either zero or one element and returns either null or that one element in the form of string.

It’s useful in cases when a resource created using conditional expression is represented as either a zero- or one-element list, and you need to get a single value which may be either null or string, for example:

resource "aws_kms_key" "main" {
  count = var.ebs_encrypted ? 1 : 0

  enable_key_rotation = true
  tags = var.tags
}

resource "aws_kms_alias" "main" {
  count = var.ebs_encrypted ? 1 : 0

  name = "alias/encrypt-ebs"
  target_key_id = one(aws_kms_key.main[*]key_id)
}

Write YAML or JSON as Terraform code (HCL)

Sometimes you need to supply JSON or YAML files to the services you manage with Terraform. For example, if you want to create something with CloudFormation using Terraform (and I am not kidding). Sometimes the AWS Terraform provider does not support the needed resource, and you want to maintain the whole infrastructure code using only one tool.

Instead of maintaining another file in JSON or YAML format, you can embed JSON or YAML code management into HCL by taking benefit of the jsonencode() or yamlencode() functions.

The attractiveness of this approach is that you can reference other Terraform resources or their attributes right in the code of your object, and you have more freedom in terms of the code syntax and its formatting comparable to native JSON or YAML.

Here is how it looks like:

locals {
    some_string = "ult"
  myjson_object = jsonencode({
    "Hashicorp Products": {
      Terra: "form"
      Con: "sul"
      Vag: "rant"
      Va: local.some_string
    }
  })
}

The value of the myjson_object local variable would look like this:

{
  "Hashicorp Products": {
    "Con": "sul",
    "Terra": "form",
    "Va": "ult",
    "Vag": "rant"
  }
}

And here is a piece of real-world example:

locals {
  cf_template_body = jsonencode({
    Resources : {
      DedicatedHostGroup : {
        Type : "AWS::ResourceGroups::Group"
        Properties : {
          Name : var.service_name
          Configuration : [
            {
              Type : "AWS::EC2::HostManagement"
              Parameters : [
                {
                  Name : "auto-allocate-host"
                  Values : [var.auto_allocate_host]
                },
            ...
            ...

Templatize stuff

The last case in this blog but not the least by its efficacy — render source file content as a template in Terraform.

Let’s review the following scenario: you launch an EC2 instance and want to supply it with a bash script (via the user-data parameter) for some additional configuration at launch.

Suppose we have the following bash script instance-init.sh that sets the hostname and registers our instance in a monitoring system:

#!/bin/bash

hostname example.com
bash /opt/system-init/register-monitoring.sh

But what if you want to set a different hostname per instance, and some instances should not be registered in the monitoring system?

In such a case, here is how the script file content will look:

#!/bin/bash

hostname ${system_hostname}
%{ if register_monitoring }
bash /opt/system-init/register-monitoring.sh
%{endif}

And when you supply this file as an argument for the EC2 instance resource in Terraform, you will use the templatefile() function to make the magic happen:

resource "aws_instance" "web" {
  ami = var.my_ami_id
  instance_type = var.instance_type
  ...
  user_data = templatefile("${path.module}/instance-init.tftpl", {
    system_hostname = var.system_hostname
    register_monitoring = var.add_to_monitoring
  })
  ...
}

And of course, you can create a template from any file type. The only requirement here is that the template file must exist on the disk at the beginning of the Terraform execution.

Key takeaways

Terraform is far beyond the standard resource management operations. With the power of built-in functions, you can write more versatile code and reusable Terraform modules.

✅ Use conditional expressions with count and for_each meta-arguments, when the creation of a resource depends on some context or user inout.

✅ Take advantage of implicit types conversion when working with input variables and their values to keep your code cleaner.

✅ Embed YAML and JSON-based objects right into your Terraform code using built-in encoding functions.

✅ And when you need to pass some files to the managed service, you can treat them as templates and make them multipurpose.

Thank you for reading down to this point! 🤗

And if you have some favorite Terraform tricks — I would love to know!

Serhii VasylenkoFollow

Apply Cloudfront Security Headers Policy With Terraform

Serhii Vasylenko — Fri, 05 Nov 2021 13:23:17 +0000

In November 2021, AWS announced Response Headers Policies — native support of response headers in CloudFront. You can read the full announcement here: Amazon CloudFront introduces Response Headers Policies

I said “native” because previously you could set response headers either using CloudFront Functions or Lambda@Edge.

And one of the common use cases for that was to set security headers. So now you don’t need to add intermediate requests processing to modify the headers: CloudFront does that for you with no additional fee.

Manage Security Headers as Code

Starting from the 3.64.0 version of Terraform AWS provider, you can create the security headers policies and apply them for your distribution.

Let’s see how that looks!

First, you need to describe the aws_cloudfront_response_headers_policy resource:

 resource "aws_cloudfront_response_headers_policy" "security_headers_policy" {
  name = "my-security-headers-policy"
  security_headers_config {
    content_type_options {
      override = true
    }
    frame_options {
      frame_option = "DENY"
      override = true
    }
    referrer_policy {
      referrer_policy = "same-origin"
      override = true
    }
    xss_protection {
      mode_block = true
      protection = true
      override = true
    }
    strict_transport_security {
      access_control_max_age_sec = "63072000"
      include_subdomains = true
      preload = true
      override = true
    }
    content_security_policy {
      content_security_policy = "frame-ancestors 'none'; default-src 'none'; img-src 'self'; script-src 'self'; style-src 'self'; object-src 'none'"
      override = true
    }
  }
}

List of security headers used:

The values for the security headers can be different, of course. However, the provided ones cover the majority of cases. And you can always get the up to date info about these headers and possible values here: Mozilla web Security Guidelines

Also, you could notice that provided example uses the override argument a lot. The override argument tells CloudFront to set these values for specified headers despite the values received from the origin. This way, you can enforce your security headers configuration.

Once you have the aws_cloudfront_response_headers_policy resource, you can refer to it in the code of aws_cloudfront_distribution resource inside cache behavior block (default or ordered). For example, in your default_cache_behavior:

resource "aws_cloudfront_distribution" "test" {
  default_cache_behavior {
    target_origin_id = aws_s3_bucket.my_origin.id
    allowed_methods = ["GET", "HEAD", "OPTIONS"]
    cached_methods = ["GET", "HEAD"]
    viewer_protocol_policy = "redirect-to-https"

    # some arguments skipped from listing for the sake of simplicity

    response_headers_policy_id = aws_cloudfront_response_headers_policy.security_headers_policy.id

  }

  # some arguments skipped from listing for the sake of simplicity
}

Serhii VasylenkoFollow

Auto Scaling Group for your macOS EC2 Instances fleet

Serhii Vasylenko — Mon, 25 Oct 2021 08:33:05 +0000

It’s been almost a year since I started using macOS EC2 instances on AWS: there were ups and downs in service offerings and a lot of discoveries with macOS AMI build automation.

And I like this small but so helpful update of EC2 service very much: with mac1.metal instances, seamless integration of Apple-oriented CI/CD with other AWS infrastructure could finally happen.

While management of a single mac1.metal node (or a tiny number of ones) is not a big deal (especially when Dedicated Host support was added to Terraform provider), governing the fleet of instances is still complicated. Or it has been complicated until recent days.

Official / Unofficial Auto Scaling for macOS

With a growing number of instances, the following challenges arise:

Scale mac1.metal instances horizontally
Automatically allocate and release Dedicated Hosts needed for instances
Automatically replace unhealthy instances

If you have worked with AWS before, you know that Auto Scaling Group service can solve such things.

However, official documentation (as of October 2021) states: “You cannot use Mac instances with Amazon EC2 Auto Scaling”.

But in fact, you can.

Combining services to get real power

So how does all that work?

Let’s review the diagram that illustrates the interconnection between involved services:

With the help of the Licence Manager service and Launch Templates, you can set up EC2 Auto Scaling Group for mac1.metal and leave the automated instance provisioning to the service.

License Configuration

First, you need to create a License Configuration so that the Host resource group can allocate the hots.

Go to AWS License Manager -> Customer managed licenses -> Create customer-managed license.

Specify Sockets as the Licence type. You may skip setting the Number of Sockets. However, the actual limit of mac1.metal instances per account is regulated by Service Quota. The default number of mac instances allowed per account is 3. Therefore, consider increasing this to a more significant number.

Host resource group

Second, create the Host resource group: AWS License Manager -> Host resource groups -> Create host resource group.

When creating the Host resource group, check “Allocate hosts automatically” and “Release hosts automatically” but leave “Recover hosts automatically” unchecked. Dedicated Host does not support host recovery for mac1.metal.
However, Auto Scaling Group will maintain the desired number of instances if one fails the health check (which assumes the case of host failure as well).

Also, I recommend specifying “mac1” as an allowed Instance family for the sake of transparent resource management: only this instance type is permitted to allocate hosts in the group.

Optionally, you may specify the license association here (the Host group will pick any compatible license) or select the license you created on step one.

Launch Template

Create Launch Template: EC2 -> Launch templates -> Create launch template.

I will skip the description of all Launch Template parameters (but here is a nice tutorial), if you don’t mind, and keep focus only on the items relevant to the current case.

Specify mac1.metal as the Instance type. Later, in Advanced details: find the Tenancy parameter and set it to “Dedicated host”; for Target host by select “Host resource group”, and once selected the new parameter Tenancy host resource group will appear where you should choose your host group; select your license in License configurations parameter.

Auto Scaling Group

Finally, create the Auto Scaling Group: EC2 -> Auto Scaling groups -> Create Auto Scaling group.

The vital thing to note here — is the availability of the mac1.metal instance in particular AZ.

Mac instances are available in us-east-1 and 7 more regions, but not every Availability Zone in the region supports it. So you must figure out which AZ supports the needed instance type.

There is no documentation for that, but there is an AWS CLI command that can answer this question: describe-instance-type-offerings — AWS CLI 2.3.0 Command Reference

Here is an example for the us-east-1 region:

> aws ec2 describe-instance-type-offerings --location-type availability-zone-id --filters Name=instance-type,Values=mac1.metal --region us-east-1 --output text

INSTANCETYPEOFFERINGS   mac1.metal  use1-az6    availability-zone-id
INSTANCETYPEOFFERINGS   mac1.metal  use1-az4    availability-zone-id

Keep that nuance in mind when selecting a subnet for the mac1.metal instances.

When you know the AZ, specify the respective Subnet in the Auto Scaling Group settings, and you're ready to go!

Bring Infrastructure as Code here

I suggest describing all that as a code. I prefer Terraform, and its AWS provider supports the needed resources. Except one.

As of October 2021, resources supported :

The Host resource group is not yet supported by the provider, unfortunately. However, we can use CloudFormation in Terraform to overcome that: describe the Host resource group as aws_cloudformation_stack Terraform resource using CloudFormation template from a file.

Here is how it looks like:
Click here to see the code snippet

resource "aws_licensemanager_license_configuration" "this" {
  name                     = local.full_name
  license_counting_type    = "Socket"
}

resource "aws_cloudformation_stack" "this" {
  name          = local.full_name # the name of CloudFormation stack
  template_body = file("${path.module}/resource-group-cf-stack-template.json")
  parameters = {
    GroupName = local.full_name # the name for the Host group, passed to CloudFormation template
  }
  on_failure = "DELETE"
}

And the next code snippet explains the CloudFromation template (which is the resource-group-cf-stack-template.json file in the code snippet above)

Click here to see the code snippet

{
  "Parameters" : {
    "GroupName" : {
      "Type" : "String",
      "Description" : "The name of Host Group"
    }
  },
  "Resources" : {
    "DedicatedHostGroup": {
      "Type": "AWS::ResourceGroups::Group",
      "Properties": {
        "Name": { "Ref": "GroupName" },
        "Configuration": [
          {
            "Type": "AWS::ResourceGroups::Generic",
            "Parameters": [
              {
                "Name": "allowed-resource-types",
                "Values": ["AWS::EC2::Host"]
              },
              {
                "Name": "deletion-protection",
                "Values": ["UNLESS_EMPTY"]
              }
            ]
          },
          {
            "Type": "AWS::EC2::HostManagement",
            "Parameters": [
              {
                "Name": "allowed-host-families",
                "Values": ["mac1"]
              },
              {
                "Name": "auto-allocate-host",
                "Values": ["true"]
              },
              {
                "Name": "auto-release-host",
                "Values": ["true"]
              },
              {
                "Name": "any-host-based-license-configuration",
                "Values": ["true"]
              }
            ]
          }
        ]
      }
    }
  },
  "Outputs" : {
    "ResourceGroupARN" : {
      "Description": "ResourceGroupARN",
      "Value" : { "Fn::GetAtt" : ["DedicatedHostGroup", "Arn"] }
    }
  }
}

The aws_cloudformation_stack resource will export the DedicatedHostGroup attribute (see the code of CloudFromation template), which you will use later in the Launch Template resource.

Pro tips

If you manage an AWS Organization, I have good news: Host groups and Licenses are supported by Resource Access Manager service. Hence, you can host all mac instances in one account and share them with other accounts — it might be helpful for costs allocation, for example. Also, check out my blog about AWS RAM if you are very new to this service.

To solve the “which AZ supports mac metal” puzzle, you can leverage the aws_ec2_instance_type_offerings and aws_subnet_ids data sources.

Costs considerations

License Manager is a free of charge service, as well as Auto Scaling, and Launch Template.

So it’s all about the price for mac1.metal Dedicated Host which is $1.083 per hour as of October 2021. However, Saving Plans can be applied.

Please note that the minimum allocation time for that type of host is 24 hours. Maybe someday AWS will change that to 1-hour minimum (fingers crossed).

Oh. So. ASG.

The Auto Scaling for mac1.metal opens new possibilities for CI/CD: you can integrate that to your favorite tool (GitLab, Jenkins, whatsoever) using AWS Lambda and provision new instances when your development/testing environments need that. Or you can use other cool ASG stuff, such as Lifecycle hooks, to create even more custom scenarios.

Considering the “hidden” (undocumented) nature of the described setup, I suggest treating it as rather testing than production-ready for now. However, my tests show that everything works pretty well: hosts are allocated, instances are spawned, and the monthly bill grows.

I suppose AWS will officially announce all this in the nearest future. Along with that, I am looking forward to the announcement of Monterey-based AMIs and maybe even M1 chip-based instances (will it be mac2.metal?).

And I want to say thanks (thanks, pal!) to OliverKoo, who started digging into that back in April'21.