DEV Community

Debapriya Dey
Debapriya Dey

Posted on

WAFR Automation Toolkit

#architecture #automation #aws #infrastructure

The Problem: A Real Customer Story

The Situation

A large enterprise customer, operating a multi-service AWS environment, approached Storm Reply for an AWS Well-Architected Framework Review. The goal was straightforward: understand where their cloud architecture stood against AWS best practices and identify the most critical risks to address.

What wasn't straightforward was the scale of the review itself.

A full Well-Architected Framework Review spans six pillars, ~60+ questions, and hundreds of individual best-practice choices — each requiring evidence from the customer's AWS environment to justify a yes, no, or not applicable answer. For the reviewer, this meant:

  • Logging into the customer's AWS account and manually running dozens of CLI commands across multiple services
  • Cross-referencing outputs against AWS documentation to determine whether a practice was actually in place
  • Writing detailed justification notes for every answered question
  • Repeating this across all six pillars before a single answer could be submitted to the Well-Architected Tool

In practice, a thorough WAFR was taking several days of consultant time — most of it spent on repetitive, low-judgement tasks like running aws guardduty list-detectors or checking whether Multi-AZ was enabled on RDS instances.

What the Review Found

When the review was finally completed, the findings painted a clear picture of risk. The customer's environment had several high-priority gaps:

  • Identity and Access Management — excessive permissions and no consistent enforcement of least-privilege principles, creating significant unauthorized access risk (Business Risk Score: 72/100)
  • Single Points of Failure — critical components deployed without redundancy or failover, leaving the architecture vulnerable to outages (Business Risk Score: 64)
  • Cloud Financial Management — no formal FinOps practice, no cost governance processes, and uncontrolled resource sprawl leading to significant overspend (Business Risk Score: 64)
  • Operational Excellence gaps — no structured process for learning from incidents or continuously improving operations (Business Risk Score: 56)
  • Incomplete data encryption in transit — not all services enforcing TLS, leaving data transfers exposed

These weren't surprises. But the time it took to surface them — and the manual effort required to document them — was the real problem.

The Deeper Challenge

Beyond the time cost, there was a quality and consistency problem. Different reviewers, reviewing similar environments, would ask the same CLI commands but interpret the outputs differently. Notes were written inconsistently. Some choices were skipped under time pressure. The final report quality depended heavily on the individual consultant's experience and thoroughness.

There was also no way to efficiently reuse knowledge across engagements. If a customer had already confirmed "yes, we have a FinOps team" in a previous conversation, there was no structured way to carry that forward into the review tool.

What We Built to Solve It

The WAFR Automation Toolkit was designed to tackle this problem directly. The core idea: eliminate the manual, API-checkable work entirely, so consultants can focus their time on the judgement calls that actually require human expertise.

Automated Evidence Gathering

Instead of a consultant manually running commands, the tool runs them automatically. For each of the hundreds of best-practice choices across all six pillars, we mapped the right AWS CLI, Prowler, or Steampipe command to gather evidence. The tool executes these in parallel against the customer's account and collects the raw output.

For the IAM gaps found in the customer review above, for example, the tool automatically runs permission checks, checks for multi-account separation via AWS Organizations, and verifies whether IAM Access Analyzer is enabled — without the reviewer touching a terminal.

AI-Assisted Verdict Engine

Raw command output is not always a clear pass or fail. The tool applies a two-stage evaluation:

  1. Fast heuristic rules — pattern matching for obvious signals like "not enabled", empty resource lists, or Steampipe alarm rows. These cover the majority of cases instantly.
  2. AWS Bedrock (Claude) — when the heuristic can't decide, the raw output is sent to Claude Haiku along with the choice title and context. Claude returns a structured PASS, FAIL, or PARTIAL verdict with its reasoning. This handles the ambiguous cases that previously required a consultant to interpret manually.

Claude also drafts the per-question Notes automatically — 2–4 sentence reviewer-voice summaries that get submitted directly to the Well-Architected Tool. Notes that used to take 10 minutes each to write are now generated in seconds and edited, not authored, by the reviewer.

Prowler Integration

For Security and Reliability pillars, the tool runs Prowler's official WAFR compliance frameworks automatically and maps results directly to WAFR choice IDs. This covers a significant portion of the security findings — like the IAM and encryption gaps identified in the customer review — without any manual effort.

Customer Profile Presets

Not everything in a WAFR can be checked via API. Does the customer have a FinOps team? Is there a formal incident response process? An Architecture Review Board?

These questions came up repeatedly in the customer engagement described above — and there was no automated way to answer them. The toolkit solves this with Customer Profile Presets: a structured form that captures organisational attributes once, maps them to the relevant WAFR choices, and applies them automatically to every scan. Presets are reusable across multiple reviews for the same customer.

The Result

What previously took several days of a consultant's time can now be completed in a fraction of that. The reviewer enters credentials, runs the scan, spends time reviewing and adjusting the AI-generated answers, and submits — all from one interface. The quality is consistent, the evidence is traceable, and the notes are written.

The toolkit doesn't replace the consultant's judgement. It removes the work that doesn't require it.

What is this tool?

The WAFR Automation Toolkit is a local web application that automates AWS Well-Architected Framework Reviews (WAFRs). It takes a reviewer from raw AWS credentials all the way to submitted answers in the AWS Well-Architected Tool — with automated evidence gathering, AI-assisted evaluation, and minimal manual effort along the way.

A traditional WAFR involves manually checking hundreds of AWS configurations across six pillars and writing detailed justification notes for each answer. This tool automates the majority of that work by scanning the customer's AWS environment directly, evaluating the findings using a mix of rule-based heuristics and AI (AWS Bedrock / Claude), and pre-populating answers in the AWS Well-Architected Tool on behalf of the reviewer.

Who is it for?

This tool is designed for Storm Reply consultants conducting Well-Architected Framework Reviews on behalf of customers. It requires:

  • A valid @reply.* email address to log in
  • Temporary AWS credentials for the customer account being reviewed
  • An AWS profile (SSO or standard) for the reviewer's own account, where the Well-Architected Tool workloads are managed

How does it work? (High-Level Flow) 

1. Log in (Storm Reply email)
         ↓
2. Enter customer AWS credentials
         ↓
3. Validate credentials + check IAM permissions
         ↓
4. Run automated scans across all 6 WAFR pillars
   ├── Prowler compliance framework scans
   ├── AWS CLI commands per choice
   ├── Steampipe topology detection
   ├── Local code scanning (optional)
   └── AI evaluation via AWS Bedrock (Claude)
         ↓
5. Review auto-populated answers in the UI
         ↓
6. Submit finalised answers to the AWS Well-Architected Tool
Enter fullscreen mode Exit fullscreen mode

Core Concepts

Dual-Account Model

The tool operates across two AWS accounts:

Account Purpose How credentials are provided
Customer account The AWS environment being reviewed Pasted into the UI as temporary credentials (access key, secret, session token). Held in memory only — never written to disk.
Reviewer account Where the Well-Architected Tool workload lives Resolved automatically from the local AWS profile/SSO chain. Never entered in the UI.

AWS Bedrock (for AI evaluation) also uses the reviewer's local credentials.

The Six WAFR Pillars

The tool covers all six standard Well-Architected pillars:

  1. Operational Excellence
  2. Security
  3. Reliability
  4. Performance Efficiency
  5. Cost Optimisation
  6. Sustainability

Each pillar contains multiple questions, and each question contains multiple best-practice choices. The tool evaluates every choice independently and produces a PASS, FAIL, or PARTIAL verdict.

How Choices are Evaluated

For each best-practice choice, the tool applies a priority-ordered evaluation chain:

  1. Prowler — If Prowler's WAFR compliance framework already assessed this choice during its scan, that verdict is used first. Prowler is run once per scan (for Security and Reliability) and covers a significant portion of choices automatically.

  2. CLI commands — For choices not covered by Prowler, the tool runs targeted aws-cli or steampipe commands against the customer account. Outputs are evaluated using:

    • Heuristic rules first — fast pattern matching for common signals (permission errors, empty results, "not enabled" responses, Steampipe alarm/ok rows).
    • AWS Bedrock (Claude Haiku) as fallback — when the heuristic can't reach a clear verdict, the raw command output is sent to Claude with the choice title and context, and the AI returns a structured PASS/FAIL/PARTIAL decision with reasoning.

  3. Code scan — For a small number of Reliability choices that are invisible to AWS API scanning (e.g., idempotency patterns, circuit breakers, client timeouts), the tool optionally scans a local source code repository for relevant patterns.

  4. Customer preset — Organisational attributes that can't be verified via any API (e.g., does a FinOps team exist? Is there a formal incident response process?) are captured as a reusable "customer profile preset" and used as a fallback for applicable choices.

  5. Topology detection — A background Steampipe query maps the customer's workload topology (ECS, EKS, Lambda, RDS, DynamoDB, CloudFront, Direct Connect, etc.) and is used as a hint for topology-gated choices (e.g., a reliability check about VPN/Direct Connect only applies if the workload actually uses them).

The Application (Page by Page)

Login

A simple login gate using a Storm Reply email address (@reply.* domain). No password — this is a local-only tool for internal use.

Credentials

Enter the customer's temporary AWS credentials (Access Key ID, Secret Access Key, Session Token) and the AWS regions to scan. On submission, the tool validates the credentials via STS and silently checks which IAM actions are permitted — so that scans can skip checks gracefully where access is denied, rather than failing noisily.

Review (Overview)

The main dashboard. Shows:

  • Overall review progress (questions answered vs. total)
  • Evaluation method breakdown (how many choices were assessed by Prowler, heuristic, LLM, preset, or code scan)
  • Per-pillar status cards
  • Buttons to scan all pillars at once or individually
  • Controls to apply a customer preset and optionally provide a local code repository path for code scanning

Pillar Detail

Drill into any pillar to see all questions and their choices. Each choice shows its scan result badge, the raw command output, and the AI's reasoning. PASS choices are auto-selected. Reviewers can manually override selections, toggle "Does not apply" for irrelevant choices (with a reason), and edit the AI-drafted question notes.

Create / Update Workload

Connect to an existing Well-Architected Tool workload or create a new one. Submit all reviewed answers and AI-drafted notes directly to the AWS Well-Architected Tool API in one click.

Presets

Create and manage reusable customer profile presets. Each preset captures organisational attributes (e.g., whether a SecOps team, FinOps team, or Architecture Review Board exists) and maps them to specific WAFR choices that can't be verified via AWS APIs. Presets are stored locally and can be reused across multiple reviews for the same customer.

Additional Components

Prowler-as-a-Service

A cloud-native extension that runs Prowler as an ephemeral ECS Fargate task rather than locally. It is triggered via API Gateway → Lambda, assumes a read-only cross-account IAM role in the customer account, runs the Prowler scan, and stores results in an encrypted S3 bucket. A companion Python script can auto-update Well-Architected Tool answers from the CSV output. This component has its own Terraform infrastructure and Streamlit UI.

Steampipe / Powerpipe Mod

A standalone Powerpipe mod that provides 200+ automated WAFR checks as interactive dashboards (accessible via browser) or CLI benchmarks. Reviewers can run it independently for a deeper visual assessment, separate from the main toolkit scan flow.

Technology Stack

Layer Technology
Frontend React, TypeScript, Vite, Tailwind CSS, Zustand
Backend Node.js, Express, TypeScript
AWS SDK AWS SDK v3 (JavaScript)
AI / LLM AWS Bedrock — Claude Haiku (cross-region inference)
Scanning Prowler 5.x, Steampipe, AWS CLI
Infrastructure Terraform / OpenTofu (ECS Fargate, ECR, VPC, S3, IAM)
CI/CD GitLab CI (build → test → infrastructure validate → deploy)

Security Design

  • Customer credentials are never persisted. They are held in memory on the backend for the duration of the scan and stored in sessionStorage on the frontend (cleared when the browser tab closes, unless "Stay signed in" is checked).
  • No credentials are written to disk or logged. CLI commands receive credentials via environment variables injected into spawned processes.
  • Reviewer credentials are never entered in the UI. The tool uses the local AWS credential chain (fromNodeProviderChain) for Bedrock and the Well-Architected Tool API.
  • The backend only accepts connections from localhost:5173 (the Vite frontend). It is not intended to be exposed to a network.

Running Locally 

# Install all dependencies (frontend + backend)
npm install

# Start both the frontend (port 5173) and backend (port 3001)
npm run dev
Enter fullscreen mode Exit fullscreen mode

Environment variables (optional overrides):

Variable Default Description
BEDROCK_MODEL_ID eu.anthropic.claude-haiku-4-5-20251001-v1:0 Override the Bedrock model used for AI evaluation
BEDROCK_REGION eu-central-1 Region for Bedrock API calls
WAFR_REGION eu-central-1 Region for the Well-Architected Tool API
PORT 3001 Backend server port

Prerequisites:

  • Node.js 18+
  • AWS CLI configured with a local profile/SSO for the reviewer account
  • Prowler installed and available on PATH (for Security + Reliability scans)
  • Steampipe with the AWS plugin (for topology detection and some choice scans)

Project Structure 

wafr-automation-toolkit-gas/
├── frontend/           React + TypeScript UI (Vite)
│   └── src/
│       ├── routes/     Pages: Login, Credentials, Review, PillarDetail, WafrCreate, Presets
│       ├── state/      Zustand stores (auth, credentials, scan, forms, presets, topology)
│       ├── api/        API client wrappers
│       └── components/ Shared UI components
├── backend/            Express + TypeScript API server
│   └── src/
│       ├── routes/     REST endpoints (sts, permissions, scanner, wafr, topology, presets)
│       └── services/   Core engines (scanner, evaluator, prowler, topology, codescan, notes)
├── mappings/           WAFR pillar-to-scanner mapping JSON files (one per pillar)
├── catalog/            Reference WAFR answer sets (default + LIE engagement baseline)
├── config/             Scanner configuration (scanners.yaml)
├── infra/              Terraform/OpenTofu for cloud-deployed infrastructure
├── prowler-as-a-service/  Cloud-native Prowler ECS runner (separate architecture)
├── steampipe/          Powerpipe/Steampipe mod for standalone WAFR dashboards
└── docker/             Dockerfile for ECS container image
Enter fullscreen mode Exit fullscreen mode

Key AWS Services Used

Service How it's used
STS Validate customer credentials (GetCallerIdentity)
IAM SimulatePrincipalPolicy to check permissions before scanning
AWS CLI (various) Targeted commands per WAFR choice across EC2, S3, RDS, Lambda, GuardDuty, CloudWatch, Cost Explorer, and more
Bedrock (Claude) Evaluate CLI output for verdict, judge code scan snippets, draft question notes
Steampipe Multi-service topology queries and individual choice assessments
Prowler Full WAFR security and reliability compliance framework scans
Well-Architected Tool Read/create workloads, submit final answers with notes
Direct Connect Topology detection — determines if the workload uses DX connections
ECS / ECR / S3 / VPC / IAM Infrastructure for the Prowler-as-a-Service cloud deployment

Reach Out to Us

Interested in modernizing your cloud infrastructure and building enterprise-grade solutions? Storm Reply is driven by continuous learning and practical innovation. We specialize in designing and delivering scalable AWS architectures that support customers throughout their cloud journey, from early assessment to production-ready deployment.

With deep experience in AWS architecture, data engineering, and security best practices, we help enterprises migrate with confidence and move faster on their cloud transformation goals.

Let’s connect and explore how we can support your modernization initiatives.

🌐 Website: https://www.stormreply.cloud/

💼 LinkedIn: https://www.linkedin.com/company/storm-reply/posts/?feedView=all
Date: August 2026

Top comments (0)