DEV Community: Alpesh

Automating Amazon EVS (Elastic VMware Service) Day-0 to Day-2 Operations with CloudFormation and Step Functions

Alpesh — Wed, 29 Jul 2026 10:38:55 +0000

Amazon Elastic VMware Service (EVS) reached general availability in August 2025, enabling customers to run VMware Cloud Foundation (VCF) directly within their Amazon VPC. While EVS simplifies the infrastructure layer, operationalizing it at scale — provisioning environments, configuring vCenter, setting up NSX segments, attaching storage, and integrating with ITSM — still requires significant automation.

This post presents a pattern for automating EVS from Day-0 (environment creation) through Day-2 (monitoring, alerting, billing) using AWS CloudFormation and Step Functions. The approach is modular, allowing teams to adopt individual components independently.

The Challenge with EVS at Scale

Standing up a single EVS environment involves:

Creating a dedicated VPC with correct subnet sizing (/20 minimum)
Deploying the EVS environment with Route Server and BGP peering
Configuring vCenter and SDDC Manager (SSL certificates, depot credentials, domain settings)
Creating NSX-T segments for workload networking
Optionally deploying FSx for NetApp ONTAP as shared storage
Setting up CloudWatch alarms for bare-metal host health
Integrating with ITSM (ServiceNow) for incident management
Tracking host counts for billing and reporting

When managing EVS across multiple accounts and regions, this must be repeatable, auditable, and hands-off.

Architecture: Modular CloudFormation with Step Function Orchestration

The solution uses a modular approach — each operational concern is a separate CloudFormation stack, orchestrated by a central Step Function:

                    ┌─────────────────────────┐
                    │  Step Function Orchestrator  │
                    └──────────┬──────────────┘
                               │
        ┌──────────────────────┼──────────────────────┐
        │                      │                      │
        ▼                      ▼                      ▼
┌───────────────┐    ┌──────────────────┐    ┌──────────────────┐
│  Pre-Setup    │    │ Network Creation │    │  EVS Environment │
│  (DDB, IAM)  │    │ (VPC, Subnets)   │    │  (Route Server,  │
│               │    │                  │    │   BGP, Deploy)   │
└───────────────┘    └──────────────────┘    └──────────────────┘
        │                      │                      │
        ▼                      ▼                      ▼
┌───────────────┐    ┌──────────────────┐    ┌──────────────────┐
│  Configure    │    │  NSX Segment     │    │  FSx ONTAP       │
│  vCenter/SDDC │    │  Creation        │    │  (Optional NFS)  │
└───────────────┘    └──────────────────┘    └──────────────────┘
        │                      │                      │
        ▼                      ▼                      ▼
┌───────────────┐    ┌──────────────────┐    ┌──────────────────┐
│  CW Alarms    │    │  ITSM / Alarm    │    │  Billing &       │
│  (Host Health)│    │  Event Handler   │    │  Reporting       │
└───────────────┘    └──────────────────┘    └──────────────────┘

Design Principles

One stack, one concern — Network, compute, storage, monitoring, and ITSM are independent stacks
Step Function as orchestrator — Handles sequencing, retries, and cross-account coordination
DynamoDB for state — Tracks deployment status, lock management, and host inventory
SSM Parameter Store as configuration registry — All naming conventions and cross-stack references flow through SSM

Day-0: Environment Provisioning

Network Foundation

EVS requires a /20 CIDR minimum for the management VPC. The network stack provisions:

Dedicated VPC with DNS support enabled
Private subnets in the target AZ (EVS is single-AZ per environment)
Route tables prepared for BGP route injection from Route Server

# Key design decision: /20 minimum for EVS VPC
Parameters:
  CidrBlock:
    Type: String
    Default: 10.23.0.0/20
    Description: "CIDR block for EVS VPC (minimum /20)"

Route Server and EVS Deployment

The EVS environment stack handles:

Route Server creation with a private ASN (64512-65534 range)
BGP peer configuration for NSX Edge connectivity (EVS uses BGP for overlay-to-underlay routing)
EVS environment deployment with i4i.metal instances, VMware license keys, and VCF configuration
Route persistence — keeps routes active briefly after BGP session drops to avoid routing flaps

# Route Server enables dynamic routing between EVS NSX and AWS VPC
Parameters:
  RouteServerASN:
    Type: Number
    Default: 65001
    Description: ASN for Route Server (private range)
  PeerASN:
    Type: Number
    Default: 65000
    Description: ASN for EVS NSX Edge nodes

Step Function Orchestration

The central Step Function coordinates the multi-step deployment:

Check DynamoDB — Validates the target account doesn't already have an EVS deployment (prevents duplicates)
Pre-setup — Creates DynamoDB state tables, IAM roles, and SSM parameters
Network deployment — Provisions VPC, subnets, and route tables
EVS environment creation — Deploys Route Server and EVS cluster
Post-install tasks — Configures vCenter, creates NSX segments, deploys optional storage

The Step Function uses a DynamoDB-based locking mechanism to prevent concurrent deployments to the same account — critical in multi-tenant managed service environments.

Day-1: Configuration Automation

vCenter and SDDC Manager Configuration

After EVS deploys the VCF stack, vCenter and SDDC Manager need configuration:

SSL certificate rotation (replacing self-signed certs with organization certificates)
VMware Depot credentials for patching and updates
Domain and DNS configuration
NFS datastore mounting (if FSx ONTAP is deployed)

This automation uses an EC2 jumphost (Windows Server) with SSM Run Command to execute PowerShell/PowerCLI scripts against vCenter and SDDC Manager APIs — necessary because these APIs are only accessible from within the VPC.

NSX-T Segment Creation

Workload VMs need network segments. The NSX automation:

Retrieves NSX Manager credentials from Secrets Manager (stored during EVS deployment)
Connects to the NSX Manager REST API via the EC2 jumphost
Creates DHCP server profiles for workload subnets
Provisions overlay segments with gateway and DHCP configuration

# NSX segment creation via REST API (executed on jumphost via SSM)
$nsxManager = "evs01-nsx01.customer.domain"
$headers = @{ Authorization = "Basic $encodedCreds" }
# Create DHCP profile, then segment with gateway

FSx for NetApp ONTAP (Optional NFS Datastore)

For environments needing shared storage beyond vSAN:

FSx ONTAP file system deployed in the EVS VPC subnet
Storage Virtual Machine (SVM) created with NFS protocol
Volume provisioned with specified capacity
Security groups configured for NFS traffic from ESXi hosts

# FSx ONTAP supports multiple deployment types
Parameters:
  DeploymentType:
    Type: String
    AllowedValues:
      - MULTI_AZ_2   # Recommended: standby in second AZ
      - SINGLE_AZ_2  # High performance: 6+ GBps
      - MULTI_AZ_1
      - SINGLE_AZ_1

Day-2: Monitoring, Alerting, and Billing

CloudWatch Alarms for EVS Hosts

EVS runs on bare-metal i4i.metal instances. The monitoring stack:

Maintains a DynamoDB table of active EVS host instance IDs
Uses DynamoDB Streams to detect when hosts are added or removed
Automatically creates CloudWatch alarms for each host (CPU, status checks, network)
All alarms publish to a central SNS topic

This is event-driven — when a new host is added to the DynamoDB table, alarms are created automatically. When a host is removed, alarms are cleaned up.

ITSM Integration (ServiceNow)

Alarm events are routed through SQS to a Lambda that formats and forwards to ServiceNow:

EVS-specific event category (Cloud.PaaS.PCS-AWS-EVS) for proper routing
SQS queue receives SNS alarm notifications
Lambda formats events with severity, CI mapping, and remediation context
Events forwarded to ServiceNow for automatic incident creation

Billing and Reporting

A scheduled Lambda collects EVS environment details across accounts:

Queries EVS API for host counts and configuration per account
Stores data in DynamoDB (partitioned by AccountID + NodeName)
Exports to S3 for reporting dashboards
Enables per-tenant billing based on actual host consumption

Key Takeaways

EVS needs /20 CIDR minimum — plan IP address space carefully, especially in multi-environment setups
Route Server + BGP is mandatory — EVS NSX overlay routing requires dynamic BGP peering with a Route Server in the VPC
Use a jumphost for VMware API automation — vCenter, SDDC Manager, and NSX APIs are VPC-internal; SSM Run Command on an EC2 instance is the cleanest pattern
DynamoDB Streams for reactive monitoring — automatically provision/deprovision alarms as hosts come and go
Step Functions with DDB locking — prevents race conditions in multi-tenant environments where multiple deployments could target the same account

Summary

Amazon EVS brings VMware workloads into your VPC, but operationalizing it requires the same automation discipline as any other AWS service. The modular pattern described here — independent CloudFormation stacks orchestrated by Step Functions, with DynamoDB for state and SSM for configuration — scales from single environments to multi-account managed service platforms.

For teams adopting EVS, investing in Day-0 automation pays dividends immediately: consistent deployments, faster time-to-environment, and a foundation for Day-2 operations that doesn't depend on manual intervention.

Alpesh Kumbhare is a Cloud Architect at Atos, specializing in AWS infrastructure automation and VMware cloud solutions. Connect on LinkedIn.

Automating Dell PowerProtect Backup on AWS: A Terraform-Based Approach

Alpesh — Wed, 29 Jul 2026 10:20:07 +0000

Organizations running Dell PowerProtect backup on AWS face a recurring challenge: the multi-appliance ecosystem — PPDM (PowerProtect Data Manager), DDVE (Data Domain Virtual Edition), and DDMC (Data Domain Management Center) — requires careful, sequenced provisioning that is difficult to repeat consistently across regions and accounts.

This post presents an approach to automating Dell PowerProtect deployment on AWS using Terraform for infrastructure provisioning and AWS Step Functions for post-deployment orchestration. The patterns described here are applicable to any enterprise running Dell backup appliances on AWS and looking to move from manual or semi-automated deployments to a fully codified, repeatable framework.

Why Automate Dell Backup on AWS?

Dell PowerProtect components on AWS are typically deployed as EC2 instances from marketplace AMIs. While the deployment itself is straightforward, the challenge lies in what comes after:

DDMC requires an interactive first-login wizard (EULA acceptance, password change, filesystem setup)
PPDM needs REST API-driven configuration (licensing, NTP, credential setup, policy creation)
DDVE must be registered with both PPDM and DDMC via coordinated API calls
DNS resolution must include both forward and reverse records — PPDM validates bidirectional DNS before accepting integration partners
Credentials need secure storage and rotation from day one

When done manually, this process introduces configuration drift, creates undocumented tribal knowledge, and doesn't scale. Infrastructure as Code eliminates these risks.

Reference Architecture

A production-grade Dell PowerProtect automation framework on AWS operates across four layers:

┌─────────────────────────────────────────────────────────────────────┐
│                    Layer 1: Terraform - Infrastructure                │
│  VPC / Subnets / Security Groups / Route53 / IAM / KMS / Secrets    │
├─────────────────────────────────────────────────────────────────────┤
│                    Layer 2: Terraform - Appliance Deployment          │
│  PPDM EC2 Instance / DDVE EC2 Instance / DDMC EC2 Instance          │
│  EBS Volumes / Network Interfaces / Instance Profiles                │
├─────────────────────────────────────────────────────────────────────┤
│                    Layer 3: Step Functions - Configuration            │
│  Lambda: Configure DDMC (SSH)                                        │
│  Lambda: Configure PPDM (REST API)                                   │
│  Lambda: Integrate DDVE ↔ PPDM/DDMC (REST API)                      │
│  Lambda: Credential Rotation (Secrets Manager)                       │
├─────────────────────────────────────────────────────────────────────┤
│                    Layer 4: Operational Monitoring                    │
│  EventBridge → Monitor Lambda → DynamoDB → ITSM Integration         │
└─────────────────────────────────────────────────────────────────────┘

Each layer has a clear responsibility boundary, and changes in one layer don't cascade unpredictably into others.

Layer 1: Network and Security Foundation

VPC and Subnet Design

Dell backup appliances communicate on specific ports (2051, 2052, 3009 for Data Domain protocols, 443 for management APIs). Your network module should provision:

A dedicated VPC (or subnet range within an existing VPC) for backup appliances
Private subnets across multiple Availability Zones for resilience
Security groups with granular ingress/egress rules scoped to Dell protocol ports
NAT Gateway access for marketplace license validation and updates

DNS: The Hidden Requirement

A common deployment failure with PPDM is integration rejection due to DNS resolution issues. PPDM performs bidirectional DNS validation — it resolves the DDVE hostname to an IP, then reverse-resolves that IP back to a hostname, and expects them to match.

You must automate both:

Forward DNS (A records) in Route53 private hosted zones
Reverse DNS (PTR records) in reverse lookup zones (e.g., 10.in-addr.arpa)

Without PTR records, PPDM will fail with errors like "Network validation has failed" or "Unable to resolve FQDN/IP address." This is not well documented and catches many teams off guard.

# Terraform reverse DNS zone and PTR record
resource "aws_route53_zone" "reverse" {
  name = "10.168.192.in-addr.arpa"
  vpc {
    vpc_id = aws_vpc.backup.id
  }
}

resource "aws_route53_record" "ppdm_ptr" {
  zone_id = aws_route53_zone.reverse.zone_id
  name    = "15.10.168.192.in-addr.arpa"
  type    = "PTR"
  ttl     = 300
  records = ["ppdm.backup.internal"]
}

IAM and Secrets Management

Each appliance role should follow least privilege:

PPDM instance role: access to Secrets Manager (own credentials), S3 (backup targets), SSM (for agent deployment)
DDVE instance role: KMS decrypt for encrypted volumes, Secrets Manager for rotation
DDMC instance role: SSM for parameter retrieval, EC2 DescribeInstances for discovery

Store initial and rotated credentials in AWS Secrets Manager from the start. Avoid embedding credentials in EC2 user data or relying on instance IDs as default passwords.

Layer 2: Appliance Deployment with Terraform

Each Dell appliance is an EC2 instance provisioned from marketplace AMIs. Key considerations:

PPDM Deployment

Instance type: m5.xlarge or larger (4 vCPU, 16 GB RAM minimum)
Root volume: 100 GB gp3
Additional data volumes based on catalog size
Elastic IP or stable private IP for DNS consistency

DDVE Deployment

Instance type: Storage-optimized (m5.2xlarge or r5.xlarge depending on capacity tier)
Multiple EBS volumes for metadata and data (DDVE uses specific volume layouts)
High IOPS configuration for deduplication workloads
Consider io2 Block Express for large-scale deployments

DDMC Deployment

Instance type: m5.large (lighter requirements as it's a management plane)
Data volumes for its internal database
Network access to all DDVE instances it will manage

# Parameterized appliance deployment
module "ppdm" {
  source         = "./modules/dell-appliance"
  appliance_type = "ppdm"
  ami_id         = var.ppdm_ami_id
  instance_type  = var.ppdm_instance_type
  subnet_id      = module.network.private_subnet_ids[0]
  security_groups = [module.network.ppdm_sg_id]
  iam_profile    = module.iam.ppdm_instance_profile

  tags = {
    Application = "dell-backup"
    Component   = "ppdm"
    Environment = var.environment
  }
}

The key principle: Terraform handles the "what" (infrastructure state) but not the "how" (appliance-internal configuration). That's where Step Functions come in.

Layer 3: Post-Deployment Configuration with Step Functions

Why Not Terraform Provisioners?

Terraform provisioners (remote-exec, local-exec) are tempting but problematic for multi-step appliance configuration:

They run at terraform apply time and can't be retried independently
They have no built-in state machine for sequential steps with wait conditions
Failures leave Terraform state in an inconsistent position
They break the declarative model — infrastructure becomes coupled with runtime behavior

AWS Step Functions provide a better fit: visible state machines with retry, timeout, branching, and independent execution.

Configuration Lambda Functions

DDMC Configuration (SSH-based)

DDMC's first-login process is an interactive SSH shell wizard. A Lambda function using Paramiko (Python SSH library) automates this:

Connect to DDMC via SSH (initial default credentials from Secrets Manager)
Accept EULA prompts by pattern-matching terminal output
Change the sysadmin password to a generated value
Complete the setup wizard (timezone, hostname, network confirmation)
Create the backup filesystem with defined quotas
Store new credentials in Secrets Manager

Key challenge: The wizard prompt text varies by DDMC firmware version. Build defensive pattern matching with timeouts and fallback detection.

PPDM Configuration (REST API-based)

PPDM exposes a REST API on port 443 after boot, but it takes 3-8 minutes after EC2 "running" state before the API becomes responsive. Your Lambda needs:

Poll the PPDM API health endpoint with exponential backoff
Authenticate with default credentials
Accept the EULA via API
Change the admin password
Configure NTP settings
Activate the license
Create default protection policies

DDVE-PPDM-DDMC Integration

The integration Lambda orchestrates cross-appliance registration:

Authenticate to PPDM API
Add DDVE as a storage target in PPDM (requires DDVE credentials)
Authenticate to DDMC API
Register DDVE instance in DDMC
Validate integration via status checks
Run a test backup to confirm end-to-end connectivity

Step Function Definition

{
  "StartAt": "WaitForAppliances",
  "States": {
    "WaitForAppliances": {
      "Type": "Wait",
      "Seconds": 300,
      "Next": "ConfigureDDMC"
    },
    "ConfigureDDMC": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:...:configure-ddmc",
      "Retry": [
        {
          "ErrorEquals": ["States.ALL"],
          "IntervalSeconds": 30,
          "MaxAttempts": 3,
          "BackoffRate": 2.0
        }
      ],
      "Next": "ConfigurePPDM"
    },
    "ConfigurePPDM": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:...:configure-ppdm",
      "Retry": [
        {
          "ErrorEquals": ["States.ALL"],
          "IntervalSeconds": 60,
          "MaxAttempts": 5,
          "BackoffRate": 2.0
        }
      ],
      "Next": "IntegrateDDVE"
    },
    "IntegrateDDVE": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:...:integrate-ddve",
      "Retry": [
        {
          "ErrorEquals": ["States.ALL"],
          "IntervalSeconds": 30,
          "MaxAttempts": 3,
          "BackoffRate": 2.0
        }
      ],
      "End": true
    }
  }
}

Layer 4: Operational Monitoring and ITSM Integration

Once deployed, Dell backup appliances need proactive monitoring. A serverless monitoring pattern works well:

Monitoring Architecture

EventBridge Scheduled Rule — Triggers a monitor Lambda every 10 minutes
Monitor Lambda — Authenticates to DDMC REST API, retrieves DDVE health status and active alerts
DynamoDB Table — Stores current alert state for deduplication and tracking
DynamoDB Streams — Triggers ITSM Lambda on new or changed alerts
ITSM Lambda — Formats events and forwards to your incident management system (ServiceNow, Jira Service Management, etc.)

Alert Event Schema

Standardize your event format for downstream consumption:

{
  "event_id": "unique-uuid",
  "source": "dell-backup-monitor",
  "severity": "critical",
  "appliance": "ddve-01.backup.internal",
  "alert_name": "Appliance Not Responding",
  "timestamp": "2026-07-15T10:30:00Z",
  "region": "eu-west-1",
  "action_required": "Investigate DDVE connectivity"
}

Event Archival

Archive all events to S3 with structured paths for audit and compliance:

s3://backup-events/{service}/{account_id}/{region}/{date}/{event_id}-{severity}.json

EC2 Backup Onboarding at Scale

The final automation piece is onboarding EC2 instances to Dell Backup protection:

Tag-based targeting — Instances tagged with Backup: enabled are discovered
SSM Automation — AWS Systems Manager installs the PPDM agent without SSH access
Agent registration — The agent registers with PPDM using pre-configured server endpoints
Policy assignment — Protection policies are assigned based on workload tags (e.g., BackupPolicy: daily-30day-retention)
Health validation — Post-installation check confirms agent connectivity to PPDM

Keep backup onboarding as a standalone automation — don't couple it with general EC2 lifecycle management. Backup policies change independently of OS patching or scaling events.

Common Pitfalls and How to Avoid Them

Pitfall	Impact	Solution
Missing PTR records	PPDM integration fails silently	Automate reverse DNS in your network module
Default passwords left in place	Security vulnerability	Rotate on first boot, store in Secrets Manager
Using Terraform provisioners for config	Brittle, non-retryable deployments	Use Step Functions + Lambda instead
No health checks between steps	Integration attempts on unready appliances	Implement polling with backoff in each Lambda
Coupling backup to OS management	Change in one breaks the other	Separate Terraform modules and pipelines
No monitoring of the monitoring	Silent alerting failures	CloudWatch Alarms on Lambda error metrics

Summary

Automating Dell PowerProtect on AWS requires thinking in layers:

Terraform for infrastructure — VPC, EC2, IAM, DNS (including reverse), KMS, Secrets Manager
Step Functions for orchestration — Sequential, retryable configuration of appliances via Lambda
EventBridge + Lambda for monitoring — Proactive health checks with ITSM integration
SSM for onboarding — Agent deployment at scale without SSH access

The pattern separates concerns cleanly: Terraform owns state, Step Functions own workflow, Lambda owns appliance-specific logic. This separation makes each layer independently testable, updatable, and debuggable.

For teams deploying Dell backup on AWS manually today, adopting this layered automation approach eliminates configuration drift, improves security posture, and scales across regions without additional engineering effort per deployment.

Alpesh Kumbhare is a Cloud Architect at Atos business, specializing in AWS infrastructure automation and enterprise data protection solutions. Connect on LinkedIn.