Alpesh

Posted on Jul 29

Automating Amazon EVS (Elastic VMware Service) Day-0 to Day-2 Operations with CloudFormation and Step Functions

#aws #vmw #cloudformation #devops

Amazon Elastic VMware Service (EVS) reached general availability in August 2025, enabling customers to run VMware Cloud Foundation (VCF) directly within their Amazon VPC. While EVS simplifies the infrastructure layer, operationalizing it at scale — provisioning environments, configuring vCenter, setting up NSX segments, attaching storage, and integrating with ITSM — still requires significant automation.

This post presents a pattern for automating EVS from Day-0 (environment creation) through Day-2 (monitoring, alerting, billing) using AWS CloudFormation and Step Functions. The approach is modular, allowing teams to adopt individual components independently.

The Challenge with EVS at Scale

Standing up a single EVS environment involves:

Creating a dedicated VPC with correct subnet sizing (/20 minimum)
Deploying the EVS environment with Route Server and BGP peering
Configuring vCenter and SDDC Manager (SSL certificates, depot credentials, domain settings)
Creating NSX-T segments for workload networking
Optionally deploying FSx for NetApp ONTAP as shared storage
Setting up CloudWatch alarms for bare-metal host health
Integrating with ITSM (ServiceNow) for incident management
Tracking host counts for billing and reporting

When managing EVS across multiple accounts and regions, this must be repeatable, auditable, and hands-off.

Architecture: Modular CloudFormation with Step Function Orchestration

The solution uses a modular approach — each operational concern is a separate CloudFormation stack, orchestrated by a central Step Function:

                    ┌─────────────────────────┐
                    │  Step Function Orchestrator  │
                    └──────────┬──────────────┘
                               │
        ┌──────────────────────┼──────────────────────┐
        │                      │                      │
        ▼                      ▼                      ▼
┌───────────────┐    ┌──────────────────┐    ┌──────────────────┐
│  Pre-Setup    │    │ Network Creation │    │  EVS Environment │
│  (DDB, IAM)  │    │ (VPC, Subnets)   │    │  (Route Server,  │
│               │    │                  │    │   BGP, Deploy)   │
└───────────────┘    └──────────────────┘    └──────────────────┘
        │                      │                      │
        ▼                      ▼                      ▼
┌───────────────┐    ┌──────────────────┐    ┌──────────────────┐
│  Configure    │    │  NSX Segment     │    │  FSx ONTAP       │
│  vCenter/SDDC │    │  Creation        │    │  (Optional NFS)  │
└───────────────┘    └──────────────────┘    └──────────────────┘
        │                      │                      │
        ▼                      ▼                      ▼
┌───────────────┐    ┌──────────────────┐    ┌──────────────────┐
│  CW Alarms    │    │  ITSM / Alarm    │    │  Billing &       │
│  (Host Health)│    │  Event Handler   │    │  Reporting       │
└───────────────┘    └──────────────────┘    └──────────────────┘

Design Principles

One stack, one concern — Network, compute, storage, monitoring, and ITSM are independent stacks
Step Function as orchestrator — Handles sequencing, retries, and cross-account coordination
DynamoDB for state — Tracks deployment status, lock management, and host inventory
SSM Parameter Store as configuration registry — All naming conventions and cross-stack references flow through SSM

Day-0: Environment Provisioning

Network Foundation

EVS requires a /20 CIDR minimum for the management VPC. The network stack provisions:

Dedicated VPC with DNS support enabled
Private subnets in the target AZ (EVS is single-AZ per environment)
Route tables prepared for BGP route injection from Route Server

# Key design decision: /20 minimum for EVS VPC
Parameters:
  CidrBlock:
    Type: String
    Default: 10.23.0.0/20
    Description: "CIDR block for EVS VPC (minimum /20)"

Route Server and EVS Deployment

The EVS environment stack handles:

Route Server creation with a private ASN (64512-65534 range)
BGP peer configuration for NSX Edge connectivity (EVS uses BGP for overlay-to-underlay routing)
EVS environment deployment with i4i.metal instances, VMware license keys, and VCF configuration
Route persistence — keeps routes active briefly after BGP session drops to avoid routing flaps

# Route Server enables dynamic routing between EVS NSX and AWS VPC
Parameters:
  RouteServerASN:
    Type: Number
    Default: 65001
    Description: ASN for Route Server (private range)
  PeerASN:
    Type: Number
    Default: 65000
    Description: ASN for EVS NSX Edge nodes

Step Function Orchestration

The central Step Function coordinates the multi-step deployment:

Check DynamoDB — Validates the target account doesn't already have an EVS deployment (prevents duplicates)
Pre-setup — Creates DynamoDB state tables, IAM roles, and SSM parameters
Network deployment — Provisions VPC, subnets, and route tables
EVS environment creation — Deploys Route Server and EVS cluster
Post-install tasks — Configures vCenter, creates NSX segments, deploys optional storage

The Step Function uses a DynamoDB-based locking mechanism to prevent concurrent deployments to the same account — critical in multi-tenant managed service environments.

Day-1: Configuration Automation

vCenter and SDDC Manager Configuration

After EVS deploys the VCF stack, vCenter and SDDC Manager need configuration:

SSL certificate rotation (replacing self-signed certs with organization certificates)
VMware Depot credentials for patching and updates
Domain and DNS configuration
NFS datastore mounting (if FSx ONTAP is deployed)

This automation uses an EC2 jumphost (Windows Server) with SSM Run Command to execute PowerShell/PowerCLI scripts against vCenter and SDDC Manager APIs — necessary because these APIs are only accessible from within the VPC.

NSX-T Segment Creation

Workload VMs need network segments. The NSX automation:

Retrieves NSX Manager credentials from Secrets Manager (stored during EVS deployment)
Connects to the NSX Manager REST API via the EC2 jumphost
Creates DHCP server profiles for workload subnets
Provisions overlay segments with gateway and DHCP configuration

# NSX segment creation via REST API (executed on jumphost via SSM)
$nsxManager = "evs01-nsx01.customer.domain"
$headers = @{ Authorization = "Basic $encodedCreds" }
# Create DHCP profile, then segment with gateway

FSx for NetApp ONTAP (Optional NFS Datastore)

For environments needing shared storage beyond vSAN:

FSx ONTAP file system deployed in the EVS VPC subnet
Storage Virtual Machine (SVM) created with NFS protocol
Volume provisioned with specified capacity
Security groups configured for NFS traffic from ESXi hosts

# FSx ONTAP supports multiple deployment types
Parameters:
  DeploymentType:
    Type: String
    AllowedValues:
      - MULTI_AZ_2   # Recommended: standby in second AZ
      - SINGLE_AZ_2  # High performance: 6+ GBps
      - MULTI_AZ_1
      - SINGLE_AZ_1

Day-2: Monitoring, Alerting, and Billing

CloudWatch Alarms for EVS Hosts

EVS runs on bare-metal i4i.metal instances. The monitoring stack:

Maintains a DynamoDB table of active EVS host instance IDs
Uses DynamoDB Streams to detect when hosts are added or removed
Automatically creates CloudWatch alarms for each host (CPU, status checks, network)
All alarms publish to a central SNS topic

This is event-driven — when a new host is added to the DynamoDB table, alarms are created automatically. When a host is removed, alarms are cleaned up.

ITSM Integration (ServiceNow)

Alarm events are routed through SQS to a Lambda that formats and forwards to ServiceNow:

EVS-specific event category (Cloud.PaaS.PCS-AWS-EVS) for proper routing
SQS queue receives SNS alarm notifications
Lambda formats events with severity, CI mapping, and remediation context
Events forwarded to ServiceNow for automatic incident creation

Billing and Reporting

A scheduled Lambda collects EVS environment details across accounts:

Queries EVS API for host counts and configuration per account
Stores data in DynamoDB (partitioned by AccountID + NodeName)
Exports to S3 for reporting dashboards
Enables per-tenant billing based on actual host consumption

Key Takeaways

EVS needs /20 CIDR minimum — plan IP address space carefully, especially in multi-environment setups
Route Server + BGP is mandatory — EVS NSX overlay routing requires dynamic BGP peering with a Route Server in the VPC
Use a jumphost for VMware API automation — vCenter, SDDC Manager, and NSX APIs are VPC-internal; SSM Run Command on an EC2 instance is the cleanest pattern
DynamoDB Streams for reactive monitoring — automatically provision/deprovision alarms as hosts come and go
Step Functions with DDB locking — prevents race conditions in multi-tenant environments where multiple deployments could target the same account

Summary

Amazon EVS brings VMware workloads into your VPC, but operationalizing it requires the same automation discipline as any other AWS service. The modular pattern described here — independent CloudFormation stacks orchestrated by Step Functions, with DynamoDB for state and SSM for configuration — scales from single environments to multi-account managed service platforms.

For teams adopting EVS, investing in Day-0 automation pays dividends immediately: consistent deployments, faster time-to-environment, and a foundation for Day-2 operations that doesn't depend on manual intervention.

Alpesh Kumbhare is a Cloud Architect at Atos, specializing in AWS infrastructure automation and VMware cloud solutions. Connect on LinkedIn.

DEV Community