env0 Workflows: Simplifying Advanced IaC Setups and Managing Dependencies

As infrastructure scales, managing its increasing complexity and the dependencies between different components becomes more challenging. This is where the env0 Workflows feature becomes essential, offering a solution to address today's complexities in Infrastructure as Code (IaC) workloads.

In this write-up, we’ll discuss the capabilities, use cases, and features of Workflows in detail.

What are env0 Workflows?

env0 Workflows provide a structured approach to managing groups of related environments by defining their dependencies and orchestrating the entire process. This ensures that all dependencies are satisfied and the workflow runs smoothly from start to finish.

Workflows are particularly useful for managing systems divided into multiple sub-systems, allowing you to oversee the entire system as a cohesive unit while retaining control over individual stacks. They simplify the process of breaking down IaC stacks into smaller, more manageable parts.

With Workflows, you can describe complex deployment relationships and initiate partial or full runs to create and update sub-environments. 

This ensures proper sequencing of resource deployment and allows changes to any sub-environment to auto-update dependent components.

Workflows also help manage environment variables and sensitive credentials across different stacks, ensuring consistency and security.

Whether your IaC is monolithic or split into microservice-based stacks, Workflows are a massive step forward in describing and managing complex deployments.

Disclaimer
Workflows is an Enterprise plan feature. Visit here to learn more about env0 pricing and plans.

How Workflows Work

A workflow is based on a workflow template along with the IaC stack templates that you intend to deploy in your workflow environment.

The Workflow YAML file, env0.workflow.yml, is used to define the hierarchy of the dependencies and configuration of IaC sub-environments within a workflow declaratively.

To better understand how Workflows work, let us take an example.

Below is the definition of the env0.workflow.yml consisting of two environments that deploy VPC, and an EC2 instance that depends on that VPC:

environments:
  vpc:
    name: 'Virtual Network'
    templateName: 'VPC'
  vm:
    name: 'VM'
    templateName: 'EC2'
    needs:
      - vpc

This method works when we have the respective IaC templates created in place for Virtual Network and VM stacks. 

It is also important to note that each Workflow sub-environment can also reference the directories for individual IaC components, not only from different repositories but even from different VCS providers.

Let’s take the same example above to make it clearer. 

Here we are referencing the VPC directory from our GitHub VCS repo and the EC2 directory from our Gitlab VCS repo to define our sub-environments in the Workflow YAML:

environments:
  vpc:
    name: 'Virtual Network'
    vcs:
      type: 'terraform'
      terraformVersion: '1.5.7'
      repository: 'https://github.com/adityamurali155/env0-workflow-example'
      path: 'vpc'
  vm:
    name: 'VM'
    vcs:
      type: 'terraform'
      terraformVersion: '1.5.7'
      repository: 'https://gitlab.com/adityamurali155/env0-workflow-example'
      path: 'ec2'
    needs:
      - vpc

Notice the needs block makes one environment dependent on another (EC2 depends on VPC, in this case). You can add more parameters as you see fit to define your custom Workflow YAML file.

Following on the template creation approach, with our VPC and EC2 templates now in place, we are set to develop our workflow template, which references the env0.workflow.yml file located in the root directory of our repository.

The next step is to deploy a workflow environment using our workflow template. This workflow environment shows us a vivid dependency graph that illustrates the relationship between our Virtual Network and VM sub-environments.

In this example, no variables or information are shared between the sub-environments. This presents an opportunity to leverage the Workflows' core capability of Environment Outputs, which allows one IaC sub-environment's output to serve as another's input.

For instance, you can output the VPC subnet ID from the Virtual Network environment and use it as input for the VM environment, ensuring the EC2 instance is deployed into the correct VPC.

This method facilitates the secure transfer of sensitive credentials or data between sub-environments dynamically via the env0 platform abd eliminates the need for local storage of sensitive information or hard-coded variables in your IaC -enhancing the security of your sub-environments.

We’ll go into the detailed mechanics of Workflows with Environment Outputs in a moment.

Advantages of env0 Workflows

The capabilities of Workflows prove to be invaluable, particularly in scenarios like those discussed below:

• Dependency Management: As emphasized earlier, Workflows excels at managing dependencies between different environments. The Workflow YAML file allows you to declaratively define all the IaC stacks you plan to deploy and specifies any dependencies between them. Additionally, Workflows visually represent these dependencies, making it easier to understand and manage complex infrastructure setups. This visual aid simplifies troubleshooting and planning, enhancing overall deployment efficiency.

• Targeted Deploy/Destroy: With env0 Workflows, you can deploy or destroy specific parts of your infrastructure without affecting the entire workflow. This partial or selective deployment feature is handy in scenarios where you might only need to update or roll back certain IaC components. You could also destroy a specific IaC environment without affecting others, reducing the risk of unintended changes to other parts of your infrastructure.

• Multiple IaC Tools Support: Workflows enable teams to use different IaC tools within the same workflow. For instance, one team can use Terraform to provision infrastructure, while another team uses CloudFormation. Both teams can still share environment variables or sensitive credentials through Environment Outputs for their infrastructure, ensuring seamless collaboration and integration.

• Custom integration: Workflows can seamlessly integrate with other env0 features, such as custom flows, drift detection, and approval policies. For example, you can configure custom flows to run hooks (like installing a web server) after deploying an EC2 IaC stack in a workflow. 

You can also configure the workflow to run policies using an OPA plugin after every environment is deployed to ensure meeting security and compliance requirements.

Practical Example

Now, let us delve into a practical working example of how Workflows operate.

Here are three environments where we need to deploy a Virtual Network, an AKS cluster that depends on the virtual network, and a Helm chart that installs FluxCD, which depends on the AKS cluster.

The respective env0.workflow.yml looks like this:

environments:
  vnet: 
    name: 'Virtual Network'
    templateName: 'Virtual Network'
  aks:
    name: 'AKS Cluster'
    templateName: 'AKS Cluster'
    needs:
      - vnet
  flux:
    name: 'Flux Installation'
    templateName: 'FluxCD'
    needs:
      - aks

First, make sure that you have created your IaC stack templates: 

Create your workflow template and select Run Now to create a workflow environment.

Make sure that your environment name exactly matches that of the name defined in the Workflow YAML file.

‍

We have defined the subnet ID as an output value to be used by the AKS cluster in the VNet config, and the subnet ID as an input value in the AKS config.

#vnet config
output “aks_subnet_id” {
  value       = azurerm_subnet.az-subnet.id
  description = "Output the subnet ID to AKS cluster"
}

# aks config
variable "vnet_subnet_id" {
  type        = string
  description = "Take the subnet ID as input."
}

Before deploying the VNet environment, we’ll switch to the AKS environment and set the input value for the AKS environment (vnet_subnet_id) as the environment output value of the VNet environment (aks_subnet_id): 

Similarly, the Flux environment expects input values from the AKS environment, and the outputs are defined in the AKS config.

#aks config
output "host" {
  value     = module.cluster.cluster_fqdn
  sensitive = true
}
output "client_certificate" {
  value     = module.cluster.client_certificate
  sensitive = true
}
output "client_key" {
  value     = module.cluster.client_key
  sensitive = true
}
output "cluster_ca_certificate" {
  value     = module.cluster.cluster_ca_certificate
  sensitive = true
}

#flux config
variable "aks_cluster_fqdn" {
  type        = string
}
variable "aks_client_certificate" {
  type        = string
  sensitive   = true
}
variable "aks_client_key" {
  type        = string
  sensitive   = true
}
variable "aks_cluster_ca_certificate" {
  type        = string
  sensitive   = true
}

We’ll switch to the Flux environment and set our input variables. 

Here, we are able to pass sensitive Kubernetes connection credentials as environment output to Helm for configuring and installing flux in our AKS cluster.

Note: Environment Outputs cannot be marked as sensitive in UI on the fly. Therefore, it is necessary to make the sensitive variables marked as “sensitive=true” in your IaC config itself.

Lastly, after configuring all the values, we switch to the VNet environment (as it should deploy first) and run the workflow:

After a while, we can see our entire workflow functioning properly:

Less is More: Partial Workflow Deployment 

Often, we might forget to set up values correctly between different environments. Redeploying all environments to fix these issues is not ideal and can be time-consuming.

With Partial Workflow Deployment, you can redeploy or destroy a single sub-environment without affecting others.

Let's consider the previous example and take a scenario where we forgot to configure the environment output for the AKS environment. 

In this case, the workflow would fail, as it relies on these outputs to function correctly.

To solve this issue, navigate to the AKS environment, configure the environment output, then click on the three dots in the AKS cluster block and select the Run from here option.

We will select the AKS cluster and re-run the environment. After this, we can confirm that our workflow is operating correctly again.

Additional Use Cases

There are more possibilities for utilizing Workflows than what we have explored so far. Here are key use cases to consider:

1. Multi-tool Setup

As infrastructure grows more complex with the business expansion, different teams may handle multiple environments. 

For example, one team can manage the DB stack and its dependencies (Billing and Configuration Services), while another team handles the EKS stack and its dependencies (Configuration and Notification Services). 

Over time, as organizations scale, IaC deployments can grow even more intricate than the example we saw above. However, by using Workflows to define your IaC resources in a declarative approach, you can efficiently manage and track your entire IaC stack. Due to a clear presentation of their IaC stack statuses, the teams can collaborate and ensure smoother operations. 

2. Bulk Operations

Bulk operations refer to performing the same operation across multiple environments or resources simultaneously. 

In this example, the EC2 and S3 environments depend on an IAM role, both configured with a Deny effect for any operation on SNS topics. Later, the team decides to change the IAM role policy to Allow operations on the SNS topics, enabling the necessary permissions for EC2 and S3 environments to interact with those topics.

Utilizing Workflows simplifies making changes to one environment, automatically applying them to all dependent environments. 

This streamlines tasks such as updating software versions, applying security patches, and modifying configurations across multiple IaC environments, thereby reducing the time these tasks take.

Wrap Up

In conclusion, env0 Workflows stand out as a powerful and efficient way to manage complex infrastructure setups and dependencies in an Infrastructure-as-Code (IaC) environment.

Interested to learn more? Schedule a technical demo to see env0 in action!