DEV Community: Marcin Wyszynski

How to Quickly Improve Your Infrastructure as Code

Marcin Wyszynski — Wed, 17 Nov 2021 22:40:00 +0000

The introduction of Infrastructure as Code or IaC has transformed the way you can provision and deploy high-performance cloud-based IT infrastructures.

IaC tools, such as Terraform, have been integrated into DevOps toolchains, saving DevOps IaC teams from the excessive manual effort.

While these tools undoubtedly help accelerate building IT infrastructures, their limitations can impact DevOps’ ability to optimize and improve control of their IaC processes supporting future business needs.

In this article, you’ll learn about IT infrastructure limitations that IaC DevOps teams deal with on a daily basis and how Spacelift is able to get past them.

1) Workflow

One of the most frequent challenges while using more generic IaC tools is the non-intuitive workflow driven by its reliance on pull requests. Some solutions offer multiple workspaces, but the result can be fragile and nondeterministic.

Since there’s no concept of mapping projects to branches or tags, anyone commenting on an approved pull request can deploy arbitrary code to production, even if the approval was meant for a short-lived experimental environment.

Spacelift does not depend on pull requests. It is mostly driven by push and tag events, so building a sophisticated Git flow is much easier. Spacelift reports the outcome of its jobs as commit status checks, allowing you to block merging the code on a failing Spacelift check.

Triggering a run can be customized using Git push policies. Thanks to that, Spacelift can provide the same level of comfort and security to teams using one project per repository and those using mono repo with hundreds of interdependent projects. You can read more about our approach to VCS integration here.

2) Access Control

The majority of generic tools don’t offer access control models but rely on comments on pull requests to drive infrastructure deployments. While it is usually fine when a single repository drives a single Terraform project, it becomes a huge liability for more complex scenarios.

Spacelift ships with a sophisticated mechanism allowing administrators to declare who can log in (and under what circumstances) and what their level of access to each of the managed projects should be. Even our Slack integration can be subject to policy controls, allowing an admin to grant access to a project based on Slack-specific data (think team, channel, user, etc.).

3) Policy Framework

One thing that’s not in scope for most IaC solutions is the way to ensure that your infrastructure is compliant with industry best practices and your company policies.

Spacelift puts policy-as-code in the center of its value proposition and builds a consistent, robust policy framework on top of Open Policy Agent. Apart from providing a comprehensive automated change review and ensuring compliance of your Terraform changes, Spacelift uses the same approach to allow you to declare rules around the account and project access, handling push notifications, starting runs and triggering tasks, and creating relationships between projects.

4) Complex Workflows

Handling interdependencies between projects has always been Terraform’s Achilles’ heel. The usual approach to this problem is adding another layer of abstraction in the form of a Terraform wrapper like Terragrunt. But it’s only a partial solution as it breaks the problem into smaller chunks.

Spacelift’s trigger policies on the other hand provide a smart, declarative automation layer on top of vanilla Terraform. They allow you to plug into state changes of individual projects and declare dependencies that should be resolved, following the changes that have just been applied. Read more here to discover other exciting possibilities.

5) Private Module Registry

Another problem to solve externally when using some of the generic tools is authoring and maintaining reusable Terraform modules for your organization. Terraform is flexible in allowing modules to come from various sources, but ensuring confidential access, as well as testing and versioning, are left to you, the user.

Until now, the golden standard in that regard has been the private module registry from HashiCorp. But Spacelift offers much more. Far from being just a glorified package manager, Spacelift adds a full CI solution for Terraform modules, out of the box and free of charge. You can thus ensure that your private modules are healthy before you distribute them to the rest of your organization.

6) Effortless Setup and Customization

If you manage a single or a handful of rarely changing projects, it’s likely that you just set your IaC up once and forget about it. But in a more dynamic environment, where microservices come and go, new environments proliferate and new product teams require their own Terraform workspaces. The need to configure it each and every time become a major nuisance, putting a lot of pressure on your DevOps team.

Enter Spacelift. In Spacelift, much of the configuration can be handled by the project owners themselves—you can add Terraform and/or environment variables and mount files (even inject Terraform code!) programmatically or through the GUI without the need for administrative privileges or changing the central server configuration. For administrators, adding new projects requires minimal hassle since there’s no need to set up webhooks or change any YAML. And it can all be done programmatically using Terraform.

7) Programmatic Configuration

What comes as a pleasant surprise to users of generic CI tools, Spacelift entities such as stacks, contexts, modules or policies can be managed in a declarative way using your favorite infra-as-code tool (this rule applies also to their configuration). Yes, that’s right—Spacelift offers a Terraform provider that allows you to programmatically manage the lifecycle of its own resources.

Administrative stacks get credential-less access to the subset of our GraphQL API that does not involve managing the actual infrastructure. For more sophisticated use cases, Spacelift allows you to generate API keys that are subject to the same access controls as normal users are, allowing you to create single-purpose tokens for restricted use by your internal scripts.

8) Drift Detection

Generic IaC platforms do not provide any mechanisms to detect if your infrastructure is undergoing drift. Drift is a condition that represents the difference between the desired and the actual state of the infrastructure managed by your tool of choice – Terraform, Pulumi, CloudFormation or another. Drift can be caused by either or a combination of changes directly introduced by external actors – either humans or machines (scripts) or via the dependency of your resources on external data sources. In any case, drift is not good.

Spacelift has got you covered here. You can configure periodic drift detection to notify you whenever drift happens, and take immediate action. You can even go a step further with optional automatic reconciliation, ensuring your infrastructure always resembles your Terraform configuration.

9) Resource Visualization

General-purpose CI/CD platforms provide little to no insight into resource utilization from either a real-time or historical perspective. Which resources are over-or underutilized?

Developers need to be able to intimately understand the material they’re working with. With regards to infra-as-code, the most important part of this story is understanding the managed resources in-depth. Both from the current perspective and through being able to put each resource in its historical context.

The resources view in Spacelift gives you greater visibility into each and every resource. With this deep insight into resources, DevOps are able to gain an understanding of the lifecycle of each resource managed by Spacelift and document it, regardless of the technology used — Terraform, Terragrunt, Pulumi, or CloudFormation.

Creature Comforts

Last but not least, Spacelift puts an emphasis on great user experience, offering a myriad of creature comforts. Contexts for example allow you to attach entire collections of configuration to individual stacks and modules. Tasks provide a powerful audited way of running one-off administrative commands on an initialized Terraform environment – subject to their own policy constraints. Stack locking allows a single individual to take exclusive control over a stack to ensure that nobody is able to modify its state while crucial changes are being made.

Why Spacelift

Spacelift is an innovative and sophisticated SaaS product for Infrastructure as Code which helps IaC DevOps develop and deploy new infrastructures or changes quickly and with confidence.

Spacelift offers a unique set of IaC management capabilities:

a more intuitive, versatile and robust workflow
extremely granular access controls on account and project level that work well with an identity provider of your choice (SSO);
an automated code review and threat detection
the ability to declare complex workflows between projects across multiple repositories
a built-in private module registry with a full CI system for modules
effortless setup and customization with per-project environment management and Docker integration
programmatic configuration using Terraform;
drift detection
resource visualization

… and a myriad of creature comforts like contexts, tasks or stack locking.

Key Points

There are many ways of working with Terraform. Each way is different in terms of complexity and offers a different set of features. It is important to keep in mind that choosing one way or another should be based on business and technical requirements. Most often, there is no point in implementing an in-house solution as the cost and effort of building and maintaining it may exceed its potential benefits. It is much easier and more efficient to leverage platforms such as Spacelift to provide these features for you instead. You can try Spacelift for free or book a demo with one of our engineers.

Alternative to Atlantis

Marcin Wyszynski — Sun, 24 Oct 2021 23:13:32 +0000

Atlantis is an old and battle-tested self-hosted tool that provides a layer of automation and collaboration for Terraform. Lacking a user interface other than server flags and YAML configuration files, Atlantis uses your VCS as one, listening for pull request notifications and posting feedback as comments.

Using Atlantis has two major benefits over running Terraform in a “generic” CI system. First, it strikes the right balance between specialized and flexible: custom workflows give you a level of control that’s not possible within a tool like Terraform Cloud. Second, it runs on your infrastructure, which allows it to access resources inside a VPC or use cloud-specific dynamic credentials.

Spacelift embraces all those characteristics of Atlantis but offers them in as a modern and sophisticated SaaS product that provides:

a more intuitive, versatile and robust workflow;
extremely granular access controls on account and project level that work well with an identity provider of your choice (SSO);
an automated code review and threat detection using a consistent open-source policy-as-code framework;
the ability to declare complex workflows between projects across multiple repositories;
a built-in private module registry with a full CI system for modules;
effortless setup and customization with per-project environment management and Docker integration;
programmatic configuration using Terraform;
drift detection;
… and a myriad of creature comforts like contexts, tasks or stack locking;

Let’s talk through each of these features one by one.

1) Workflow

One of most frequent challenges while using Atlantis in practice is a non-intuitive workflow driven by its reliance on pull requests. Not only is it a lot of work to set up multiple workspaces pointing to the same Git repository but the result is fragile and nondeterministic. Since there’s no concept of mapping projects to branches or tags, anyone that can comment on an approved pull request can deploy arbitrary code to production, even if the approval was meant for a short-lived experimental environment.

Spacelift does not depend on pull requests. Sure, it will post beautiful comments there for your convenience, but it’s mostly driven by push and tag events, so it’s easy to build a sophisticated Git flow. Spacelift will report the outcome of its jobs as commit status checks, which then allows you to block merging the code on a failing Spacelift check. There are two main types of runs – proposed and tracked. By default, each project points at a particular branch (and optionally also a path) – pushing to that branch will trigger a tracked run – one that can be applied. This behavior can be customized using Git push policies. Thanks to that, Spacelift can provide the same level of comfort and security to teams using one project per repository and those using monorepos with hundreds of interdependent projects. You can read more about our approach to VCS integration here.

2) Access Control

Another pain point when using Atlantis at scale is its access control model. By relying on comments on pull requests to drive infrastructure deployments, Atlantis effectively yields control over your infrastructure to the VCS provider. While this is usually fine when a single repository drives a single Terraform project, this becomes a huge liability for more complex scenarios.

Spacelift ships with a sophisticated mechanism allowing administrators to declare who can log in (and under what circumstances) and what should be their level of access to each of the managed projects. Since it’s 2020, Spacelift ships with SSO out of the box, supporting every identity provider that speaks SAML 2.0 protocol. Even our Slack integration can be subject to policy controls, allowing an admin to grant access to a project based on Slack-specific data (think team, channel, user etc.). For extra protection, powerful features like Tasks have their own independent policy controls.

3) Policy Framework

Both Spacelift and Atlantis let you use Open Policy Agent – an open-source solution endorsed by Cloud Native Computing Foundation – to ensure that your infrastructure is compliant with industry best practices and your company policies. Atlantis integrates with conftest to achieve that and lets you verify each Terraform plan using OPA policies.

Spacelift goes a few steps further though and puts policy-as-code in the center of its value proposition and builds a consistent, robust policy framework. Apart from providing a comprehensive automated change review and ensuring compliance of your Terraform changes, Spacelift uses the same approach to allow you to declare rules around account and project access, handling push notifications, starting runs and triggering tasks, and creating relationships between projects. It also provides the Policy Workbench, which lets you view past executions of your policy – including the inputs and decisions that have been made – and lets you interactively edit your policy, while simulating its execution on these previous inputs. You can use our starter repo to quickly provision a bunch of policies and get a feel for how you can work with them in Spacelift.

4) Complex Workflows

Handling interdependencies between projects has always been Terraform’s Achilles’ heel and Atlantis is no different in that regard. The usual approach to this problem is adding another layer of abstraction in the form of a Terraform wrapper like Terragrunt. While this breaks the problem into smaller chunks, it’s only a partial solution achieved at a cost of locking yourself into another dependency.

Spacelift’s trigger policies on the other hand provide a smart, declarative automation layer on top of vanilla Terraform. These allow you to plug into state changes of individual projects and declare dependencies that should be resolved following the changes that have just been applied. In fact, this is just a subset of all use cases covered by trigger policies. Read more to discover other exciting possibilities.

5) Private Module Registry

Another problem that one needs to solve externally when using Atlantis is authoring and maintaining reusable Terraform modules for your organization. Terraform is flexible in allowing modules to come from various sources but ensuring confidential access as well as testing and versioning are left to you, the user.

Until now, the golden standard in that regard has been the private module registry from HashiCorp. But Spacelift offers that, and much more. Far from being just a glorified package manager, Spacelift adds a full CI solution for Terraform modules, out of the box and free of charge. You can thus ensure that your private modules are healthy before you distribute them to the rest of your organization.

6) Effortless Setup and Customization

If you manage a single or a handful of rarely changing projects using Atlantis, it’s likely that you just set it up once and forget about it. But in a more dynamic environment, where microservices come and go, new environments proliferate and new product teams require their own Terraform workspaces, the need to configure Atlantis each and every time becomes a major nuisance putting a lot of pressure on your DevOps team.

Enter Spacelift. In Spacelift, much of the configuration can be handled by the project owners themselves – you can add Terraform and/or environment variables and mount files (even inject Terraform code!) programmatically or through the GUI without the need for administrative privileges or changing the central server configuration. For administrators, adding new projects requires minimal hassle since there’s no need to set up webhooks or change any YAML. And it can all be done programmatically, using Terraform.

7) Programmatic Configuration

What comes as a pleasant surprise to long-time Atlantis users, Spacelift entities like stacks, contexts, modules or policies as well as their configuration can be managed in a declarative way using your favorite infra-as-code tool. Yes, that’s right – Spacelift offers a Terraform provider that allows you to manage the lifecycle of its own resources programmatically.

8) Drift Detection

Infrastructure drift happens. Either somebody will make manual changes while debugging and then forget about them, or you might have data sources whose outputs change periodically. Normally, you would notice these changes only after you execute your next plan or apply.

Spacelift has got you covered here. You can configure periodic drift detection to be notified whenever drift happens and take immediate action. You can go a step further too with optional automatic reconciliation, ensuring your infrastructure always resembles your Terraform configuration.

DEV Community: Marcin Wyszynski

How to Quickly Improve Your Infrastructure as Code

1) Workflow

2) Access Control

3) Policy Framework

4) Complex Workflows

5) Private Module Registry

6) Effortless Setup and Customization

7) Programmatic Configuration

8) Drift Detection

9) Resource Visualization

Creature Comforts

Why Spacelift

Key Points

Alternative to Atlantis

1) Workflow

2) Access Control

3) Policy Framework

4) Complex Workflows

5) Private Module Registry

6) Effortless Setup and Customization

7) Programmatic Configuration

8) Drift Detection

9) Creature Comforts