nxs-marketplace-terraform: love and use it

Hello everyone! My name is Danil, I am a DevOps engineer at Nixys.

In today's business environment, companies increasingly need to deploy and manage various cloud environments quickly. Often customers set a task to deploy typical cloud environments in a short period once we were approached with such a request.

The customer was tasked with deploying several generic environments in the cloud as quickly as possible for their new project. They needed a solution that would ensure consistency, repeatability, and automation of the deployment process. Since deadlines were tight, they needed an approach that would minimize manual work and the potential for errors.

In these situations, traditional methods may not be effective enough, and that's when Infrastructure as Code (IaC) tools like Terraform come to the rescue.

How we came to write our own Terraform modules

At first, we looked at various existing solutions, including already-done modules and templates from public repositories. However, faced with limitations in functionality and flexibility, we decided to develop the modules we needed ourselves.

As a result, we ended up with a set of our own Terraform modules. They not only met the project requirements but also significantly simplified the deployment and infrastructure management process in the future. Thanks to this solution, the client was able to save time and resources, as well as ensure high reliability and predictability of results.

Please welcome - nxs-marketplace-terraform.

What approaches do we use and why

S3 storage as tfstate storage

When we work with Terraform to manage infrastructure, it is critical to store the state (tfstate) of that infrastructure. It is contained in a state file (terraform.tfstate) that Terraform uses to track changes and manage resources.

Terraform's state storage (tfstate) comes in several forms:

• Local storage: the state file is stored on the local machine. This is an easy way for small projects and development, but it’s unsuitable for teamwork because of the risk of data loss.
• Remote storage: used for centralized state management, which is especially important for teams. Various backends such as S3, Consul, and Terraform Cloud are supported.
• S3: cloud storage is often used for tfstate storage because of its reliability and the ability to customize locks using DynamoDB easily, preventing state conflicts.

These types of storage provide the flexibility to manage your infrastructure, allowing you to choose the appropriate method based on project and team needs.

If the state file is stored locally, this creates several problems:

• Synchronization issues: in multi-user environments where multiple engineers work with the same infrastructure, locally stored state can lead to inconsistencies and conflicts.
• Risk of data loss: a locally stored state file can be accidentally deleted or corrupted, resulting in the loss of critical infrastructure information.
• Limited accessibility: The local state file is not available to all team members, especially if they are working remotely or on different machines.

To solve these problems, a Remote State is used. It provides centralized, accessible, and secure storage of the state file for all team members.

Terraform provides a Remote Backend mechanism that allows you to store state in remote storage systems. Remote Backend solves the following problems:

• Centralized storage. The infrastructure state is stored in a single location accessible to all team members.
• Security. Remote storage often offers data encryption and access control capabilities.
• Scalability. Remote storage can handle large amounts of data and multiple operations in parallel.

Terraform supports various types of remote storage, including AWS S3, Google Cloud Storage, Azure Blob Storage, and others.

Why it is recommended to use S3 rather than git

Using git to store tfstate can lead to several problems:

• Synchronization complexity. Git is not designed for simultaneous access and changes, which can cause conflicts when multiple users are working at the same time.
• Limited security. Storing state files in a git repository requires extra effort to keep them secure and private.
• Lack of versioning of state. While git provides version control for source code, it is not optimized for storing and managing infrastructure state files.

In turn, S3 storage is well suited for storing Terraform state. And here's why:

• Constant availability: S3 provides high availability and data fault tolerance, which is critical for state files.
• Versioning: S3 supports object versioning, which allows you to track and restore previous versions of state files.
• Encryption and Security: S3 offers built-in data encryption mechanisms at both the storage and transmission levels, as well as access control capabilities with Identity and Access Management (IAM).
• Speed and scalability: S3 is optimized to handle large amounts of data quickly and scalably, making it more suitable for storing state files than git.

Dynamic blocks

Let's first understand what these are and why to use them.

Dynamic blocks in Terraform allow you to create configurations with a higher level of flexibility and automation. They are used to generate recurring blocks of code based on input data, which simplifies infrastructure management and reduces duplicate code.

A dynamic block is a configuration block that is generated based on a for_each loop. It allows you to create many similar blocks by applying the same parameters and settings to them. This is especially useful when you need to manage multiple resources with similar configurations.

What are the benefits of dynamic blocks

There are many benefits to using dynamic blocks in Terraform:

1. Reduced code duplication: dynamic blocks avoid repeatedly writing the same configuration for each resource. This simplifies code maintenance and reduces the possibility of bugs.
2. Flexibility and scalability: with dynamic blocks, you can easily add or remove resources by simply changing the input data. This makes the configuration more adaptable to changing requirements and scalable as the infrastructure grows.
3. Simplify management: Dynamic blocks make complex configurations easier to manage by providing centralized management of parameters and settings for a group of resources. This allows changes to be made more quickly and applied to all related resources.
4. Improved code readability: although dynamic blocks may seem complex at first glance, they help make code more organized and structured. This improves readability and makes it easier for other team members to understand configurations.
5. Examples of use cases:

• Managing a set of resources with the same configuration, such as virtual machines, databases, or network resources.
• Create multiple instances of a resource with slight differences in configuration, such as different security settings for different groups of resources.

Example of a dynamic block in Terraform

To illustrate, here is an example of using a dynamic block to create multiple virtual machines with different parameters:

variable "instances" {
  type = list(object({
    name = string
    size = string
  }))
  default = [
    { name = "vm1", size = "small" },
    { name = "vm2", size = "medium" },
    { name = "vm3", size = "large" }
  ]
}

resource "aws_instance" "example" {
  for_each = { for instance in var.instances : instance.name => instance }

  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = each.value.size

  tags = {
    Name = each.value.name
  }
}

In this example, the instances variable contains a list of configurations for virtual machines. The dynamic for_each block is used to create aws_instance resources for each virtual machine in the list. This makes adding or removing machines easy by changing only the instances variable.

Practical part: deploying Terraform modules to Yandex Cloud

Let's go through the process of deploying infrastructure to Yandex Cloud using Terraform modules from the nixys/nxs-marketplace-terraform repository. We will be deploying an S3 bucket and Managed Kubernetes with worker nodes.

Preliminary step: Getting access to work with Yandex Cloud
Before we get started, you need to follow these steps to get access to Yandex Cloud:

Create a service account and access key:

In the Yandex Cloud management console, create a service account and assign it the necessary roles (e.g. storage.admin, editor, viewer).

Generate an access key (IAM key) for the service account. This key will be used for authentication when working with Terraform.

Install Yandex CLI:

Install the Yandex CLI (yc) and authenticate.

curl -sSL https://storage.yandexcloud.net/yandexcloud-yc/install.sh | bash
yc init

Export environment variables:
Export environment variables with authentication data.

export YC_SERVICE_ACCOUNT_KEY_FILE=<path_to_service_account_key>
export YC_CLOUD_ID=<your_cloud_id>
export YC_FOLDER_ID=<your_folder_id>

Deploying S3 via Terraform module

To deploy S3-buckets, use the YandexCloud/Storage/buckets module.

Create a working directory and initialize the project:

mkdir terraform-yc-storage
cd terraform-yc-storage
terraform init

Create a main.tf file and add the configuration for the S3 bucket:

module "yc_storage_buckets" {
  source = "github.com/nixys/nxs-marketplace-terraform//YandexCloud/Storage/buckets?ref=main"

  yc_cloud_id    = var.yc_cloud_id
  yc_folder_id   = var.yc_folder_id
  yc_sa_key_file = var.yc_sa_key_file

  buckets = [
    {
      name        = "my-tfstate-bucket"
      access_key  = var.access_key
      secret_key  = var.secret_key
    }
  ]
}

Create a variables.tf file to define variables:

variable "yc_cloud_id" {}
variable "yc_folder_id" {}
variable "yc_sa_key_file" {}
variable "access_key" {}
variable "secret_key" {}

Create the terraform.tfvars file and set the values of the variables:

yc_cloud_id    = "<your_cloud_id>"
yc_folder_id   = "<your_folder_id>"
yc_sa_key_file = "<path_to_service_account_key>"
access_key     = "<your_access_key>"
secret_key     = "<your_secret_key>"

Run Terraform to deploy the S3 bundle:

terraform apply

Why nxs-marketplace-terraform

Using nxs-marketplace-terraform has many benefits that make infrastructure management more efficient. Here are the main ones:

Ready-made tested modules

• Reliability and validation: all modules and modules provided in nxs-marketplace-terraform are thoroughly tested. We put them to work ourselves on real projects.
• Risk mitigation: using proven modules will reduce the likelihood of errors and simplify the process of deploying and managing your infrastructure. Fewer errors mean less downtime.

Unified style

• Unified approach: we've designed all modules in a unified style to make them easier for everyone to understand and use. For example, if you change cloud providers, you won't have to learn new approaches and rewrite configurations from scratch.
• Time savings: A unified style allows your team to adapt faster to a new environment and manage your infrastructure more efficiently, regardless of cloud provider.

Constant updates and improvements

• Relevance: we are constantly monitoring new trends in the cloud technology world. Therefore, all modules are regularly updated to meet the latest standards and best practices.
• Community support: we would love to see you among the project participants! The more people suggest improvements and report problems, the faster nxs-marketplace-terraform will become even more convenient and useful.

nxs-marketplace-terraform will simplify and speed up your work. Stay tuned for updates and join the user community to get the most out of this tool. We look forward to seeing you :)

Conclusion

In this article, we have covered how to configure and deploy various components in Yandex Cloud including S3 storage, Managed Kubernetes, and Managed PostgreSQL using Terraform modules. In addition, we discussed the importance of properly storing Terraform state (tfstate) and the benefits of using remote storage such as S3.

Using nxs-marketplace-terraform ready and tested modules can greatly simplify the process of deployment, reduce risks, and ensure infrastructure reliability. The uniform style of the modules ensures ease of understanding and adaptation to new cloud providers, while regular updates and community support ensure the tools are up-to-date and improved.

This repository is an example of the modules we use when working on customer infrastructures. Publicly available is one of the most up-to-date parts of the set that our engineers and our customers' engineering teams work with.

Nixys has two other repositories with a similar approach:

nxs-marketplace-ansible
nxs-marketplace-k8s-apps

By following the suggested steps and guidelines, you will be able to manage your cloud infrastructure effectively, ensuring the stability and performance of your projects.

We hope that the information provided will be useful and help you in realizing your goals.

Don't forget to stay tuned and participate in the nxs-marketplace-terraform user community to get the most out of this powerful tool. Also subscribe to our social networks: Twitter, and LinkedIn. Different content is published everywhere!