DEV Community: Ankit Jain

Adopting OpenTofu as an Alternative to Terraform

Ankit Jain — Mon, 18 Mar 2024 22:20:21 +0000

Infrastructure as code (IaC) is a concept for deploying infrastructure like a software application. Ongoing advancements in infrastructure virtualization have made this possible. The rise of cloud computing transformed physical servers, networks, and hardware infrastructure into virtualized services. Add assets like databases, domain name service (DNS) providers, and authentication systems, and you get infrastructure that can be deployed like software.

With a few lines of configuration script, you can deploy operating systems, networks, and storage to a remote data center with a single command. The system can be kept in sync with updates to the configuration, which makes changing or redeploying the infrastructure a breeze.

IaC should not be confused with configuration management. Tools in this category, such as Chef, Puppet, or Ansible, typically deploy and configure software components on existing infrastructure. Terraform, on the other hand, is a provisioning tool that specializes in setting up infrastructure components like operating systems, networks, databases, users, and permissions on bare server hardware. Terraform can play hand-in-hand with configuration management tools by providing the infrastructure necessary for deploying software components.

When Terraform's licensing changed, an open-source fork called OpenTofu was spun off. In this article, you'll learn about OpenTofu's features and how OpenTofu compares to Terraform.

The goal of this article is to help you make an informed decision about using OpenTofu to manage your infrastructure requirements as code.

What Is OpenTofu

OpenTofu provides a simple but powerful configuration language to describe the infrastructure that you want to deploy. You specify the number and size of virtual servers to deploy to, the operating system to use, the virtual network overlays, the DNS settings, and anything else your infrastructure needs.

Then you run a tool that reads configuration scripts written in a specialized configuration language and interacts with infrastructure resources called providers to build and deploy the required parts of the infrastructure.

If anything needs to be changed, you update the script and run the tool again. The tool then updates the actual state of your infrastructure to match the description.

OpenTofu and Terraform

You may have come across the name Terraform in the context of IaC. Terraform and OpenTofu are closely related: OpenTofu is a fork of Terraform 1.5.

Years ago, a company called HashiCorp created Terraform, among other DevOps tools like Vagrant, Packer, Nomad, Vault, and Consul. All these tools aim at automating software orchestration and infrastructure management.

Terraform's approach to provisioning infrastructure consists of a three-step workflow: write, plan, and apply. In the write step, you define the resources needed to run the infrastructure. In the plan step, Terraform compares the written infrastructure definition against the existing infrastructure and creates an execution plan for creating, updating, or destroying resources as needed. Finally, in the apply step, Terraform applies the planned changes to the target infrastructure.

To utilize as many infrastructure resources as possible, Terraform uses a plugin system that allows resource providers, such as DNS services, container orchestration systems, database services, or logging systems, to be included.

Terraform's License Change

Terraform had become a popular IaC tool when HashiCorp, its maker, decided to change the license for their tools from the Mozilla Public License (MPL) 2.0 to the Business Source License (BSL). This move aimed to protect HashiCorp from competitors who could set up hosted Terraform offers just like HashiCorp had and reap the benefits without contributing back.

HashiCorp is not the first company to make this move. Earlier, companies like MongoDB, Redis, and Cockroach Labs also decided to restrict the ability to resell their (otherwise open source) code for similar reasons.

Nevertheless, HashiCorp's announcement raised concerns among Terraform users about the possible legal implications of the license switch. It didn't take long until the latest MPL-licensed Terraform version got forked. The fork was originally named OpenTF but was later renamed to OpenTofu due to trademark concerns.

Because Terraform 1.6.x and beyond will be under the new BSL, the OpenTofu project cannot take over changes made to Terraform anymore. While OpenTofu aims to keep feature parity with Terraform, and both projects come with a backward compatibility promise (1) and (2), the two projects may slowly diverge.

Changes Introduced after the Fork

After the fork, the changelog for OpenTofu 1.6.0 lists several significant changes, including the following:

Conditional GNU Privacy Guard (GPG) validation bypass for the default registry
Changes to the cloud and remote backends and the login and logout commands that no longer default to app.terraform.io
A new tofu test command that significantly changes how tests are written and executed
Several enhancements and bug fixes

Additionally, the OpenTofu community continues to add changes to its roadmap.

Is OpenTofu a Viable Alternative to Terraform

If you're in search of an IaC tool, you'll need to decide between Terraform and OpenTofu. Existing Terraform users even have three options to consider: Whether to stay with Terraform, switch to OpenTofu now, or stick with Terraform 1.5.x for a while and watch future developments closely.

For both audiences, there are numerous reasons for choosing the free and open source (FOSS) alternative.

Compatibility

As of this writing, OpenTofu is fully compatible with Terraform 1.5.x, but the list of features will increasingly diverge in the future. The OpenTofu FAQ has a clear statement about a community-driven approach shaping OpenTofu's future: "The community will decide what features OpenTofu will have."

However, OpenTofu will remain backward-compatible so that existing Terraform configurations (made with Terraform up to 1.5.x) continue to work with future versions of OpenTofu (per the OpenTofu Manifesto).

Additionally, the OpenTofu core team confidently predicts that "the large number of developers pledging their resources to help develop OpenTofu will accelerate the development of features and enable faster releases than Terraform managed previously." In other words, features that Terraform users have been anticipating for some time may come to life faster with OpenTofu than with Terraform.

A Free and Open License

The compatibility question may fade in the future when OpenTofu and Terraform are established as two independent projects and interested parties have a decent list of features to compare before settling on one of the tools. Switching between the two will become less frequent.

What will remain is the question of the license under which each of the projects is offered. Terraform's BSL is a source-available license that does not guarantee that users can use the code in their preferred manner. Notably, users may not include Terraform in offerings to third parties that compete with HashiCorp's offerings. (See the "Additional Use Grant" in the HashiCorp BSL.)

In contrast, OpenTofu is licensed under the MPL 2.0, a true FOSS license that grants users the right to use the code as they wish. Having a true FOSS license means that development is driven by the community that was built around OpenTofu, which is large and thriving. The list of supporters includes more than 150 companies and more than 780 individuals.

Additionally, OpenTofu has been adopted by the Linux Foundation, a step that guarantees vendor-neutral governance of the project. The Linux Foundation maintains several projects that are used worldwide, including Linux, Kubernetes, and Node.js.

When to Choose OpenTofu over Terraform

Reasons for choosing OpenTofu vary based on the user.

Companies may have the largest incentive to choose OpenTofu. Use cases for an IaC tool are constantly at risk of conflicting with the BSL. Moreover, the Terraform project changed its license unexpectedly, and it may do so again. With a true open source license and under the umbrella of a nonprofit foundation, OpenTofu provides a far more predictable future.

Digital agencies and consultants should strive to offer their clients a solution whose legal basis is predictably stable and immune to the opaque decisions of a single company. OpenTofu is not only that. It's also backed by a large list of supporters who joined forces to enhance and extend OpenTofu based on the wishes of the community.

Individual users may feel unaffected by Terraform's switch to the BSL. But then there is little that speaks against using OpenTofu instead, which comes under a more open license. While the BSL might have little effect on individual, noncommercial users, nobody knows what possible future changes to the Terraform license will bring. Choosing an open source project now, when switching costs are low, is the sensible choice.

Can Terraform Still Be a Good Choice?

With all the obvious advantages of truly open-source projects, it would seem that no one would choose to use Terraform. Don't judge too quickly, though. It's not all black and white.

For instance, Terraform’s availability as a cloud service when paired with Hashicorp’s enterprise-level support, could be a combo that's difficult for other competitors to match.
Moreover, some customers might conclude that their Terraform use cases are not negatively affected by Terraform's switch from an open-source license to a source-available license.

Granted, these advantages are rather specific to particular customers and use cases. OpenTofu might catch up on these aspects quickly—never underestimate the power of open source.

OpenTofu Continues to Evolve

As mentioned previously, OpenTofu and Terraform are likely to take different paths in the future. With hundreds of supporters committed to shaping the future of OpenTofu, the open source path is likely the one that leads to a more feature-rich and mature product than Terraform.

For example, in August 2023, OpenTofu announced an experimental implementation of end-to-end encryption for state files, a feature that Terraform users have been waiting for since 2014.

Chances are that there will be more of these initiatives to implement long-awaited features soon.

The Future of IaC Tools Is Open Source

The power of open source should not be underestimated. For any company, consultant, or developer whose business model might be threatened by the sudden license change of Terraform, the OpenTofu project provides a solid open source alternative.

OpenTofu is compatible with Terraform and aims to stay that way. It's a Linux Foundation project free of business tactics and legal insecurity. And it's backed by a large and active community.

While the BSL forbids certain kinds of uses of Terraform, the MPL 2.0 grants users complete freedom in using OpenTofu. The Linux Foundation has a track record of widely successful open source projects, and OpenTofu is a worthy addition.

Existing Terraform users may envision a few risks when making the switch to OpenTofu, but those risks are more than mitigated by the stability that an open source, foundation-led project provides. The best time for making the switch is now, while the two projects are still similar enough to enable a smooth transition.

Pre and post-merge tests using a merge queue

Ankit Jain — Thu, 14 Mar 2024 22:43:06 +0000

One of the key ingredients making developers productive is faster feedback loops. A fast feedback loop allows developers to identify and address issues promptly, leading to higher-quality code and faster release cycles.

Pre-merge and Post-merge tests

To maintain the good health of your system, there are a few types of tests that may be required. Validating these tests can take anywhere from a sub-second to several hours.

In the traditional waterfall model, testing is often a phase that occurs after development. However, the emphasis is on catching issues early in agile development and CI/CD workflows. But running all the tests can be extremely time-consuming, and provides slow feedback to the developers.

Instead, consider dividing tests into “pre-Merge” and “post-Merge” buckets. For instance, explore how LinkedIn manages pre-merge and post-merge workflows.

Splitting the tests

There are a couple of considerations to split the tests effectively:

The pre-merge tests should be faster to execute ensuring a fast feedback cycle for the developers
On the other hand, the post-merge tests should be generally more stable. If these tests fail often, we end up in a constant state of failed mainline. But, we should still expect these tests to fail occasionally. If a test doesn’t ever fail, it’s not worth running!

So let’s apply that to some types of tests, these answers may vary depending on your setup but should be generally agreeable.

Code linting

Pre-merge: Linting could catch a bug early in the developer lifecycle, and is both cheap and fast to run.

Unit tests

Pre-merge: They help catch bugs at the smallest level, ensuring that each piece of the puzzle works independently.

Integration testing

Pre-merge: Integration tests verify the interactions between various modules, ensuring a cohesive and functional application.

Automated regression testing

Post-merge: Running tests to identify anything that was previously working that may have regressed. We expect these would fail less often but are still essential to maintain code health.

Performance testing

Post-merge: As changes are deployed over time, the performance of the software may be impacted, this is where performance testing is important. We do not expect every change to impact the performance, and these tests should not fail often.

User Acceptance Testing (UAT)

Post-merge: Any type of manual QA or UAT should be handled post-merge as we expect these to be extremely slow. In most cases, if we identify a failure, it is typically resolved in future releases.

Managing the split

Now that we have the tests split, we can still catch most of the issues in the development cycle, and provide fast feedback to the developer. But what happens when the post-merge tests fail?

In most cases, you coordinate with the release team to roll back the change that caused the failure or roll forward a fix manually. This manual process can be annoying and can also cause poor developer experience.

This is where MergeQueue could play an interesting role.

Understanding the MergeQueue

Before delving into the intricacies of managing pre and post-merge tests with the queue, let’s take a moment to understand the role of MergeQueue in the CI/CD pipeline. MergeQueue acts as a gatekeeper, managing code merges and orchestrating the deployment process. If you are unfamiliar with merge queues in general, here’s a good primer.

Most of the modern merge queues offer some variance of an optimistic parallel mode. In such a mode, as changes are submitted to the queue, it creates an optimistic batch that contains all queued changes including the most recent submitted one. This batch is then validated again for all the tests to ensure that it does not break the mainline before merging the PRs.

Fast forwarding merge

A small variation of this optimistic parallel mode is called a fast-forwarding merge. The main difference in the case of fast-forwarding is that you are fast-forwarding the mainline (e.g. master) to these validated commit SHAs instead of creating new commit SHAs when the PRs are actually merged post-validation.

Post-merge is actually post-queue

Using the image above, you can think of the queued PRs as part of a “staging” branch. So as new commits are being added, those get “merged” into this staging. branch before they land on mainline. If we use that lens, we can run the “post-merge” test in the queue instead of running them after the changes are merged in mainline.

From the developer feedback viewpoint, the experience of “handing over” the PR to the queue is the same as merging the PR. But now, your rollbacks can be automated. Since the queue validates the changes before forwarding the mainline, any failure detected gets force pushed out.

Developer experience

As a developer, the workflow would be:

Open a PR, run the pre-merge test, and request a code review
Once the changes are approved and the tests pass, instead of merging the PR, they can enqueue it
MergeQueue will create a new branch with the squash commit of changes and run the CI on it. Developers can still access this staging branch if they want to access the latest codemix, knowing that it may still not have all the tests validated.
If all the tests pass, the mainline is fast-forwarded to this commit SHA, the original PR is flagged as merged.
If any of the required tests fail, the mainline is not impacted, and the developer is notified about the failed tests and the PR remains open for the developer to fix the issue and submit again.

Behind the scenes

The queue runs all the post-merge tests, which may be slow but usually pass. The mainline is always green because the changes are completely validated before they reach mainline. You would want to configure separate test execution and validation for the PRs created by developers and the branches created by the queue. Aviator MergeQueue provides a simple branching structure to split the test execution in all the common CI platforms for efficient CI usage.

Conclusion

In one of our old posts, we also talked about the death by a thousand papercuts due to the broken mainline, but maintaining a faster feedback cycle is also critical for an improved developer experience.

Using MergeQueue to split the tests (pre-queue and post-queue) is a great way to balance both sides of the problem while improving developer productivity.

This branch is out-of-date

Ankit Jain — Mon, 02 Aug 2021 18:17:57 +0000

When you are working by yourself or with a small school project team the source control requirements are pretty low. You do not have to worry about keeping builds healthy all the time or the impact broken builds may have on others. But the story is very different when one is working in large engineering teams. Therefore, most engineering teams rely on Github protected branch rules to ensure stability of the application.

One of the most common restrictions that developers use in Github protected branches is to require status checks to pass before merging. This ensures that the CI is completed successfully before merging the changes into the protected branch (typically master or main).

There’s one additional restriction that requires branches to be up to date before merging. This ensures that the pull requests have been tested with the latest code before merging in the protected branch. This prompt typically shows up in the Pull Request asking developers to update their branch. This setting is less commonly used today, and we will look into why.

Example

So what’s the point of having branches up to date before merging? I recently ran into this scenario, so let me explain with an example:

Let’s say Joseph and Ashley are working on a web application for food ordering service. They are working on a feature to offer a discount for users who would order ahead.

PR #1: Joseph made the following change to calculate the discount offered to the users.

PR 2: Ashley adds a new API endpoint that will be used to charge users.

As you can imagine, the changes independently look pretty safe and pass the test, but when combined together will break the build as the signature of charge_order has changed.

These types of scenarios are very common in large teams where multiple engineers would be working on the same code base. In those cases, it is safer to enable the restriction that requires branches to be up to date before merging.

Side effect

Before you go and turn on that knob in Github, you should understand one side-effect from using this configuration. I remember after we turned it on, engineers started spending a bunch of time playing rebase-athon. So in the above example, let’s say both Ashley and Joseph are ready to merge the changes. Now if they do not communicate with each other, they may both rebase their branches with the latest master and wait for CI to finish. Whoever notices a finished CI would merge the changes, leaving the other developer to start the rebase process from scratch.

This can be optimized using a custom Github action to keep branches up to date. Every time a PR is merged, a Github action can be triggered to update the rest of the branches. Some companies have internally built a system to manage this whole process, for example SubmitQueue at Uber or Merge Queue at Shopify. There are also some plug and play versions of similar concepts, the one I’ve used in the past is called MergeQueue.

The bottom line is that as your team grows, optimizing the code merging process may become critical. This Github setting to restrict branches to be up to date before merging may seem a bit of an overhead, but would pay off to give you and your team peace of mind.

7 Github apps to supercharge your dev teams in 2021

Ankit Jain — Fri, 12 Feb 2021 03:07:24 +0000

Today Github has become not only the hub for all open source projects but also the goto tool for most technology companies. Although the workflows and requirements for the developer teams working in companies may vary a lot from the open source community, Github’s offerings are ubiquitous. However, there are some ways to supercharge your workflows further if you are working in a high-output development team.

There are thousands of Github apps and extensions available today on Github marketplace. Most of them are centered around improving developer productivity. There are a wide variety of such tools to boost code review process, continuous integration, build and deployment automation, project management, security or monitoring. Finding and integrating the right tools can increase developer productivity with as much as 2x. Here’s some of the tools we found on Github marketplace that can visibly move the needle for the most teams.

Slack For Github

Many teams already use Slack today, adding a Github integration for Slack ensures that your team stays on top of all the action happening, and be notified when they have pending code reviews. The app is managed by Github themselves. Although, if you are working in a big team this can easily get very noisy. Therefore it is important to tune your notifications to cut through the noise. Slack for Github offers a bunch of notifications including new commits, new pull requests, new issues, code reviews, new releases and deployment statuses. They also recently launched the Slash commands to open and close pull requests and issues.

ImgBot

ImgBot automatically optimizes your images using lossless compression. If your team is specifically working on a web application, fast load times are very important for the user experience. With ImgBot you can be certain that your images are optimized without sacrificing the quality of images. ImgBot reviews your source code after you merge your new image files and sends a pull request with all the images optimized. It also crawls your source code the first time you install the app to retroactively update existing images. It offers a peace of mind for your media assets and one thing off the plate of developers.

MergeQueue

How many hours do you struggle to keep your builds green or waste hours with broken builds? MergeQueue is a new Github collaboration tool that enqueues your pull requests to be merged sequentially. Github also recently launched an auto-merge feature that enables developers to automatically merge after CI is complete, but misses the cases where multiple developers can collectively cause build breakage. MergeQueue works with any type of CI that your current Github environment supports.

Codecov

Codecov provides great visibility into the source code coverage. The tool has come a long way, and now offers support for pretty much every programming language. You can define the requirements of coverage percentage before a PR can be merged into the main repository. It also gives you coverage reports posted as comments on the PR itself.

CodeFactor

CodeFactor offers static analysis for your code. They also support most of the common programming languages. It’s a great way to keep your code quality high without sacrificing developer time to do thorough code reviews. It is also a great way to identify silly mistakes before they make it to the main source branch.

Stale

Stale bot is managed by Github themselves. It automatically closes the stale PRs and issues after a period of inactivity. The bot also has a great feature where it first labels the PR as stale, reminding folks that this change would be closed if no further action is taken. And then eventually closes the change after a duration of inactivity. A simple configuration file helps you define the duration of flagging something as stale and the duration of inactivity to close the PR or issue.

Dependabot Preview

Many of you may already be familiar with the Dependabot that helps keep all your dependencies up to date. When it identifies a stale dependency, it creates a pull request to update the build configuration to the latest version. It manages the versioning using the semantic versioning server called semver. Dependabot is now officially managed by Github and is natively supported without any apps needed. All you need to do is set a dependabot.yml file in your root directory.

There are clearly a variety of Github apps, and there may not be a one-size-fits-all approach, but we have found using this set of tools puts a lot of manual work on autopilot and frees up time for developers to do more meaningful work. Configuring some of these tools may feel time consuming, but one time configuration can yield a lot of saved hours for your team.

If you have found any other interesting Github apps, please leave a comment or DM me, I would love to check them out and update my list.