DEV Community: Utpal Nadiger

Streamlining Infrastructure as Code: A Guide to Terraform Automation, Collaboration, and Governance in Large Organizations

Utpal Nadiger — Thu, 15 Feb 2024 17:51:03 +0000

Terraform/OpenTofu is the most widely adopted infrastructure as code tool that allows teams to define and provision infrastructure using a high-level configuration language.

In large organizations, where the complexity and scale of infrastructure can be significant, in addition to using the tool itself, the organisations layer automation, collaboration, and governance features on top using relevant tools/product, as they are crucial for maintaining efficiency and compliance. Let's dissect the phases involved when Infrastructure as code is used in large teams.

Development Phase

The process begins in the development phase, where changes to the infrastructure is proposed.

Here, developers use static analysis tools to ensure code quality and security standards are met before submitting changes for review.

Checkov is a popular Static Analysis tool used in a lot of enterprise set ups

This phase may involve an orchestrator, which is responsible for managing the workflow of tasks, and a plan generator that creates an execution plan for the proposed changes.

Plan Stage

Once the initial code review is passed, the plan stage involves generating a detailed plan of the proposed infrastructure changes.

This includes detecting drift from the current state, scaling considerations, cost estimations, and compliance checks.

The plan preview is essential for understanding the impact of the changes before they are applied.

Tools such as digger can help in plan previews in PR comments

State Management

State management is a critical aspect of Terraform's operation in large organizations.

It involves tracking the state of the infrastructure in a state file.
This allows Terraform to map real-world resources to the configuration, keep track of metadata, and improve performance for large infrastructures.

Risk Assessments

At the center of the workflow is the risk assessment process. This is likely a tool or set of metrics that evaluates the potential risk associated with the changes. This is where the engineering-security team typically steps in.

Factors may include benchmarks and baselines for performance, cost, and security.

The risk assessment is crucial for governance, ensuring that changes adhere to organizational policies and do not introduce unacceptable levels of risk.

Governance and Approvals

Governance in Terraform is enforced through policies that dictate what can be deployed and under what conditions.

Approvals are a part of this governance framework, requiring oversight by senior engineers or automated checks to ensure all standards are met before proceeding.

Version labeling and manifest generation add additional layers of tracking and accountability.

Deployment and Testing

Once the proposed changes have passed through all the previous checks and assessments, they are ready for deployment. The deployment process involves applying the changes to the infrastructure.

This is followed by a series of tests to ensure that the deployment was successful and the infrastructure is operating as expected.

Metrics and Versioning

The final part of the Terraform workflow in large organizations includes metrics collection and versioning.

Metrics and detectors are used to monitor the infrastructure's performance and risk post-deployment, ensuring any anomalies are detected quickly.

Version control is important for tracking changes over time, allowing for audits and rollbacks if necessary.

Digger

Thank you for reading until the end. Before you go, just wanted to share the following:

We're building an Open Source Tool that helps you orchestrate Terraform within CI/CD systems such as GitHub Actions while providing RBAC via OPA, Drift Detection and Concurrency with a self hostable orchestrator backend. Our goal is to essentially provide the set up mentioned above for your team. Would love your feedback!
Star us on GitHub | Check out Docs | Blog | Slack

Terraform drift detection and remediation - a primer

Utpal Nadiger — Thu, 08 Feb 2024 17:54:36 +0000

We know Terraform/OpenTofu is the most widely used IaC tool for automating and managing cloud infrastructure.

However, with the power of managing infrastructure as code, comes the challenge of managing 'drift' - the divergence between the intended state of infrastructure as defined in Terraform configurations and its actual state in the cloud.

Drift can manifest for various reasons, from manual interventions to overlapping tool functionalities, and poses significant risks including security vulnerabilities, compliance issues, and financial overheads. This article delves deep into Terraform drift detection and remediation, exploring the underlying causes, the potential impacts, and the strategies to detect and rectify this aspect of cloud infrastructure management.

Common Causes of Drift

The occurrence of drift in Terraform-managed infrastructure can be attributed to several factors. A primary cause is manual changes made directly in the infrastructure through interfaces like the AWS Console, which are not recorded in the Terraform state file. This discrepancy leads to a state of drift. Another significant factor contributing to drift is the use of multiple automation tools with overlapping capabilities.

For example, using Terraform alongside Ansible, which also possesses infrastructure provisioning capabilities, might result in inconsistencies and drift if not managed properly.

Additionally, user-defined scripts, such as bash or shell scripts in cloud platforms, can also lead to drift by modifying network configurations or other resources.

Emergency changes or hotfixes applied directly through interfaces like the AWS Console can bypass Terraform's regular processes, further leading to drift.

A lack of adequate training among non infra savvy developers in the team (yes - "shifting left" gone wrong) regarding Terraform's use can result in them making direct updates through cloud service consoles, inadvertently introducing drift.

Implications of Drift

Drift in Terraform can have far-reaching implications. It can expose security vulnerabilities, leading to potential data breaches and system compromises. In terms of compliance, drift can lead to violations, especially when it results in unauthorized access or exposure of sensitive data. Operationally, drift can introduce challenges, increasing system downtime and impacting performance. Untracked changes can also lead to increased operational costs, particularly if they involve the use of higher-spec resources that were not part of the original plan.

Detecting and Managing Drift

Detecting and managing drift in Terraform is a multifaceted process. The use of Terraform commands such as terraform refresh and terraform plan plays a critical role in identifying drifts. Additionally, periodic monitoring of the infrastructure using these commands can aid in early detection and prevention of larger issues. Tools like Digger and Terraform Cloud offer dedicated drift detection and remediation mechanisms. These tools provide continuous monitoring and notifications, enabling teams to stay informed about the state of their infrastructure.

Drift Detection in Digger - Slack Alerts

Drift Remediation Approaches

There are two primary approaches to remediate drift in Terraform. The first involves reconciling the changes, which means restoring the infrastructure to the state defined in the Terraform code. This typically involves reapplying the configuration. The second approach is aligning the Terraform code with the current state of the infrastructure. This is often necessary when changes made outside of Terraform are deemed necessary and should be maintained. This involves updating the Terraform code to reflect the real-time state of the infrastructure.

In conclusion, managing infrastructure drift in Terraform environments requires diligent monitoring, effective use of tools, and clear policies and training for team members. This comprehensive approach is vital to ensure that the infrastructure remains secure, compliant, and aligned with the organizational goals and requirements.

Digger

Thank you for reading until the end. Before you go, just wanted to share the following:

We're building an Open Source Tool that helps you orchestrate Terraform within CI/CD systems such as GitHub Actions while providing RBAC via OPA, Drift Detection and Concurrency with a self hostable orchestrator backend. Would love your feedback!
Star us on GitHub | Check out Docs | Blog | Slack

Typical challenges faced while setting up CI/CD for Terraform at scale

Utpal Nadiger — Wed, 31 Jan 2024 15:29:55 +0000

Say, hypothetically, that you are a part of a startup handling payments, and after a good few months of effort, you have now codified all of your infrastructure in Terraform. Your next task is to set up CI/CD for terraform to automate terraform deployments and enable collaborative, self-service deployments for all the (hypthetical) 500 developers in your team. This article aims to explain the challenges you are likely to face while doing this.

Setting up CI/CD for Terraform at scale involves addressing challenges such as managing state files, ensuring environment consistency, and maintaining collaboration and version control.

The process also includes implementing automated testing, modularizing code, and establishing workflows to manage the dynamic nature of IaC.

Let's get into the specifics, then.

Modularization of Code

In CI/CD for IaC, code modularization is crucial. It involves breaking down infrastructure code into distinct, reusable modules. This approach enhances manageability, allowing developers to focus on specific areas without impacting others. It also reduces duplication, as common functionalities are abstracted into modules that can be reused across different parts of the infrastructure. This method streamlines updates and maintenance, as changes to a module propagate wherever it's used, ensuring consistency and reducing the likelihood of errors in large-scale deployments.

(Image credit - FreeCodeCamp)

Secret Management

Managing secrets, such as API keys and passwords, is essential in IaC. Exposing these secrets can lead to security vulnerabilities. Using secret management tools is a recommended practice (Hashicorp Vault, or Infisical are highly recommended by the community).

(Image credit - IN4IT)

These tools store secrets securely and provide controlled access to them, ensuring that sensitive information is not hardcoded in the IaC scripts. Proper secrets management not only enhances security but also simplifies the process of rotating and updating secrets, which is a critical aspect of maintaining the security posture in CI/CD environments.

Infrastructure Monitoring and Logging

Setting up comprehensive monitoring and logging is vital for maintaining the health and performance of infrastructure. This involves collecting, analyzing, and storing logs from various components of the infrastructure.

Effective monitoring enables teams to detect and respond to issues proactively, minimizing downtime. Logging provides valuable insights into the performance and behavior of the infrastructure, aiding in troubleshooting and optimizing resource utilization. In large-scale deployments, automated monitoring and alerting systems are indispensable for maintaining stability and performance.

Handling Dependencies

Effective management of dependencies between different infrastructure components is crucial. Dependencies must be clearly defined and managed to ensure that changes in one component do not adversely affect others.

This requires a thorough understanding of the infrastructure's architecture and the interrelations between its components. Proper dependency management facilitates smoother updates and minimizes the risk of failures during deployment. It also aids in predicting the impact of changes, allowing for better planning and testing of updates.

Cost Management

Monitoring and managing the costs associated with deployed infrastructure is essential to avoid budget overruns. This includes regular reviews of resource utilization, identifying underutilized or unnecessary resources, and optimizing infrastructure to balance performance with cost.

Implementing cost monitoring tools and setting up alerts for budget thresholds helps in maintaining financial control. Effective cost management ensures that the infrastructure meets the required performance standards without incurring unnecessary expenses, which is especially important in large-scale operations where costs can escalate quickly.

Digger

Thank you for reading until the end. Before you go, just wanted to share the following:

We're building an Open Source Tool that helps you orchestrate Terraform within CI/CD systems such as GitHub Actions while providing RBAC via OPA, Drift Detection and Concurrency with a self hostable orchestrator backend. Would love your feedback!
Star us on GitHub | Check out Docs | Blog | Slack

GitHub issues from top Open Source Golang Repositories that you should contribute to

Utpal Nadiger — Mon, 15 Jan 2024 17:36:50 +0000

This article is for contributors who are looking to contribute to cool open source go projects.

We collated a bunch of issues that we thought the community would be interested in contributing to, feel feel to check them out!

Digger - an Open Source Terraform Automation & Collaboration tool

Before we dive in to the issues, let me share Digger with you. It's an open-source Infrastructure as Code Management tool that helps you securely use IaC such as Terraform and OpenTofu as a team. Digger is an alternative to Hashicorp's Terraform cloud, and the best part is that it's completely open-source!

I would be extremely grateful if you could give us a star & share your thoughts in the comments section below
https://github.com/diggerhq/digger

We've picked up issues from projects such as:

Flipt (Open Source Feature Management tool)
CasaOS (open-source Personal Cloud system)
Tailscale (VPN service for secure networks)
Memos (A lightweight note taking service)
s5cmd (Parallel S3 and local filesystem execution tool.)
AdGuardHome (Network-wide ads & trackers blocking DNS server)
Transfer.sh (File sharing from the command-line.)
Digger (Terraform automation and collaboration tool)

Diving right in to the issues! 👇

The Syntax of the list below will be {project name} - {Issue Title} and will link directly to the issue in the repository of the project that it is a part of.

Tools used by the top 1% of Platform Engineers and their Commercial Open Source Alternatives

Utpal Nadiger — Fri, 12 Jan 2024 16:15:41 +0000

In today’s article, we are diving deep into the top tools used by platform engineers and their commercial open source alternatives

The tools that we will be talking about will around:

Infrastructure as code management
Service Catalogs
Observability
CI/CD within Hosted Git

Problem that tool solves	Open Source	Proprietary SaaS
Infrastructure as Code Management	Digger	Hashicorp Terraform Cloud
Service Catalog	Backstage	Port
Observability	Signoz	Datadog
CI/CD within VCS	Gitlab CI	Github Actions

Lets dive into each of them one by one:

Infrastructure as code (IaC) management

Infrastructure as Code management involves being able to use IaC as a team in a reproducible and consistent manner. Teams use either OpenTofu or Terraform to provision their infrastructure as code, but use IaC managment tools, often called “Terraform automation and Collaboration Software” (Tacos) to manage and automate this in a team setting - specifically for things such as state management, role based access controls, drift detection and concurrent runs.

Digger is a Commercial Open Source IaC management platform and Hashicorp’s Terraform Cloud is the proprietary SaaS company offering IaC management.

Check Digger's repo on GitHub

Service Catalogs

A service catalog centralizes all infrastructure tools, services, and documentation, streamlining the development process. It acts as a single point of access for developers, enhancing efficiency and coherence in the deployment and management of infrastructure. By consolidating resources and guidelines, it simplifies decision-making and promotes best practices, ensuring a more unified and effective development environment. In addition to this it includes information about deliverables, prices, contact points, and ordering and request processes.

Backstage by Spotify is an Open Source Service Catalog and Port is a Proprietary SaaS offering the same service.

Check the Backstage repo on GitHub

Observability

Observability platforms provide a comprehensive view of environments by visualizing metrics, traces, and logs in a single interface. They allow monitoring of key performance indicators like p99 latency, error rates, and API calls. Users can pinpoint issues' root causes by examining specific traces and detailed flame-graphs of request traces. These platforms enable running aggregates on trace data to derive business-relevant metrics. Additionally, they offer functionalities to filter and query logs, create custom dashboards and alerts, and automatically record exceptions in various programming languages. Their easy-to-use query builders facilitate setting up custom alerts efficiently.

Signoz is a commercial Open Source Observability Tool and Datadog is a popular Proprietary SaaS product offering the same service.

Check Signoz's repo on GitHub

CI/CD within VCS

CI/CD systems integrated within version control platforms like GitHub automate the software delivery process. They enable continuous integration (CI) by automatically testing code changes, ensuring that new code integrates smoothly into the existing codebase. Continuous Delivery (CD) follows, where the tested changes are automatically deployed to production environments.

Using CI/CD within version control systems (VCS) like GitHub, as opposed to external services such as CircleCI, offers several advantages. Integrated CI/CD is seamlessly connected with the codebase, streamlining workflows and reducing the need for additional configuration or context switching. Built-in CI/CD tools in VCS are often more user-friendly for teams already familiar with the VCS environment, and they may provide better integration with other features of the VCS platform.

Gitlab CI is a commercial Open Source CI/CD Tool within Gitlab VCS and GitHub Actions is Proprietary product offering a similar service.

10 open source tools that platform, SRE and DevOps engineers should consider in 2024.

Utpal Nadiger — Thu, 04 Jan 2024 15:44:55 +0000

This article highlights ten open source tools that have gained significant attention amongst infrastructure engineers and are considered essential for professionals in Platform Engineering/DevOps/Site Reliability engineering.

These tools cover a wide range of functionalities, including Infrastructure as Code management, secret management, distributed filesystems, internal developer portals, continuous integration and deployment (CI/CD), and self-hosted Git services.

Each of these open-source projects, from Digger's Infrastructure as Code platform to Gitea's self-hosted Git service, represents a key component in the modern DevOps toolkit, helping engineers to build, deploy, and maintain scalable and efficient software systems.

The tools are:

Digger - an Open Source Infrastructure as Code management platform.[Infrastrucutre as code automation]
Git Secret - A bash-tool to store your private data inside a git repository. [Secret Management]
Infisical - Open source end-to-end encrypted secrets sync for teams and infrastructure. [Secret Management]
Lade - Automatically load secrets from your preferred vault as environment variables. [Secret Management]
Ceph - Highly scalable object, block and file-based storage under one whole system. [Distributed Filesystems]
Backstage - An open platform for building developer portals. [Internal Developer Portal]
Kraken CI - Modern CI/CD, open-source, on-premise system that is highly scalable and focused on testing. [CI/CD]
Buildbot - automate all aspects of the software development cycle. [CI framework]
Gogs - A self-hosted Git service. [Git]
Gitea - Another self-hosted Git service. [Git]

Now lets dive into each tool one by one:

Digger

Digger is an IaC management tool for Terraform and OpenTofu, addressing the complexities often encountered with specialized IaC CI systems like Terraform Cloud and Atlantis.

Its unique approach integrates Terraform/OpenTofu directly into your existing CI infrastructure, leveraging its asynchronous jobs, compute, orchestration, and logging capabilities.

This integration not only enhances security by keeping cloud access secrets within your CI environment but also proves cost-effective by eliminating the need for extra compute resources. Digger's feature set includes Terraform plan and apply within pull request comments, private runners utilizing existing CI compute environments, Open Policy Agent (OPA) support for robust access control, and PR-level locks to prevent race conditions. Additionally, it supports advanced functionalities like Terragrunt, multiple Terraform versions, and drift detection, making it an all-encompassing solution for managing Terraform/OpenTofu deployments efficiently and securely.

Star Digger on GitHub ✨

Check out self hosting documentation

Git Secret

Git Secret is an essential bash tool for developers and DevOps professionals, offering a robust solution for secret management within a Git repository. This open-source tool effectively encrypts sensitive files and data, ensuring that confidential information like passwords, keys, and credentials are securely stored in the repository.

By encrypting files with the public keys of allowed users, Git Secret ensures that only authorized personnel can access and decrypt these secrets. This method not only enhances security but also simplifies the process of sharing sensitive data among team members. It's particularly valuable in collaborative environments, where managing access to sensitive information is crucial for maintaining security and compliance. Git Secret stands out as a practical, secure, and efficient way to handle private data in code repositories.

Star Git Secret on GitHub ✨

Infisical

Infisical is an open source secret management platform tailored for teams to centralize crucial data such as API keys, database credentials, and configurations. Aimed at making secret management accessible to everyone, not just security experts, it redesigns the entire developer experience. The platform offers a user-friendly dashboard for managing secrets across various projects and environments, client SDKs for on-demand secret retrieval, and a CLI tool for integrating secrets into any framework during local development.

Infisical includes native integrations with platforms like GitHub, Vercel, and Netlify, and features such as automatic Kubernetes deployment secret reloads, self-hosting options on different infrastructures, secret versioning, Point-in-Time Recovery, comprehensive audit logs, Role-based Access Controls, simplified on-premise deployments to AWS and Digital Ocean, along with secret scanning and leak prevention capabilities.

Star Infisical on GitHub ✨

Lade

Lade is a practical tool designed to enhance secret management by automatically loading secrets from a user's chosen vault into environment variables or files. This functionality is key in minimizing the exposure of sensitive information, as it restricts access to secrets only for the duration of a specific command's execution. By ensuring that secrets are only available when absolutely necessary, Lade significantly reduces the risk of unauthorized access or leaks. This approach is particularly beneficial in environments where security and data privacy are paramount. Lade is part of the Metatype ecosystem. Consider checking out how this component integrates with the whole ecosystem and browse the documentation to see more examples.

Star Lade on GitHub ✨

Ceph

Ceph stands out in storage technology, offering a scalable and reliable solution where traditional systems fall short. It supports object, block, and file storage in one system, adaptable for various environments including on-premises, cloud, or container-native setups. Key benefits include scalability, enabled by the CRUSH algorithm, allowing for expansion without typical downtime. This makes Ceph suitable for businesses and institutions needing to grow their storage capacity rapidly.

Ceph is also notable for its reliability. It is self-managing and self-healing, with Monitor and Manager daemons enhancing data availability. The CRUSH algorithm reduces failure risks, ensuring a robust storage solution.Performance-wise, Ceph's customizable deployment suits diverse needs without compromising efficiency. As a software-defined system, it performs well regardless of the infrastructure, addressing the limitations of traditional storage systems.

Backstage

Backstage is an innovative open platform designed for creating internal developer portals, streamlining the developer experience within organizations. As a centralized hub, it allows teams to manage software components, monitor services, and access tools and documentation from a single interface.

This enhances collaboration and increases efficiency by reducing the complexity often associated with accessing various development tools and resources. By providing a unified, customizable environment, Backstage fosters a more organized and coherent workflow. Its open-source nature invites contributions and adaptations to suit specific organizational needs, making it an invaluable tool for companies looking to optimize their internal software development processes.

Star Backstage on GitHub ✨

Kraken CI

Kraken CI is a modern CI/CD system that operates on the Continuous Integration philosophy, focusing on pre-commit and post-commit phases in software development. In the pre-commit phase, developers and testers prepare code changes, aiming to minimize the risk of breaking production code. Kraken CI facilitates this by providing a validation environment that simplifies testing, making it easier to produce quality code. It reduces the likelihood of large, risky changes and helps manage code integration more effectively.

In the post-commit phase, the emphasis is on maintaining the stability of production code. Kraken CI's effective post-commit validation delivers clear, unambiguous information about the production code, reducing the time to feedback and allowing for quick response to any issues. This results in greater stability and release-readiness of the production code. By improving both pre-commit and post-commit phases, Kraken CI fosters a culture shift in software development. It moves away from a gate-focused approach, where each stage of development is a barrier, to a more fluid process where small changes are made frequently. This shift reduces the impact of breaks and improves the overall quality and efficiency of the engineering process, allowing teams to focus on innovation and delivering unique value to customers.

Buildbot

Buildbot is a versatile CI framework designed to automate all aspects of the software development cycle, enhancing efficiency and reliability. As an open-source platform, it is highly customizable, allowing teams to tailor the automation process to their specific needs. Buildbot excels in integrating various stages of development, from code integration, testing, to deployment, ensuring a seamless and coherent workflow. This framework supports multiple development environments, making it adaptable to different technologies and project requirements. Its ability to streamline complex processes and foster continuous integration and deployment makes Buildbot a valuable tool for teams seeking to optimize their software development lifecycle.

Gogs

The Gogs project is dedicated to creating a simple, stable, and extensible self-hosted Git service, emphasizing ease of setup. Utilizing Go, Gogs offers an independent binary distribution compatible across multiple platforms, including Linux, macOS, Windows, and ARM systems. The platform features a comprehensive user dashboard, profile, and activity timeline, and supports repository access through SSH, HTTP, and HTTPS.

It includes robust management tools for users, organizations, and repositories, alongside webhooks and Git hooks. Gogs facilitates repository issues, pull requests, wiki, and collaboration features. It also offers migration and mirroring of repositories, a web editor for repository files, Jupyter Notebook and PDF rendering, and supports various authentication methods including SMTP, LDAP, and GitHub integration. Additionally, Gogs is customizable, supports a range of databases like PostgreSQL and MySQL, and is localized in over 31 languages, making it a versatile and user-friendly solution for Git hosting.

Star Gogs on GitHub ✨

Gitea

Gitea is a versatile tool for creating and managing git-based repositories, streamlining Code Review to enhance code quality for users and businesses. It integrates a CI/CD system, Gitea Actions, compatible with GitHub Actions, allowing users to create workflows in YAML or use existing plugins. Gitea's project management features include issue tasks, labeling, and kanban boards for efficient management of requirements, features, and bugs. These tools integrate with branches, tags, milestones, assignments, time tracking, and dependencies to plan and track development progress. Furthermore, Gitea supports over 20 package management types, such as Cargo, Composer, NPM, and PyPI, catering to a wide range of public or private package management needs. This comprehensive suite of features makes Gitea a powerful platform for managing development projects and packages.

5 Open Source tools written in Golang that you should know about

Utpal Nadiger — Fri, 15 Dec 2023 15:23:43 +0000

Most modern backend developers absolutely LOVE golang.

Developers continue to build simple, secure, scalable systems with Go, and those who do, swear by it. In this blog, we will explore five essential open-source tools written in Go that are turning heads.

Lets dive right in 👇

Digger

Digger is an Open Source Infrastructure as Code management tool that helps orchestrate IaC such as Terraform & OpenTofu within GitHub Actions. Digger reuses compute used for application code so that you don't overpay for 3rd party managed compute for IaC. This approach eliminates the duplication of CI/CD infrastructure such as compute, jobs, and logs, and reduces security concerns by keeping sensitive data within the CI job. Digger's integration with existing CI systems offers scalability by leveraging on-demand compute resources and enhances security by confining data within the existing CI environment.

Star Digger on GitHub ✨

Good first issues

Buf.build

The Buf CLI is a versatile tool designed for handling Protocol Buffers (Protobuf), a method of serializing structured data. It offers several key features, including managing Protobuf assets through the Buf Schema Registry (BSR), providing a linter to enforce optimal API design and structure, and a breaking change detector to maintain compatibility either in source code or at the wire level. Additionally, the Buf CLI includes a generator that activates plugins based on user-defined templates and a formatter to standardize the formatting of Protobuf files according to industry norms. It also integrates seamlessly with the Buf Schema Registry, supporting comprehensive dependency management.

Star Buf on GitHub ✨

Feature Requests

Permify

Permify is an open-source service for creating and managing complex permissions in applications, inspired by Google Zanzibar. It offers a flexible authorization language compatible with various models like RBAC, ReBAC, and ABAC, and allows for efficient authorization data management in preferred databases. Permify's API facilitates access checks, resource filtering, and bulk permission analyses. It also includes comprehensive testing tools for authorization logic, including scenario-based testing and policy coverage analysis. Additionally, Permify supports multi-tenancy, enabling distinct authorization models for different applications within a single instance.

Star Permify on GitHub ✨

Open Bounties

JuiceFS

JuiceFS under the Apache License 2.0, is a high-performance POSIX file system optimized for cloud-native environments. It stores data in Object Storage (e.g., Amazon S3) and metadata in databases like Redis, MySQL, or TiKV. JuiceFS integrates massive cloud storage with big data, machine learning, and AI applications efficiently, akin to local storage. It features full POSIX and Hadoop compatibility, S3 interface, Kubernetes support, and shared file storage for numerous clients. Some cool features are - strong consistency, scalable performance, data encryption, global file locks, and compression with LZ4 or Zstandard.

Star JuiceFS on GitHub ✨

Open Issues

Steampipe

Steampipe is a tool that simplifies data extraction from APIs and services, eliminating the need for ETL (Extract, Transform, Load) processes. It includes several components: the Steampipe CLI for querying APIs, ensuring compliance, and creating dashboards; Steampipe Postgres FDWs (Foreign Data Wrappers) which turn APIs into foreign tables in Postgres; Steampipe SQLite extensions that convert APIs into SQLite virtual tables; and standalone export tools for directly exporting data from APIs without requiring a database. Additionally, Turbot Pipes, a part of Steampipe, offers capabilities for querying, checking, and visualizing data, tailored for DevOps teams with a focus on intelligence, automation, and security.

Star Steampipe on GitHub ✨

Good first issues

Top 10 terraform tools you should know about.

Utpal Nadiger — Mon, 11 Dec 2023 17:00:00 +0000

Terraform stands out as a powerful Infrastructure-as-Code (IaC) tool on its own, yet as the sophistication of your infrastructure grows, you might discover the need for additional tooling for specific use-cases.

In this article, we will explore some of the leading tools currently employed in deployments managed by Terraform.

Let's dive right in 👇

Digger

Digger is an Open Source IaC management platform that allows you to orchestrate terraform/OpenTofu in your CI/CD system. It helps you resue async jobs infrastructure with compute, orchestration, logs, etc of your existing CI. Digger also has a pro version built on top of Digger’s community edition. Digger’s “bring your own compute” ensures that users have private runners by defualt and don’t have to pay for it additionally. Digger pro gives team leads, managers and IaC practitioners dashboards, Drift Detection, RBAC via OPA policies and concurrency so they can help guide the team.

Star Digger on GitHub ⭐️

Checkov

Checkov is a versatile static code analysis tool designed for infrastructure as code (IaC) and software composition analysis (SCA). It supports a wide range of technologies, including Terraform, CloudFormation, Kubernetes, Docker, and others, to detect security and compliance issues through graph-based scanning. Checkov also performs SCA scans, identifying vulnerabilities in open source packages and images by checking for Common Vulnerabilities and Exposures (CVEs). Additionally, it is integrated into Prisma Cloud Application Security, a platform that helps developers secure cloud resources and infrastructure-as-code files, enabling the identification, rectification, and prevention of misconfigurations throughout the development lifecycle.

Star Checkov on GitHub ⭐️

Former2

Former2 is a tool that automates the creation of Infrastructure-as-Code (IaC) scripts from existing AWS resources. It utilizes the AWS JavaScript SDK to scan the user’s AWS infrastructure, identifying all available resources. Users can then select from this list which resources they want to include in their IaC outputs. This process simplifies the task of writing IaC scripts, especially for complex environments, by directly converting current AWS configurations into ready-to-use code. Former2 is particularly useful for documenting existing infrastructure or for migrating manually created resources into an IaC framework.

Star Former2 on GitHub ⭐️

Infracost

Infracost is a tool that provides cloud cost estimates for infrastructure managed by Terraform. It enables engineers to view and understand the financial impact of their infrastructure changes before they are applied. Infracost integrates directly into the workflow, offering cost breakdowns in various environments like the terminal, Visual Studio Code, or directly within pull requests. This feature allows for more informed decision-making regarding infrastructure modifications, promoting cost-awareness and budget management in the early stages of development. Infracost is particularly useful for teams looking to balance cloud resource utilization with budget constraints. Infracost Cloud is their SaaS product that builds on top of Infracost open source and works with CI/CD integrations. It gives team leads, managers and FinOps practitioners dashboards, guardrails, centralized cost policies and Jira integration so they can help guide the team (e.g. switch AWS GP2 volumes to GP3).

Star Infracost on GitHub ⭐️

Terragrunt

Created and maintained by Gruntwork, Terragrunt is a tool designed to enhance Terraform’s capabilities. It acts as a thin wrapper around Terraform, offering additional features to streamline and optimise Terraform usage. Key functions of Terragrunt include helping users keep their Terraform configurations DRY (Don’t Repeat Yourself), efficiently managing multiple Terraform modules, and handling remote state management. By reducing repetition in Terraform code and simplifying the management of complex module dependencies and remote state, Terragrunt makes working with Terraform more efficient, especially for larger or more complex infrastructure deployments.

Star Terragrunt on GitHub ⭐️

Sato

Sato is a conversion tool designed to translate CloudFormation and ARM (Azure Resource Manager) templates into Terraform configurations. Developed in Go, Sato stands out for its speed and efficiency in this conversion process. By automating the translation of existing templates into Terraform’s syntax, Sato facilitates a smoother and quicker migration to Terraform’s ecosystem.

Star Sato on GitHub ⭐️

Prettyplan

Prettyplan is a user-friendly tool designed to simplify the review of large Terraform plan outputs. It enhances readability by providing an online interface where users can paste their Terraform plan output, which is then reorganized into a more manageable format. Key features include expandable and collapsible sections for a comprehensive yet detailed view, a tabular layout for straightforward comparison of old and new values, and improved display formatting for multi-line strings like JSON documents. Initially created for Terraform versions up to 0.11, Prettyplan’s relevance has diminished with Terraform’s 0.12 update, which incorporated many of Prettyplan’s functionalities, leading to no further updates for the tool.

Star Pretty Plan on GitHub ⭐️

Regula

Regula is a dynamic tool designed for pre-deployment security and compliance checks of infrastructure as code (IaC) for multiple cloud providers and Kubernetes. It supports an array of file types, including CloudFormation JSON/YAML templates, Terraform source code and JSON plans, Kubernetes YAML manifests, and Azure Resource Manager (ARM) JSON templates (currently in preview). Regula leverages a rule library written in Rego, the language used by the Open Policy Agent (OPA) project, offering robust policy evaluation. It integrates seamlessly with popular CI/CD tools like Jenkins, Circle CI, and AWS CodePipeline, and even includes a GitHub Actions example for easy setup. Regula’s policies are aligned with CIS Benchmarks for AWS, Azure, Google Cloud, and Kubernetes, aiding in comprehensive compliance assessments. This tool is actively developed and maintained by the team at Fugue.

Star Regula on GitHub ⭐️

Terraboard

Terraboard is a web-based dashboard designed for visualizing and querying Terraform states. It offers several key features: an overview page that lists the most recently updated state files along with their activities; a detailed state page showing versions and resource attributes of state files; a search interface for querying resources by type, name, or attributes; and a diff interface for comparing state versions. Terraboard supports various remote state backend providers, including AWS S3 for state management and DynamoDB for locking, S3-compatible backends like MinIO, Google Cloud Storage, Terraform Cloud (remote), and GitLab. This makes it a versatile tool for managing and understanding Terraform state files.

Star Terraboard on GitHub ⭐️

TFLint

TFlint is a powerful linter for Terraform, designed to catch errors and issues that terraform plan may not detect. As Terraform grows in popularity for infrastructure as code, the need for robust tools to ensure code quality and reliability becomes paramount. TFlint fulfills this need by analyzing Terraform configurations to find problems that are not covered by syntax checks. It checks for things like unsuitable AWS instance types, incorrect IAM policy syntax, and the use of deprecated syntax or features. By integrating TFlint into the development process, users can proactively identify potential problems, improving the stability and efficiency of their infrastructure deployments. This additional layer of validation is crucial for maintaining high standards in complex, cloud-based infrastructures.

Star TFLint on GitHub ⭐️

This article was orginally published on medium - link here

Migrating from Terraform Cloud to Amazon S3 and DynamoDB: A Guide

Utpal Nadiger — Thu, 25 May 2023 08:54:49 +0000

Terraform’s recent move to the resources under management pricing model has frustrated a lot of users. As one user on this reddit thread remarked:

The pricing on Terraform literally just jumped to almost same as we are paying for the AWS resources it manages.We’re on the free plan, with only 2 devs — were looking at moving to the paid plan this month which was going to be $20/user/month — but now we’re looking at a ~ $250 monthly bill for our dev, test and prod environments. We’re only paying $300 for the AWS resources.

Migrating your backend from Terraform Cloud to an infrastructure based on Amazon S3 and DynamoDB may help save you some $$ and help extend that end of runway.

It involves configuring Terraform to use these services for remote state storage. Here’s a guide on how to perform this migration:

Prepare the infrastructure:

Create an S3 bucket: Log in to the AWS Management Console and create a new S3 bucket to store your Terraform state files. Choose a unique name for the bucket.

Create a DynamoDB table: In the AWS Management Console, create a new DynamoDB table to manage Terraform state locking. Provide a table name and a primary key (e.g., “LockID” of type String).

Update Terraform configuration:

Open your Terraform configuration files (e.g., main.tf, backend.tf).

Remove the existing Terraform Cloud backend configuration.
Add a new backend configuration to use S3 and DynamoDB. Here’s an example:

terraform {
  backend "s3" {
    bucket         = "<your-s3-bucket-name>"
    key            = "terraform.tfstate"
    region         = "<aws-region>"
    dynamodb_table = "<your-dynamodb-table-name>"
  }
}

NOTE: Replace <your-s3-bucket-name> with the name of the S3 bucket you created in step 1, <aws-region> with the appropriate AWS region, and <your-dynamodb-table-name> with the name of the DynamoDB table you created.

Initialize the backend:

Open a terminal or command prompt in the directory containing your Terraform configuration files.
Run terraform init to initialize the new backend configuration and download any necessary provider plugins.

Migrate the existing state:

(Hashicorp Help Center Docs)

If you have an existing Terraform state in Terraform Cloud, you’ll need to download it and migrate it to the new backend.
Use the terraform state pull command to download the current state from Terraform Cloud.

Run terraform init -reconfigure to reconfigure Terraform with the new backend and upload the state to S3 and DynamoDB.

Test the new backend

Run any Terraform commands (e.g., terraform plan, terraform apply) to ensure the new backend is working correctly.
Verify that the state files are stored in the S3 bucket and that the DynamoDB table is being used for state locking.

Clean up:

If you’re confident that the migration was successful, you can remove the old Terraform Cloud backend configuration and associated state files.

Migrate your Terraform CI/CD:

There are plenty of options for cheaper alternatives to Terraform Cloud — This article on Terraform automation and collaboration software speaks about it in detail.

There are some DIY Github Actions that can be added as steps to your own workflow files. Here is an example of one such tool.

Disclosure: We’re building Digger — an open source GitOps tool for Terraform that reuses the async jobs infrastructure with compute, orchestration, logs, etc of your existing CI/CD. Do feel free to try it out!