DEV Community: Thulasiraj Komminar

Understanding AWS Lambda Event Source Mapping

Thulasiraj Komminar — Fri, 27 Dec 2024 20:33:48 +0000

Attending AWS re:Invent is always a highlight for me, and this year’s session, SVS407-R: Understanding AWS Lambda Event Source Mapping, turned out to be a goldmine of insights. As someone who builds and maintains event-driven architectures, I walked away with a deeper understanding of the complexities and best practices around Event Source Mapping (ESM), particularly in handling event streams from services like Amazon Kinesis, DynamoDB Streams, and Amazon SQS.

The Fundamentals: Event Source Mapping in AWS Lambda

To kick things off, the speaker provided an overview of how AWS Lambda integrates with event sources through ESM. Event source mapping ensures that Lambda functions are automatically triggered as new events arrive, making it a crucial component for real-time and asynchronous workloads. The session quickly moved into advanced topics, focusing on optimizing performance, ensuring fault tolerance, and managing error scenarios.

Partition Key Management: The Core of Event Order

One of the key concepts covered was partition key management in systems like Amazon Kinesis and DynamoDB Streams. These partition keys play a vital role in ensuring order and scalability:

Sequential Processing Within Shards:

Events with the same partition key are processed sequentially within their shard. This guarantees that related events (e.g., all transactions for a user) are handled in order. However, if retries or failures cause events to arrive out of order, consistency issues can arise.

Shard Behavior:

Shards are responsible for processing sets of partition keys, and they scale horizontally to handle increased event volumes. Even as the system scales, the partition key ensures that related events are processed within the same shard, preserving order.

This deep dive reinforced the importance of designing systems with careful consideration of partition key selection to balance performance and maintain order.

Error Handling in Distributed Systems: Preventing Bottlenecks

Error handling emerged as a critical focus area in the session. Distributed systems, especially those processing real-time event streams, require robust strategies to handle failures without cascading effects. Key insights included:

Infinite Retries and Dead Letter Queues (DLQs):

Without proper error handling, a shard encountering persistent errors could end up in an infinite retry loop, stalling other events in the shard. Routing unresolvable errors to DLQs, such as SQS or S3, ensures failed events are isolated and can be analyzed or resolved manually.

Failure Isolation:

Kinesis-specific event source mappings allow retrying failed records individually or in smaller batches. This prevents an entire shard from stalling due to one problematic record.

These practices are invaluable for maintaining the reliability of event-driven systems while ensuring that errors are handled gracefully.

Performance Optimization: Parallelization, Batch Processing, and Scaling

The session also addressed parallelization and batch processing, two critical levers for optimizing performance in Lambda-based architectures:

Parallelization Factor:

The parallelization factor controls how many workers process events from a single shard concurrently. While a higher factor improves throughput, it can introduce challenges in maintaining event order across partition keys. The speaker emphasized tuning this setting to match workload complexity and downstream capacity.

Batch Processing and Error Mitigation:

Features like bisect batch on error allow Lambda to split failed batches into smaller subsets, isolating problematic records for retries. This reduces redundant retries and improves system efficiency.Additionally, batch item failure reporting lets Lambda identify specific records in a failed batch, ensuring only those records are retried or sent to a DLQ.

Kinesis-Specific Features: A Competitive Edge

The session highlighted several features unique to Amazon Kinesis that make it particularly powerful for event-driven architectures:

Error Recovery:

With tools like checkpointing and offset management, Kinesis ensures graceful recovery from errors without reprocessing already-successful events.

Batch Item Failure:

This feature complements bisect batch by isolating failed records within a batch, making retries even more precise.

The speaker contrasted these capabilities with Kafka, noting that while Kafka offers flexibility, it lacks the granular error-handling features natively provided by Kinesis. Developers using Kafka often have to build custom solutions for similar functionality, increasing complexity.

Scaling Challenges: Balancing Upstream and Downstream Loads

Scaling event-driven architectures can be tricky, especially when traffic spikes or downstream systems face capacity limits. Key points included:

Handling Traffic Spikes:

Lambda and Kinesis scale efficiently to handle spikes, but parameters like shard count, parallelization factor, and batch size need fine-tuning.

Downstream System Limits:

Scaling upstream services is relatively easy, but downstream systems like databases or APIs often have rate limits. Strategies like throttling, backpressure mechanisms, and pre-filtering events can prevent overwhelming these systems.

Record Filters:

Pre-filtering events at the Kinesis or Lambda level ensures only relevant events are processed, reducing unnecessary load on downstream components.

Best Practices: Building Robust Event-Driven Systems

To wrap up the session, the speaker shared actionable best practices for building and maintaining event-driven architectures:

Set Up DLQs and Monitoring:
Always configure DLQs for unresolved errors and monitor them for analysis. Use alarms to track error rates and address issues proactively.
Tune Parallelization Factor:
Match the parallelization factor to your workload complexity and downstream capacity to optimize throughput.
Monitor Metrics:
Regularly monitor metrics for Lambda, Kinesis, and downstream systems to identify bottlenecks and failure patterns.
Batch Size Optimization:
Adjust batch sizes to balance latency, throughput, and error isolation based on workload characteristics.
Graceful Error Handling:
Leverage features like bisect batch on error, batch item failure, and retry strategies to handle errors without disrupting the entire pipeline.

Final Thoughts

The SVS407-R session at AWS re:Invent was a deep dive into the intricacies of AWS Lambda event source mapping. From partition key management and error handling to performance optimization and scaling strategies, the session provided a wealth of practical knowledge for building robust, scalable event-driven systems.

One of the biggest takeaways for me was the importance of balancing throughput, order guarantees, and error isolation. Whether you’re processing billions of events with Kinesis or handling asynchronous workflows with SQS, these strategies can make all the difference in ensuring system reliability and efficiency.

AWS re:Invent continues to deliver world-class learning experiences, and this session was no exception. I’m excited to apply these insights to my projects and see the impact firsthand. If you’re building event-driven applications, I highly recommend exploring these features and best practices.

References

Terraform Stacks

Thulasiraj Komminar — Wed, 30 Oct 2024 20:01:42 +0000

Ever since the private preview of Terraform Stacks, I’ve been eager to dive in and explore this new approach to provisioning infrastructure. After a year in private preview, the public beta was finally announced at HashiConf 2024, and I’m excited to share my experience!

In this blog, I’ll walk you through the essentials of Terraform Stacks configurations and demonstrate how to deploy a REST API as an S3 proxy using API Gateway.

What is a Stack?

Terraform excels at planning and applying changes to individual resources, but it has historically lacked a built-in solution for consistently deploying identical infrastructure across multiple environments or cloud regions. To work around this, many of us developed our own methods. Personally, I’ve used workspaces for environment-specific configurations and duplicated code with different providers for multi-region setups. However, this approach sacrifices the DRY (Don’t Repeat Yourself) principle and requires stitching together dependencies manually, complicating state management and orchestration.

Terraform Stacks are designed to streamline the coordination, deployment, and lifecycle management of complex, interdependent configurations. With Stacks, we can easily replicate infrastructure across environments and regions, set orchestration rules, and automate the propagation of changes, drastically reducing time and operational overhead.

In essence, Stacks address the “bigger picture” of infrastructure provisioning, offering a scalable solution to manage consistent, repeatable deployments across environments and regions.

*Continue to read this blog * here

Data Governance on AWS using DataZone

Thulasiraj Komminar — Thu, 08 Aug 2024 23:08:25 +0000

Introduction

In this blog, we will provide a brief introduction to data governance and show how to implement it on AWS using DataZone. We will walk through a practical example involving a multi-account setup to manage and share data stored in S3 and Redshift, highlighting key steps and best practices along the way.

What is Data Governance?

Data governance is everything you do to ensure data is secure, private, accurate, available, and usable that helps organizations accelerate data-driven decisions. Key steps in implementing data governance typically include:

Data mapping and classification. Data mapping involves documenting data assets and understanding how data flows through an organization’s systems. This process enables the identification of different data sets, which can then be classified based on various factors, such as whether they contain personal information or other sensitive data. These classifications directly influence the application of data governance policies to each data set, ensuring appropriate levels of security and compliance.
Business glossary. A business glossary provides standardized definitions for business terms and concepts used within an organization. For instance, it might define what constitutes an active customer. By establishing a common vocabulary for business data, a business glossary supports consistent understanding and interpretation, which is crucial for effective governance across the organization.
Data catalog. A data catalog is a comprehensive, indexed inventory of an organization’s data assets, created by collecting metadata from various systems. It typically includes details on data lineage, search functionalities, and collaboration tools. Data catalogs often integrate information about data governance policies and offer automated mechanisms for policy enforcement, ensuring that data is used in accordance with established governance standards.

What is AWS DataZone?

DataZone is a data management service that makes it faster and easier for customers to catalog, discover, share, and govern data stored across AWS accounts. With DataZone, administrators and data stewards who oversee an organization’s data assets can manage and govern access to data using fine-grained controls. These controls are designed to ensure access with the right level of privileges and context. DataZone makes it easier for engineers, data scientists, product managers, analysts, and business users to access data throughout an organization so that they can discover, use, and collaborate to derive data-driven insights.

Key concepts and Capabilities:

Data Portal This is a web application where different users can go to catalog, discover, govern, share, and analyze data in a self-service fashion.
Business Data Catalog In your catalog, you can define the taxonomy or the business glossary. You can use this component to catalog data across your organization with business context and thus enable everyone in your organization to ﬁnd and understand data quickly.
Data Projects & Environments You can use projects to simplify access to the AWS analytics by creating business use case–based groupings of people, data assets, and analytics tools. DataZone projects provide a space where project members can collaborate, exchange data, and share data assets. Within projects, you can create environments that provide the necessary infrastructure to project members such as analytics tools and storage so that project members can easily produce new data or consume data they have access to.
Governance and Access Control You can use built-in workflows that allow users across the organization to request access to data in the catalog and owners of the data to review and approve those subscription requests. Once a subscription request is approved, DataZone can automatically grant access by managing permission at underlying data stores such as Lake Formation and Redshift.

Setting up DataZone

Prerequisites

Set up Redshift Serverless clusters in both the Producer and Consumer accounts. This is essential for enabling database sharing across accounts.

Now that we’ve explored the fundamentals of data governance and the key concepts of DataZone, let’s move forward with the setup process.

Architecture

DataZone Account

In an organization, a central Data Team is typically responsible for setting up and managing the data marketplace using DataZone. Their key responsibilities include:

Building Security Controls : Implementing robust security measures to protect data assets.
Developing and Operating the Platform : Ensuring the data marketplace is built and maintained for seamless operation.
Simplifying Onboarding : Facilitating the onboarding process by creating training materials and actively engaging with the community.

To begin setting up a Data Marketplace, the first step is to create a domain.

Create DataZone Domain:

Login to the AWS Console as an administrator user, then navigate to Amazon DataZone.
Click on the Create domain button.

On the Create domain page, provide values for the following fields:
Domain Name: The name of the domain.
Description: This is an optional field where you can provide a description for this root domain.
Service access: DataZone will create a new service role for you if there is no suitable role exists in your AWS account. This role includes necessary permissions that authorize DataZone to make API calls on behalf of users within the domain.

Quick set up: Select this option as it automatically creates required service roles and S3 bucket required for DataZone.

Click on Create domain. Once the domain is created click Open data portal.

Associate Producer and Consumer accounts:

Associating your AWS accounts with DataZone domains enables you to publish data from these AWS accounts into the DataZone catalog and create DataZone projects to work with your data across multiple AWS accounts. Now the Data Portal is ready lets associate the producer and the consumer accounts.

Go to the domains, click on the Account associations and click Request association
Enter the Producer account id and click Request association. Repeat this step to associate the Consumer account.

Create Projects:

Projects enable a group of users to collaborate on various business use cases that involve publishing, discovering, subscribing to, and consuming data assets in the Amazon DataZone catalog. We will create projects for producer and consumer.

Log in to the Data Portal and navigate to the Create Project option. Enter a name for your project and click Create. Repeat this process to create the Consumer project as well.

With the projects set up and the Producer and Consumer accounts linked, the next step is to enable blueprints in both the Producer and Consumer accounts. Once this is done, we’ll return to the DataZone account to create environments, establish a business catalog, and then publish and subscribe to data assets.

Create Environments:

In DataZone projects, environments are defined as collections of configured resources — such as S3 buckets, Glue databases, or Athena workgroups — each associated with a specific set of IAM principals (user roles) who are granted owner or contributor permissions to manage those resources.

In our setup, we will create two environments: one for Athena (linked with S3) and another for Redshift.

Athena Environment:

Log in to the Data Portal, select the Producer project, and then navigate to the Environments tab. Click on Create Environment to proceed.

Enter the name and choose DataLakeProfile as environment profile and click Create Environment.
Enter a name for the environment, select DataLakeProfile as the environment profile, and click Create Environment.

Redshift Environment:

Navigate to the Environments tab and click Create Environment Profile. Enter a name for the profile and select Default Data Warehouse as the blueprint.

Under the Parameter set section, select the parameter set you created earlier.

Click Create Environment Profile to complete the setup.

Producer and Consumer Account

Enable Blueprints:

A blueprint with which the environment is created defines what AWS tools and services (eg, Glue or Redshift) members of the project to which the environment belongs can use as they work with assets in the DataZone catalog.

Go to DataZone service click View Associated Domain and under the Blueprints tab enable Default Data Lake and Default Data Warehouse blueprints.

Create Parameter Set for Redshift:

Parameter set is a group of keys and values, required for DataZone to establish a connection to your Redshift cluster and is used to create data warehouse environments. These parameters include the name of your Redshift cluster, database, and the secrets manager that holds credentials to the cluster.

Click on the Default Data Warehouse blueprints and under the Parameter sets click Create parameter set. Enter the name, description and choose the region.

Select either Amazon Redshift cluster or Amazon Redshift Serverless.
Select the secrets manager ARN that holds the credentials to the selected Redshift cluster or the Redshift Serverless workgroup. The AWS secret must be tagged with the AmazonDataZoneDomain : [Domain_ID] tag in order to be eligible for use within a parameter set.
Click Create parameter set.

Make sure the Redshift Manage Access Role has permissions to read the secret.

Repeat the above steps in the Consumer account to mirror the setup.

Publishing & Cataloging data product

Prerequisites

For this tutorial create two datasets in the Producer account.

Claims: Create a dataset in S3 that contains information on insurance claims filed. Additionally, use the Glue Data Catalog to catalog this data.

Customer: Create a dataset in Redshift that includes personal information and relevant details about customers.

With the projects and environments now created, we can proceed to import existing data, catalog it, and ultimately publish it and then will make the data easy to understand with business glossary and business name generation.

Publish Claims data

Log in to the Data Portal, navigate to the Data Sources tab, and click on Create Data Source.
Enter a name for the data source, select AWS Glue as the type (since the claims data is stored in S3), and choose the environment you just created.

Enter the name of the Glue database, then proceed through the subsequent tabs with the default settings. Click Create to finalize the setup.

Once the Data Source is created, initiate it to allow DataZone to import the Glue catalog into the Data Portal. After the process completes, you will see the imported asset in the portal.

Click on the asset to verify its details. The tables created have technical column names. DataZone provides a feature called Business Name Generation , which converts technical names into more user-friendly business data names. Review these generated names, and if they meet your expectations, click Accept All to apply the business data names automatically.
Business Glossary: In DataZone, a business glossary is a curated collection of business terms designed to enhance the clarity and understanding of data assets. These glossaries are created within the catalog’s root domain and can be linked to metadata forms across various domains. A business glossary can either be a flat list of terms or a hierarchical structure, where any term can have an associated sublist of related terms. This feature ensures that data is easier to interpret and aligns with business concepts.
If needed, you can create and attach Business Glossary terms to the data asset. Once everything is in place, click Publish Asset to make the data asset available for consumption.

Publish Customer data

Navigate to the Data Sources tab, create a new data source and choose Amazon Redshift as the type.

Click Next , enter the appropriate schema name, and leave the remaining settings as default. Click Create and then run the data source. Once the run is complete, you should see the imported asset. Click on the asset and then select Publish to finalize the process.

Having published both datasets, the next step is to subscribe from the Consumer account. Once subscribed, you can analyze the data using Athena and Redshift.

Discovering and subscribing data product

Data Consumer searches for data and discovers the data needed for the business use case. They also request access to the data through data subscription. Once the Data Product Owner approves the subscription, the data asset is available for use by the Data Analyst.

Claims data

Create subscription:

Use the search bar to locate the claims data and click Subscribe.

Navigate to the Producer project, where a notification for the subscription request should appear. Click View Request to review the details and approve it. You can choose to grant full access to the data or apply column/row filters as needed.

Analyze and Visualize data in Athena:

Return to the Consumer project, where you should now see the subscribed asset listed.

Click on the subscribed asset, then go to the My Subscription tab and select Query Data to view the data.

Customer data

Create subscription:

Repeat the process to search for and subscribe to the customer data. Switch back to the Producer project to approve the request. Once approved, DataZone will add the asset to the existing environment.

Analyze and Visualize data in Redshift:

Click Query Data to run queries on the Customer data in Redshift.

For authentication, select Federated User and use dev for the database. You should see a datazone_ schema with a view created under that schema.

Conclusion

This blog covered setting up data governance with AWS DataZone, including creating datasets, configuring environments, and managing data access. With these steps, you can now efficiently manage and analyze data across your organization, enhancing data-driven decision-making.

How to Migrate Amazon Redshift to a Different Account and Region: Step-by-Step Guide

Thulasiraj Komminar — Thu, 04 Jul 2024 16:01:33 +0000

Introduction

Moving Amazon Redshift to a new account and region might seem difficult, but it doesn’t have to be. You might need to follow regulations or reorganize your teams. In this guide, we will show you step by step how to move your Redshift data to a different account and region. After reading this guide, you will know how to do Redshift migrations easily, with minimal downtime and secure data. Let’s start and make your Redshift move simple!

Prerequisites

For the migration process, choose a maintenance window with minimal write activity, ensuring alignment with the organization’s RTO and RPO requirements.

Step 1: Configure cross-region snapshot

To move the cluster to a different region in a different account, you first need to configure the cross-region snapshot for the cluster in the source account where the cluster resides.

Go to your cluster and click Actions.
Select Configure cross-region snapshot.
In the Destination AWS Region drop-down menu, choose the region where you want to move the cluster in the target account.
Click Save.

Step 2: Create a manual snapshot

To share a cluster snapshot with another AWS account, you need a manual snapshot.

Go to your cluster and click Actions.
Choose Create Snapshot.
Give the snapshot a name and click Create Snapshot.

Since we configured the cross-region snapshot in the previous step, creating a snapshot now will also copy it to the destination region.

Step 3: Grant Access to KMS Key in the Target Account

When you share an encrypted snapshot, you also need to share the KMS key that was used to encrypt it. To do this, add the following policy to the KMS key. In this policy example, replace 123456789123 with the identifier of the TargetAccount.

{
    "Id": "key-policy-1",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Allow use of the key in TargetAccount",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::123456789123:root"
                ]
            },
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}

Step 4: Share snapshot

Navigate to Snapshots, find the manual snapshot you created, and click on it. Under Snapshot access, click on edit. Enter the TargetAccount ID and click Add account. Once you’re done, click Save. Now, the snapshot will be accessible in the TargetAccount and the destination region..

Step 5: Restoring a Cluster from Snapshot in the Target Account

Navigate to the TargetAccount and the destination region in Redshift. Under snapshots, you will find the shared snapshot. Click on Restore snapshot and configure options like Nodes, Networking, and more as needed.

Conclusion

In this guide, we’ve covered the essential steps to migrate Amazon Redshift to a new account and region smoothly. By following these steps carefully, you can ensure minimal downtime and maintain data integrity throughout the migration process. I hope this guide has provided clarity and confidence for your Redshift migration journey!

References:

Streamline SSO Access to AWS Redshift Query Editor with Okta and Terraform

Thulasiraj Komminar — Sat, 18 May 2024 08:41:26 +0000

Introduction

AWS Redshift Query Editor v2 is a web-based tool that allows analysts to securely explore, share, and collaborate on data using SQL within a common notebook interface. It simplifies querying data with SQL and visualizing results with charts and graphs in just a few clicks.

Integrating Redshift Query Editor v2 with your identity provider (IdP) automatically redirects users to the Query Editor v2 console instead of the Amazon Redshift console. This setup enables seamless access to Amazon Redshift clusters via federated credentials, eliminating the need to manage individual database users and passwords.

In this blog post, we’ll focus on using Okta as the IdP and guide you through configuring your Okta application and AWS IAM permissions. We’ll also demonstrate how to restrict user access to only the Query Editor v2, preventing them from performing administrative functions on the AWS Management Console.

Prerequisites

Before you begin, ensure you have the following prerequisites in place:

An Okta Account : You need an active Okta account to serve as your identity provider.
A Redshift Cluster : Ensure you have an existing Amazon Redshift cluster set up.
Configured Query Editor v2 : Make sure Redshift Query Editor v2 is configured. For more details, refer to Configuring your AWS account.

Create IAM Roles and Permissions

To enable Okta to access Amazon Redshift Query Editor v2, you need to create two IAM roles. The first role will be used by Okta to access the Redshift Query Editor, and the second role will establish a trust relationship between the IdP (Okta) and AWS. We’ll start by creating the role that Okta uses to access Amazon Redshift Query Editor v2. After setting up the Okta application, we’ll create the trust relationship role using the metadata from the Okta app.

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

data "aws_iam_policy_document" "assume_policy" {
  statement {
    effect = "Allow"
    actions = [
      "sts:AssumeRoleWithSAML",
      "sts:TagSession",
    ]

    principals {
      type = "Federated"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:saml-provider/OktaRedshiftFederation"]
    }

    condition {
      test = "StringEquals"
      variable = "SAML:aud"
      values = ["https://signin.aws.amazon.com/saml"]
    }
  }
}

data "aws_iam_policy_document" "default" {
  statement {
    actions = [
      "redshift:GetClusterCredentials",
      "redshift:CreateClusterUser",
      "redshift:JoinGroup",
    ]
    resources = [
      "arn:aws:redshift:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:cluster:${local.redshift_cluster_id}",
      "arn:aws:redshift:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:dbuser:${local.redshift_cluster_id}/*",
      "arn:aws:redshift:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:dbgroup:${local.redshift_cluster_id}/admin",
      "arn:aws:redshift:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:dbgroup:${local.redshift_cluster_id}/datascientist",
      "arn:aws:redshift:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:dbname:${local.redshift_cluster_id}/*",
    ]
  }
}

resource "aws_iam_role" "default" {
  name = "RedshiftQueryEditorFederation"
  assume_role_policy = data.aws_iam_policy_document.assume_policy.json

  tags = {
    environment = "dev"
    stack = "redshift"
  }
}

resource "aws_iam_role_policy" "default" {
  name = "RedshiftQueryEditorFederation"
  role = aws_iam_role.default.id
  policy = data.aws_iam_policy_document.default.json
}

resource "aws_iam_role_policy_attachment" "default" {
  role = aws_iam_role.default.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonRedshiftQueryEditorV2ReadSharing"
}

Set Up Okta Application

With the IAM role for Okta access created, configure the Okta SAML application and assign it to the necessary Okta groups.

data "okta_group" "admin" {
  name = "admin"
}

data "okta_group" "datascientist" {
  name = "datascientist"
}

resource "okta_app_saml" "redshift" {
  label = "Redshift Query Editor v2"
  logo = "${path.module}/images/redshift.png"
  preconfigured_app = "amazon_aws"
  saml_version = "2.0"
  default_relay_state = "https://${data.aws_region.current.name}.console.aws.amazon.com/sqlworkbench/home"

  app_settings_json = jsonencode({
    "awsEnvironmentType" : "aws.amazon",
    "appFilter" : "okta",
    "groupFilter" : "(?{{role}}[a-zA-Z0-9+=,.@\\-_]+)",
    "joinAllRoles" : false,
    "loginURL" : "https://console.aws.amazon.com/ec2/home",
    "roleValuePattern" : "arn:aws:iam::${data.aws_caller_identity.current.account_id}:saml-provider/OktaRedshiftFederation,${aws_iam_role.default.arn}",
    "sessionDuration" : 3600,
    "useGroupMapping" : true,
  })

  attribute_statements {
    type = "EXPRESSION"
    name = "https://aws.amazon.com/SAML/Attributes/PrincipalTag:RedshiftDbUser"
    namespace = "urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"
    values = ["user.username"]
  }

  attribute_statements {
    type = "EXPRESSION"
    name = "https://aws.amazon.com/SAML/Attributes/PrincipalTag:RedshiftDbGroups"
    namespace = "urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"
    values = ["String.join(\":\", isMemberOfGroupName(\"datascientist\") ? 'datascientist' : '',isMemberOfGroupName(\"admin\") ? 'admin' : '')"]
  }
}

resource "okta_app_group_assignments" "redshift" {
  app_id = okta_app_saml.redshift.id

  group {
    id = data.okta_group.datascientist.id
  }

  group {
    id = data.okta_group.admin.id
  }
}

Create IAM SAML Provider Role

After setting up the Okta application, create the IAM SAML provider to establish a trust relationship between Okta and AWS using the Okta metadata.

resource "aws_iam_saml_provider" "redshift_okta" {
  name = "OktaRedshiftFederation"
  saml_metadata_document = okta_app_saml.redshift.metadata

  tags = {
    environment = "dev"
    stack = "redshift"
  }
}

Conclusion

In this blog post, we demonstrated how to securely federate SSO access to AWS Redshift Query Editor v2 using Okta as the identity provider, leveraging Terraform for seamless infrastructure management. By creating the necessary IAM roles and configuring the Okta SAML application, we established a robust trust relationship between Okta and AWS. This setup not only simplifies user access to Redshift Query Editor v2 but also enhances security by eliminating the need to share database users credentials. With this integration, your teams can efficiently explore, share, and collaborate on data, driving insightful decisions and streamlined operations.

Enabling Cross-Account Access for AWS Lake Formation with Data Filters Using Terraform

Thulasiraj Komminar — Fri, 17 May 2024 20:34:26 +0000

Introduction

In my previous blog, we explored enabling cross-account data sharing using AWS Lake Formation with Terraform. In this post, we’ll dive deeper into enhancing that setup with data filters. Lake Formation data filtering allows for column-level, row-level, and cell-level security. This blog will focus specifically on implementing cell-level security to fine-tune data access controls.

Designed by Freepik

Data filter levels

Column-level

Granting permissions on a Data Catalog table with column-level filtering allows users to view only specific columns and nested columns they have access to. This enables defining security policies that grant access to partial sub-structures of nested columns.

Row-level

Granting permissions on a Data Catalog table with row-level filtering allows users to view only specific rows of data they have access to. Filtering is based on the values of one or more columns, and nested column structures can be included in row-filter expressions.

Cell-level

Cell-level security combines both row and column filtering, providing a highly flexible permissions model.

Creating a Data Filter in the Source Account

Assuming you have already followed the Lake Formation setup in Source Account as detailed in my previous blog, we can now proceed with creating the data filter. Let’s use an example involving IIoT measurements. Suppose you have equipment spread across multiple sites and need to grant specific IAM roles access to particular sites and columns. Here’s how to achieve this using Terraform:

In this example:

Define Local Configuration: The filter_config variable lists the sites, columns, and IAM roles in the target account that need access.
Retrieve AWS Account ID: The aws_caller_identity data source fetches the current AWS account ID.
Create Data Cell Filters: The aws_lakeformation_data_cells_filter resource iterates over the filter_config to create the necessary filters for each IAM role.

This setup ensures that specific IAM roles have access only to the defined sites and columns, enhancing security and data management.

Share Catalog with Target Account

Now that we’ve created the data filter, let’s utilize it while sharing the catalog. In the code snippet below, we’ll share the database and table with the target account. Note that when sharing the table, we’ll include the data filter created in the previous step.

In this snippet:

Share Database Permissions: The aws_lakeformation_permissions resource shares the IIoTDataLake database with the target account and grants the DESCRIBE permission.
Share Table Permissions: Similarly, the resource shares the measurements table with the target account, granting the SELECT permission. It also includes the data filter created earlier, ensuring that the target account only accesses the filtered data according to the defined criteria.

With this setup, you can securely share specific data from your catalog with the target account, ensuring compliance and data integrity.

Creating Resource Link in Target Account for Access

After sharing the catalog and table with a data filter to the target account, let’s proceed to the target account to establish a resource link for accessing the shared catalog data.

In this setup:

Create Resource Link: The aws_glue_catalog_database resource establishes a database resource link named IIoTDataLake-Target in the target account. It links to the IIoTDataLake database in the source account, enabling access to the shared catalog data.

By creating this resource link, you enable seamless access to the shared data catalog from the target account, facilitating data utilization and analysis across accounts while maintaining security and compliance measures.

Granting Permissions for IAM Roles

Now that we’ve created the resource link, we can grant access to the resource link and the shared catalog. After this step, the IAM roles will have access to the filtered data shared from the source account.

In this configuration:

Grant Database Permissions: The aws_lakeformation_permissions resource grants the DESCRIBE permission to the IAM roles for the IIoTDataLake-Target database in the target account. This allows the roles to describe the database structure and metadata.
Grant Table Permissions: Similarly, the resource grants the SELECT permission to the IAM roles for the measurements table in the shared catalog. This enables the roles to select and read data from the table.

With these permissions granted, the IAM roles now have access to the filtered data shared from the source account, allowing for seamless data analysis and utilization within the target account.

Conclusion

In this blog, we’ve delved into the intricacies of cross-account data sharing using AWS Lake Formation and Terraform. By implementing data filters and establishing resource links, we’ve ensured secure access to shared data while maintaining granular control over permissions. This streamlined approach facilitates collaborative data analysis across accounts, empowering teams to derive insights effectively while upholding data security and compliance standards.

References

Data filtering and cell-level security in Lake Formation — AWS Lake Formation (amazon.com)

How to Write Data to InfluxDB v3 with GoLang: A Step-by-Step Tutorial

Thulasiraj Komminar — Tue, 14 May 2024 15:04:59 +0000

Introduction

In this blog post, we’ll delve into the process of writing data into InfluxDB v3 using Go. Our focus will be on harnessing the capabilities of the influxdb3-go client library, particularly its support for annotated structs. Through a practical example, we’ll demonstrate how to convert a slice into a structured format and efficiently batch write it into InfluxDB.

Setting Up a Go Project for IIoT Measurements: A Step-by-Step Guide

Let’s kickstart by initializing a Go project named iiot-measurements and structuring it with three essential files:

main.go: Responsible for the main function.
influxdb3.go: Contains the struct and functions for InfluxDB writing.
config.go: Handles the loading of InfluxDB credentials from the environment.

Here’s how to set up the project:

mkdir iiot-measurements && cd iiot-measurements && go mod init iiot-measurements

Next, let’s install the latest influxdb3 Go client:

go get github.com/InfluxCommunity/influxdb3-go

Implementing Annotated Structs for InfluxDB Data Writing

With the influxdb3-go client, data can be provided in various formats such as line protocol, point, or as a struct. For this tutorial, we’ll opt for the annotated struct approach to ensure a fixed schema. This choice suits scenarios where maintaining a consistent schema is crucial. However, if your use case involves handling dynamic schemas, other options like line protocol or point may be more suitable.

Loading InfluxDB Credentials from Environment Variables

Before instantiating the InfluxDB client, it’s essential to load the credentials from the environment. To achieve this, we’ll utilize the env package to parse environment variables into our InfluxdbConfig struct. Additionally, we’ll create a NewConfig() function to facilitate the instantiation of the struct.

go get github.com/caarlos0/env/v11

Instantiating the InfluxDB Client

After loading the credentials, the next step is to create a function that instantiates the InfluxDB client. We’ll define a NewClient() function responsible for this task, utilizing the configuration struct previously defined. This function ensures seamless creation of the client with the provided configuration.

Implementing Batch Write Function for InfluxDB Data

After setting up the InfluxDB client instantiation, the next step is to implement a function for batch writing data to InfluxDB v3. We’ll create a function named BatchWrite() responsible for this task. This function will take a slice of annotated Measurement structs and convert them into an []interface{} format, facilitating batch writing to InfluxDB v3.

Writing Data to InfluxDB v3

With all the required functions in place, we are now ready to instantiate the client and write a slice of data to InfluxDB v3. First, we’ll initialize the client using the configuration obtained from environment variables. Then, we’ll utilize the BatchWrite() function to efficiently write the data as a batch to InfluxDB.

Exploring Data

Conclusion

In this tutorial, we’ve explored the process of writing data to InfluxDB v3 using GoLang and the influxdb3-go client library. By leveraging annotated structs and environment-based configuration loading, we’ve established a robust foundation for efficiently managing InfluxDB data within Go applications.

We began by setting up a Go project and initializing the InfluxDB client with credentials loaded from environment variables. Through the implementation of a BatchWrite() function, we demonstrated how to streamline the process of writing data in batches to InfluxDB v3.

By following the steps outlined in this tutorial, you’re now equipped with the knowledge to seamlessly integrate InfluxDB data writing capabilities into your Go applications, ensuring reliability and scalability for your time-series data management needs.

References

https://github.com/komminarlabs/iiot-measurements-influxdb3

influxdb3-go/README.md at main · InfluxCommunity/influxdb3-go (github.com)

Line protocol | InfluxDB Cloud Serverless Documentation (influxdata.com)

How to bridge a Terraform Provider to Pulumi

Thulasiraj Komminar — Fri, 15 Mar 2024 23:49:16 +0000

Introduction:

In this blog post, we’ll delve into the process of creating a Pulumi Resource provider sourced from a Terraform Provider developed with the Terraform Plugin Framework. Our focus will be on leveraging the bridge package to facilitate this transition seamlessly. To illustrate, we’ll demonstrate bridging the InfluxDB Terraform provider to Pulumi.

Prerequisites:

Before delving further, it’s essential to possess a foundational understanding of Terraform providers, Pulumi resource providers, and InfluxDB.

Ensure the following tools are installed and present in your %PATH:

Why Bridge a Terraform Provider to Pulumi:

The mature and vibrant Terraform Providers ecosystem benefits from contributions by numerous industry leaders in cloud and infrastructure. By bridging Terraform Providers to Pulumi, organizations gain access to reliable and battle-tested infrastructure management capabilities.

How the bridge works:

The bridge operates across two significant phases: design-time and runtime.

During design-time, the bridge meticulously examines the schema of a Terraform Provider. However, it’s important to note that this process is applicable solely to providers constructed with static schemas.

Moving on to the runtime phase, the bridge establishes a connection between the Pulumi engine and the designated Terraform Provider by harnessing Pulumi’s RPC interfaces. This interaction heavily relies on the Terraform provider schema, facilitating tasks such as validation and the computation of differences.

How to bridge a provider:

Pulumi provides two options for initializing your project: you can either utilize the template repository offered by Pulumi, or opt for the community-supported cookiecutter template.

If you choose the cookiecutter template, setting up an initial version is straightforward — just specify a few configuration settings. However, if you prefer the template repository route, follow the steps outlined below to get started:

To begin, navigate to the template repository and select Use this template.
Then, ensure the following options are configured:

Owner : Your GitHub organization or username.
Repository name : Preface your repository name with pulumi as per standard practice. For instance, pulumi-influxdb.
Description : Provide a brief description of your provider.
Repository type : Set it to Public.

After configuring these options, proceed to clone the generated repository.
Execute the following command to update the files, replacing placeholders with the name of your provider.

make prepare NAME=influxdb REPOSITORY=github.com/komminarlabs/pulumi-influxdb

This will do the following:

Rename folders in provider/cmd to pulumi-resource-influxdb and pulumi-tfgen-influxdb.
Replace dependencies in provider/go.mod to reflect your repository name.
Find and replace all instances of the boilerplate xyz with the NAME of your provider.

Ensure to accurately set your GitHub organization or username in all files where your provider is referenced as a dependency.

examples/go.mod
provider/resources.go
sdk/go.mod
provider/cmd/pulumi-resource-influxdb/main.go
provider/cmd/pulumi-tfgen-influxdb/main.go

Create a Shim

Although the New() provider function resides within an internal package, referencing it in an external Go project isn’t straightforward. However, it’s still achievable through Go linker techniques.

Create a provider/shim directory.

mkdir provider/shim

Add a go.mod file with the following content.

module github.com/komminarlabs/terraform-provider-influxdb/shim

go 1.22

require (
 github.com/hashicorp/terraform-plugin-framework v1.6.0
 github.com/komminarlabs/terraform-provider-influxdb v1.0.1
)

Add a shim.go file with the following content.

package shim

import (
 tfpf "github.com/hashicorp/terraform-plugin-framework/provider"
 "github.com/komminarlabs/terraform-provider-influxdb/internal/provider"
)

func NewProvider() tfpf.Provider {
 return provider.New("dev")()
}

Import the New Shim Provider

In provider/resources.go import the shim package.

package influxdb

import (
 "fmt"
 "path"

 // Allow embedding bridge-metadata.json in the provider.
 _ "embed"

 influxdbshim "github.com/komminarlabs/terraform-provider-influxdb/shim"

 pf "github.com/pulumi/pulumi-terraform-bridge/pf/tfbridge"
 "github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge"
 "github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge/tokens"
 shim "github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim"
 "github.com/pulumi/pulumi/sdk/v3/go/common/resource"

 // Import custom shim
 "github.com/komminarlabs/pulumi-influxdb/provider/pkg/version"
)

Instantiate the Shim Provider

In provider/resources.go, replace shimv2.NewProvider(influxdb.Provider()) with pf.ShimProvider(influxdbshim.NewProvider()):

func Provider() tfbridge.ProviderInfo {
 prov := tfbridge.ProviderInfo{
  // Instantiate the Terraform provider
  P: pf.ShimProvider(influxdbshim.NewProvider()),
}

Edit provider/go.mod and add github.com/komminarlabs/terraform-provider-influxdb/shim v0.0.0 to the requirements.

module github.com/komminarlabs/pulumi-influxdb/provider

go 1.22

replace (
 github.com/hashicorp/terraform-plugin-sdk/v2 => github.com/pulumi/terraform-plugin-sdk/v2 v2.0.0-20240202163305-e2a20ae13ef9
 github.com/komminarlabs/terraform-provider-influxdb/shim => ./shim
)

require (
 github.com/komminarlabs/terraform-provider-influxdb/shim v0.0.0
 github.com/pulumi/pulumi-terraform-bridge/pf v0.29.0
 github.com/pulumi/pulumi-terraform-bridge/v3 v3.76.0
 github.com/pulumi/pulumi/sdk/v3 v3.108.1
)

Build the Provider:

Build Generator

Create the schema by running the following command.

make tfgen

Add Mappings

In this section we will add the mappings that allow the interoperation between the Pulumi provider and the Terraform provider. Terraform resources map to an identically named concept in Pulumi. Terraform data sources map to plain old functions in your supported programming language of choice.

Resource mapping

For every resource present in the provider, include an entry in the Resources property of the tfbridge.ProviderInfo structure.

Resources: map[string]*tfbridge.ResourceInfo{
   "influxdb_authorization": {Tok: tfbridge.MakeResource(mainPkg, mainMod, "Authorization")},
   "influxdb_bucket": {Tok: tfbridge.MakeResource(mainPkg, mainMod, "Bucket")},
   "influxdb_organization": {Tok: tfbridge.MakeResource(mainPkg, mainMod, "Organization")},
  },

Data Source mapping

Add an entry in the DataSources property of the tfbridge.ProviderInfo for each data source included in the provider.

DataSources: map[string]*tfbridge.DataSourceInfo{
   "influxdb_authorization": {Tok: tfbridge.MakeDataSource(mainPkg, mainMod, "getAuthorization")},
   "influxdb_authorizations": {Tok: tfbridge.MakeDataSource(mainPkg, mainMod, "getAuthorizations")},
   "influxdb_bucket": {Tok: tfbridge.MakeDataSource(mainPkg, mainMod, "getBucket")},
   "influxdb_buckets": {Tok: tfbridge.MakeDataSource(mainPkg, mainMod, "getBuckets")},
   "influxdb_organization": {Tok: tfbridge.MakeDataSource(mainPkg, mainMod, "getOrganization")},
   "influxdb_organizations": {Tok: tfbridge.MakeDataSource(mainPkg, mainMod, "getOrganizations")},
  },

Build Provider

Generate the provider binary by executing the following command:

make provider

Build SDKs

Compile the SDKs across the range of languages supported by Pulumi, and validate that the provider code adheres to the Go language standards.

make build_sdks

make lint_provider

Write documentation:

Incorporate a docs/ directory containing template pages that correspond to the different tabs typically found on a package page within the Pulumi Registry.

Overview, installation, & configuration

_index.md, which will be displayed on the Overview tab for your package in the Pulumi Registry. The title of this page should align with the package display name, serving as the heading shown on the package detail page. The Overview section is an ideal space to include a concise description of your package’s functionality along with a straightforward example.

---
title: InfluxDB
meta_desc: Provides an overview of the InfluxDB Provider for Pulumi.
layout: package
---

installation-configuration.md, this file will be displayed on your package’s Installation & Configuration tab in the Pulumi Registry. Utilize this page to provide comprehensive instructions on setting up your package, covering aspects such as authentication procedures. Additionally, include a list of configuration options available for use with your package.

---
title: InfluxDB Installation & Configuration
meta_desc: Information on how to install the InfluxDB provider.
layout: package
---

Publish your package:

After authoring and thoroughly testing your package locally, the next step is to publish it to make it accessible to the Pulumi community. This process involves publishing several artifacts:

The npm, NuGet, Java, and Python SDK packages to the npm Registry, the NuGet Gallery, Maven Central and the Python Package Index
The Go module to your Git repository, by adding a tag.
The binary Pulumi resource provider plugin to a binary hosting provider of your choice.

To streamline this process, we’ll leverage GitHub Actions. Below are the GitHub Actions you’ll use for this purpose.

name: release

on:
  push:
    tags:
      - v*.*.*

env:
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
  NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
  PUBLISH_NPM: true
  NPM_REGISTRY_URL: https://registry.npmjs.org
  NUGET_PUBLISH_KEY: ${{ secrets.NUGET_PUBLISH_KEY }}
  NUGET_FEED_URL: https://api.nuget.org/v3/index.json
  PUBLISH_NUGET: true
  PYPI_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
  PYPI_USERNAME: " __token__"
  PYPI_REPOSITORY_URL: ""
  PUBLISH_PYPI: true

jobs:
  publish_binary:
    name: Publish Binary
    runs-on: ubuntu-latest
    permissions:
      contents: write
    strategy:
      fail-fast: true
      matrix:
        goversion:
          - 1.22.x
    steps:
      - name: Checkout Repo
        uses: actions/checkout@v3

      - name: Unshallow clone for tags
        run: git fetch --prune --unshallow --tags

      - name: Install Go
        uses: actions/setup-go@v3
        with:
          go-version: ${{matrix.goversion}}

      - name: Install pulumictl
        uses: jaxxstorm/action-install-gh-release@v1.10.0
        with:
          repo: pulumi/pulumictl

      - name: Set PreRelease Version
        run: echo "GORELEASER_CURRENT_TAG=v$(pulumictl get version --language generic)" >> $GITHUB_ENV

      - name: Run GoReleaser
        uses: goreleaser/goreleaser-action@v2
        with:
          args: -p 3 release --rm-dist
          version: latest

  publish_sdk:
    name: Publish SDK
    runs-on: ubuntu-latest
    needs: publish_binary
    strategy:
      fail-fast: true
      matrix:
        dotnetversion:
          - 6.0.x
        goversion:
          - 1.22.x
        nodeversion:
          - 16.x
        pythonversion:
          - "3.9"
        language:
          - nodejs
          - python
          - dotnet
          - go
    steps:
      - name: Checkout Repo
        uses: actions/checkout@v4

      - name: Unshallow clone for tags
        run: git fetch --prune --unshallow --tags

      - name: Install Go
        uses: actions/setup-go@v5
        with:
          go-version: ${{ matrix.goversion }}

      - name: Install pulumictl
        uses: jaxxstorm/action-install-gh-release@v1.11.0
        with:
          repo: pulumi/pulumictl

      - name: Install pulumi
        uses: pulumi/actions@v4

      - if: ${{ matrix.language == 'nodejs'}}
        name: Setup Node
        uses: actions/setup-node@v1
        with:
          node-version: ${{matrix.nodeversion}}
          registry-url: ${{env.NPM_REGISTRY_URL}}

      - if: ${{ matrix.language == 'dotnet'}}
        name: Setup DotNet
        uses: actions/setup-dotnet@v1
        with:
          dotnet-version: ${{matrix.dotnetversion}}

      - if: ${{ matrix.language == 'python'}}
        name: Setup Python
        uses: actions/setup-python@v1
        with:
          python-version: ${{matrix.pythonversion}}

      - name: Build SDK
        run: make build_${{ matrix.language }}

      - if: ${{ matrix.language == 'python' && env.PUBLISH_PYPI == 'true' }}
        name: Publish package to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          user: ${{ env.PYPI_USERNAME }}
          password: ${{ env.PYPI_PASSWORD }}
          packages_dir: ${{github.workspace}}/sdk/python/bin/dist

      - if: ${{ matrix.language == 'nodejs' && env.PUBLISH_NPM == 'true' }}
        uses: JS-DevTools/npm-publish@v1
        with:
          access: "public"
          token: ${{ env.NPM_TOKEN }}
          package: ${{github.workspace}}/sdk/nodejs/bin/package.json

      - if: ${{ matrix.language == 'dotnet' && env.PUBLISH_NUGET == 'true' }}
        name: publish nuget package
        run: |
          dotnet nuget push ${{github.workspace}}/sdk/dotnet/bin/Debug/*.nupkg -s ${{ env.NUGET_FEED_URL }} -k ${{ env.NUGET_PUBLISH_KEY }}
          echo "done publishing packages"

Publish the documentation:

To publish your package on the Pulumi Registry, all package documentation is managed through the pulumi/registry repository on GitHub. Here’s how to proceed:

Fork and clone the pulumi/registry repository to your local machine.
Add your package to the community package list within the repository.

{
  "repoSlug": "komminarlabs/pulumi-influxdb",
  "schemaFile": "provider/cmd/pulumi-resource-influxdb/schema.json"
},

After making the necessary changes to add your package to the community package list, open a pull request with the modifications. Subsequently, await review from a member of the Pulumi team.
Upon review, a Pulumi employee will collaborate with you to finalize the steps required for publishing your Pulumi Package.

Using the provider:

Now that we have successfully built and published our Pulumi provider, let’s proceed to utilize it for resource creation. In this instance, we’ll opt for Python as our preferred programming language.

Installation

mkdir python-influxdb
cd python-influxdb

# (Go through the prompts with the default values)
pulumi new python

# Use the virtual Python env that Pulumi sets up for you
source venv/bin/activate

# Install the provider package
pip install komminarlabs_influxdb

Set up environment

You have the option to configure the provider either through environment variables or by utilizing the Pulumi.dev.yaml file.

export INFLUXDB_URL="http://localhost:8086"
export INFLUXDB_TOKEN="***"

pulumi config set influxdb:url http://localhost:8086
pulumi config set influxdb:token *** --secret

Add the following contents to the __main__.py file

"""A Python Pulumi program"""

import pulumi
import komminarlabs_influxdb as influxdb

organization = influxdb.Organization(
    "IoT",
    name="IoT",
    description="IoT organization"
)

bucket = influxdb.Bucket(
    "signals",
    org_id=organization.id,
    name="signals",
    description="This is a bucket to store signals",
    retention_period=604800,
)

Next, execute the pulumi preview command to view a preview of the updates to an existing stack. Follow this by running pulumi up to either create or update the resources within the stack.

You can also view the stacks in Pulumi cloud.

Additional Resources:

Deploying and Managing InfluxDB Resources with Terraform

Thulasiraj Komminar — Thu, 15 Feb 2024 17:45:23 +0000

Introduction:

Terraform is a powerful infrastructure as code tool that automates cloud infrastructure provisioning and management through simple configuration files. If you’re interested in learning more, I’ve written a short blog outlining some key components of Terraform.
InfluxDB is a specialized time-series database tailored for efficiently handling large volumes of time-stamped data. In this blog, we’ll explore how to leverage Terraform for provisioning and managing resources within InfluxDB.

Prerequisites:

To install Terraform, you can easily follow the steps outlined in this blog post: Install Terraform.

Before diving in, ensure you have a basic understanding of InfluxDB and its components. For installation guidance, refer to this resource: Install InfluxDB.

Provider Configuration:

To create and manage InfluxDB resources using Terraform, it utilizes specialized plugins known as providers to interface with InfluxDB. I’ve developed and published a provider for InfluxDB on the Terraform registry, enabling seamless resource creation and management.

Let’s begin by configuring the provider.

Create a file named versions.tf and insert the following code to declare the InfluxDB provider. This allows Terraform to install and utilize the provider. Provider dependencies are specified within a required_providers block.

terraform {
  required_providers {
    influxdb = {
      source  = "komminarlabs/influxdb"
      version = "1.0.0"
    }
  }
}

Create another file named main.tf to initialize the InfluxDB provider. We’ll use a provider block to configure it. You can specify the provider configuration using arguments such as url and token, or alternatively, utilize environment variables like INFLUXDB_TOKEN and INFLUXDB_URL.

export INFLUXDB_TOKEN="influxdb-token"
export INFLUXDB_URL="http://localhost:8086"

provider "influxdb" {
  token = "influxdb-token"
  url   = "http://localhost:8086"
}

Creating and Managing InfluxDB Resources:

The komminarlabs/influxdb provider offers various data sources and resources.

Data Sources:

influxdb_authorization
influxdb_authorizations
influxdb_bucket
influxdb_buckets
influxdb_organization
influxdb_organizations

Resources:

influxdb_authorization
influxdb_bucket
influxdb_organization

We’ll begin by creating resources and then utilize data sources to query the created resources. Let’s create two files: resources.tf and datasources.tf.

Resources

Resources are the most important element in the Terraform language. Each resource block describes one or more infrastructure objects.

Organization:

An InfluxDB organization is a workspace for a group of users. All dashboards, tasks, buckets, members, etc., belong to an organization. Add the following code to create our organisation.

resource "influxdb_organization" "iot" {
  name        = "IoT"
  description = "An IoT organization"
}

After running a Terraform plan and verifying everything looks good, let’s proceed with applying the changes.

Terraform execution plan and apply results

Switch to the new organization in InfluxDB

Bucket:

An InfluxDB bucket is a named location where time series data is stored. All buckets have a retention period, a duration of time that each data point persists. InfluxDB drops all points with timestamps older than the bucket’s retention period. A bucket belongs to an organization.

Let’s proceed by creating a bucket named signals with a retention period of 14 days (1209600 seconds).

resource "influxdb_bucket" "signals" {
  org_id           = influxdb_organization.iot.id
  name             = "signals"
  description      = "This is a bucket to store signals"
  retention_period = 1209600
}

Terraform execution plan and apply results

Authorization:

Authorizations are InfluxDB Read/Write API tokens that grants read access, write access, or both to specific buckets in an organization.

In the following step, we will generate an authorization that enables both read and write access to the bucket established in the prior phase.

resource "influxdb_authorization" "signals_rw" {
  org_id      = influxdb_organization.iot.id
  description = "Read & Write access signals bucket"

  permissions = [{
    action   = "read"
    resource = {
      id     = influxdb_bucket.signals.id
      org_id = influxdb_organization.iot.id
      type   = "buckets"
    }
    },
    {
      action   = "write"
      resource = {
        id     = influxdb_bucket.signals.id
        org_id = influxdb_organization.iot.id
        type   = "buckets"
      }
  }]
}

Terraform execution plan and apply results

API token shown in InfluxDB

Data Sources

With all the necessary resources created in InfluxDB to manage our time series data, we can now utilize datasources to list all the resources we’ve created.

A data source is accessed via a special kind of resource known as a data resource, declared using a data block.

data "influxdb_organization" "iot" {
  name = "IoT"
}

output "iot_organization" {
  value = data.influxdb_organization.iot
}

data "influxdb_bucket" "signals" {
  name = "signals"
}

output "signals_bucket" {
  value = data.influxdb_bucket.signals
}

Terraform execution plan and apply results

Additional Resources:

Conclusion:

Now that we’ve explored how to leverage Terraform for creating and managing InfluxDB resources, it’s time to start utilizing it. If you encounter any bugs or issues while using the provider, be sure to report them promptly.

Effortless Infrastructure: Mastering Automated Deployments with Terraform and GitHub Actions

Thulasiraj Komminar — Fri, 26 Jan 2024 16:39:22 +0000

Introduction

In the earlier blog post, I showcased the steps to achieve continuous deployment using GitHub Actions and AWS CodeBuild specifically for code deployment. Expanding on those principles, let’s delve into the next phase of our workflow — incorporating infrastructure deployment through Terraform.

Deployment workflow

Setup Terraform Cloud Workspace:

The GitHub Action you’re about to set up will seamlessly integrate with Terraform Cloud, allowing you to efficiently plan and apply your configurations. Before configuring the Actions workflow, ensure you’ve taken the necessary steps: creating a workspace, incorporating your AWS credentials into your Terraform Cloud workspace, and generating a Terraform Cloud user API token.

Step 1:

To kickstart the process, initiate a new Terraform Cloud workspace. Navigate to the Projects & Workspaces page, click on New Workspace, and opt for the API-Driven Workflow option.

Step 2:

Provide a name for the workspace and proceed by clicking on the Create button.

Step 3:

Now, it’s time to include your AWS IAM credentials as environmental variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION) within the workspace. Navigate to the Variables section in the workspace, click on Add Variable, and select Environment variable as the variable category. Terraform Cloud will use these credentials to authenticate to AWS.

Step 4:

Now, let’s generate an API token to enable authentication for the Actions workflow in Terraform Cloud. Head to the Tokens page within your Terraform Cloud User Settings. Click on Create an API token, provide a description, and then click Generate token.

Set up a GitHub repository:

Establish a GitHub repository dedicated to housing your infrastructure configuration code and the associated GitHub Actions workflow.

Step 1:

Once you’ve created the repository, proceed to the Settings page. Access the Secrets and Variables menu, specifically selecting Actions. Click New Repository Secret and input the Terraform API token generated in the previous step.

Step 2:

Now, it’s time to incorporate the following GitHub Actions workflow.

Plan

name: 'Terraform'

on:
  pull_request:
    branches:
      - main
    paths:
      - terraform/**
      - modules/**

env:
  TF_API_TOKEN: ${{ secrets.TF_API_TOKEN }}
  TF_WORKING_DIRECTORY: terraform/

permissions:
  contents: read
  pull-requests: write

jobs:
  terraform:
    name: "Plan"
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: ${{ env.TF_WORKING_DIRECTORY }}

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - uses: hashicorp/setup-terraform@v3
        with:
            cli_config_credentials_token: ${{ env.TF_API_TOKEN }}

      - name: Terraform Init
        id: init
        run: |
          git config --global url."https://oauth2:${{ secrets.GITHUB_TOKEN }}@github.com".insteadOf ssh://git@github.com
          terraform init

      - name: Terraform Validate
        id: validate
        run: terraform validate -no-color

      - name: Terraform Plan
        id: plan
        run: terraform plan -no-color
        continue-on-error: true

      - name: Update PR
        uses: actions/github-script@v7
        env:
          PLAN_OUTPUT: ${{ steps.plan.outputs.stdout }}
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            // 1. Retrieve existing bot comments for the PR
            const { data: comments } = await github.rest.issues.listComments({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
            })
            const botComment = comments.find(comment => {
              return comment.user.type === 'Bot' && comment.body.includes('Terraform Plan Results')
            })

            // 2. Prepare format of the comment
            const output = `### Terraform Plan Results 🚀
            #### Terraform Initialization ⚙️\`${{ steps.init.outcome }}\`
            #### Terraform Validation 🤖\`${{ steps.validate.outcome }}\`
            <details><summary>Validation Output</summary>

            \`\`\`\n
            ${{ steps.validate.outputs.stdout }}
            \`\`\`

            </details>

            #### Terraform Plan 📖\`${{ steps.plan.outcome }}\`

            <details><summary>Show Plan</summary>

            \`\`\`\n
            ${process.env.PLAN_OUTPUT}
            \`\`\`

            </details>

            *Pusher: @${{ github.actor }}, Action: \`${{ github.event_name }}\`, Working Directory: \`${{ env.TF_WORKING_DIRECTORY }}\`, Workflow: \`${{ github.workflow }}\`*`;

            // 3. If we have a comment, update it, otherwise create a new one
            if (botComment) {
              github.rest.issues.updateComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                comment_id: botComment.id,
                body: output
              })
            } else {
              github.rest.issues.createComment({
                issue_number: context.issue.number,
                owner: context.repo.owner,
                repo: context.repo.repo,
                body: output
              })
            }

Now, let’s delve into the deployment job’s steps. We’ve included four key actions:

Check out code: I’m utilizing a GitHub action to retrieve the code from the repository.
Terraform Init: This step utilizes the hashicorp/setup-terraform@v3 action to initialize Terraform. I included the Git configuration in case the code involves private GitHub modules.
Terraform Validate: This step is dedicated to validating the Terraform code.
Terraform Plan: This step is employed to execute the Terraform plan.
Update PR: This step leverages the actions/github-script@v7 action to update the Pull Request with the results from the preceding actions.

Apply

name: 'Terraform'

on:
  push:
    branches:
      - main
    paths:
      - terraform/**
      - modules/**

env:
  TF_API_TOKEN: ${{ secrets.TF_API_TOKEN }}
  TF_WORKING_DIRECTORY: terraform/

permissions:
  contents: read

jobs:
  terraform:
    name: "Apply"
    runs-on: ubuntu-latest
    environment: prd
    defaults:
      run:
        working-directory: ${{ env.TF_WORKING_DIRECTORY }}

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - uses: hashicorp/setup-terraform@v3
        with:
            cli_config_credentials_token: ${{ env.TF_API_TOKEN }}

      - name: Terraform Init
        id: init
        run: |
          git config --global url."https://oauth2:${{ secrets.GITHUB_TOKEN }}@github.com".insteadOf ssh://git@github.com
          terraform init

      - name: Terraform Apply
        id: apply
        run: terraform apply -auto-approve -no-color

Let’s delve into the steps outlined in this workflow.

Check out code: I’m utilizing a GitHub action to retrieve the code from the repository.
Terraform Init: This step utilizes the hashicorp/setup-terraform@v3 action to initialize Terraform. Additionally, I’ve included Git configuration in case the code incorporates private GitHub modules. Executing this action is a prerequisite before proceeding with the apply step.
Terraform Apply: This step applies the Terraform infrastructure configuration to AWS.

Demo

Now, let’s put our setup into action. For demonstration purposes, I’ve included Terraform configuration files to create an S3 bucket.

Pull Request

After incorporating the Terraform configuration files into a feature branch, let’s create a pull request to the main branch. This action will initiate GitHub Actions to execute Validation and Plan, with the results seamlessly added back to the pull request for convenient review.

Pull Request showcasing Terraform Plan Results

Thorough Examination of Terraform Plan Details

Merge

After a thorough review of the pull request and ensuring the Terraform plan results meet expectations, proceed to merge the pull request. This will trigger the Terraform apply process.

Production Deployment

In-Depth Exploration of the Terraform Workspace

Wrapping it up

In conclusion, we’ve explored the powerful capabilities of GitHub Actions in automating Terraform infrastructure deployments. If you’ve already followed my previous blog on code deployment using GitHub Actions, you can seamlessly combine these processes to achieve sequential infrastructure deployment and code deployment, streamlining your development workflow.

Efficient Continuous Deployment with GitHub Environments and AWS CodeBuild

Thulasiraj Komminar — Mon, 30 Oct 2023 22:17:00 +0000

Introduction

Continuous Deployment (CD) is a methodology that leverages automation for releasing software updates efficiently. In a standard CD setup, code undergoes automatic building and testing phases prior to its deployment. With GitHub Actions, automating these software workflows becomes seamless. Configure your CD pipeline to initiate whenever there’s a GitHub event, such as a new code push or a pull request. For a comprehensive list of event triggers, click here.

Environments define common deployment targets such as production, staging, or development. They can be set up with protection rules and associated secrets. When a workflow job references an environment, it only begins after satisfying all the environment’s protection rules. Moreover, a job can only access the secrets linked to an environment after all its deployment protection criteria are met.

In this blog post, I’ll guide you through setting up a Continuous Deployment pipeline using GitHub Actions and environments, focusing on both development and production stages.

Initial setup

GitHub

Environments:

First, we’ll establish the development and production environments and implement deployment protection rules for added security.

Navigate to the repository where you intend to set up the Continuous Deployment (CD). Click on Settings, and then select Environments from the sidebar. To add environments, simply click on the New Environment button. As depicted in the image below, I have established two environments.

After adding an environment, you’ll be prompted to configure its settings. Let’s dive into that. We’ll start by setting up a protection rule for the production environment. As illustrated below, I’ve added a required reviewer to manually approve all deployments headed to production.

Next, we’ll restrict deployments to this environment based on specific branch naming patterns. As demonstrated below, I selected the Selected branches and tags rule. Click on Add deployment branch or tag rule, and in the subsequent popup, select Ref type as branch and enter main for the Name pattern.

To deploy to an AWS account, we require the AWS account ID and region. Therefore, I’ve added these as environment secrets, along with a variable to capture the environment name.

Likewise, I’ve configured deployment restrictions, environment secrets, and variables for the development environment. However, I omitted the reviewer setup since this environment is solely for development deployments.

Actions:

Now, let’s proceed to create GitHub workflows so that deployments are automatically triggered whenever a commit is made to either the develop or main branch.

main

name: CD - Production

on:
  push:
    branches:
      - main

permissions:
  id-token: write
  contents: read

jobs:
  codebuild:
    runs-on: ubuntu-latest
    environment: Production
    steps:
      - name: echo
        run: echo 'Deploying to ${{ vars.ENV }}'

      - name: Check out code
        uses: actions/checkout@v4

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNTID }}:role/GitHubTriggerRole
          aws-region: ${{ secrets.AWS_REGION }}

      - name: Run CodeBuild
        uses: aws-actions/aws-codebuild-run-build@v1
        with:
          project-name: CD-CodeBuild

develop

name: CD - Development

on:
  push:
    branches:
      - develop

permissions:
  id-token: write
  contents: read

jobs:
  codebuild:
    runs-on: ubuntu-latest
    environment: Development
    steps:
      - name: echo
        run: echo 'Deploying to ${{ vars.ENV }}'

      - name: Check out code
        uses: actions/checkout@v4

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNTID }}:role/GitHubTriggerRole
          aws-region: ${{ secrets.AWS_REGION }}

      - name: Run CodeBuild
        uses: aws-actions/aws-codebuild-run-build@v1
        with:
          project-name: CD-CodeBuild

Now, let’s delve into the deployment job’s steps. We’ve included four key actions:

Echo: This step simply prints the name of the environment being deployed.
Check out code: I’m utilizing a GitHub action to retrieve the code from the repository.
Configure AWS Credentials: In this step, I’ve set up an OIDC (OpenID Connect) configuration to enable role assumption, allowing me to make AWS service calls on behalf of the environment. I’ll provide a more detailed explanation of the OIDC setup in the next section, specifically focusing on AWS.
Run CodeBuild: This final step initiates an AWS CodeBuild project and waits for its status, ensuring a smooth and coordinated deployment process.

Note: Ensure that the environment name in the actions file corresponds with the GitHub repository’s environment name.

AWS

Now, let’s explore what needs to be configured on the AWS side.

IAM

We need to set up two roles: one for GitHub Actions to assume using OIDC, and another for the CodeBuild project to assume.

OIDC:

To begin, log in to the AWS console and navigate to the IAM service. In the sidebar, select Identity providers and click on Add Provider. In the Configure provider window, choose the OpenID Connect option.

Here are the specific details you’ll need to fill in:

For the Provider URL, use https://token.actions.githubusercontent.com.
For the Audience, enter sts.amazonaws.com.
Finally, click Add Provider to complete the setup.

More details on this can be found here.

Create IAM Roles:

1. GitHub role:

Now, within the IAM services, select Create role. Opt for the Custom trust policy as the trusted entity type.

Here’s the trust policy text that you should use in the custom trust policy text box:

Note: Replace the GitHub username/repository with your own.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::${account-id}:oidc-provider/token.actions.githubusercontent.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                },
                "StringLike": {
                    "token.actions.githubusercontent.com:sub": "repo:komminarlabs/cd-with-github-actions:*"
                }
            }
        }
    ]
}

Name the role as GitHubTriggerRole. After creating the role, attach the following policy as an inline policy to grant access for writing logs to CloudWatch and the necessary permissions to trigger a CodeBuild project.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:PutLogEvents",
                "logs:GetLogEvents",
                "logs:CreateLogStream",
                "logs:CreateLogGroup"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:logs:eu-central-1:${account-id}:*"
        },
        {
            "Action": [
                "codebuild:UpdateReport",
                "codebuild:StartBuild",
                "codebuild:CreateReportGroup",
                "codebuild:CreateReport",
                "codebuild:BatchPutTestCases",
                "codebuild:BatchPutCodeCoverages",
                "codebuild:BatchGetBuilds"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

2. CodeBuild Role:

Next, let’s create a second role for CodeBuild to assume. When you click the Create role button, select CodeBuild as the AWS Service. Attach the existing managed policy AWSCodeBuildAdminAccess to this role.

CodeBuild

Now, head over to the CodeBuild service and select Create build project. Specify the project name as CD-CodeBuild. In the Source section, choose GitHub from the dropdown and include the Repository URL: https://github.com/komminarlabs/cd-with-github-actions

Note: You have the option to authenticate with GitHub using either OAuth or a personal token.

Apply the following Environment settings, and select the role you previously created.

Select Insert build commands under Buildspec, then paste the following content. Finally, create the CodeBuild project.

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.11

  build:
    commands:
      - echo "This is a test"

Testing

Development

Alright, now that we’ve configured everything on both the GitHub and AWS sides, it’s time to put our setup to the test. To do this, I used a dummy_file.txt in the GitHub repository for testing purposes.

After pushing a change directly to the develop branch, it promptly initiated a deployment to the development environment. The deployment was successful, and you can view the corresponding CodeBuild logs within the Actions.

Production

With the successful deployment to the development environment, let’s now create a pull request to the main branch to trigger the production deployment.

As depicted below, the pull request to main displays the status of the deployment to development. You can enforce this deployment check as mandatory in the branch protection rules for added reliability.

Now, proceed to merge this pull request. The deployment to the production environment will pause and await manual approval, as we’ve included reviewers as a protection rule in the production environment.

Alright, after a thorough manual review of the changes 😊, let’s proceed with approving this deployment. To do so, click on Review deployments, select the production environment, and then click on Approve and deploy.

With the approval in place, the production deployment process is now initiated.

Wrapping it up

I hope this blog has provided you with a clear understanding of how to leverage GitHub environments to enhance your continuous deployment process.

What is HashiCorp Terraform?

Thulasiraj Komminar — Fri, 22 Sep 2023 14:29:07 +0000

Introduction

In this blog I will explain what is Terraform and some of its common terms. I will also list some good to follow standards when you start using Terraform.

What is it?

Terraform is an infrastructure as code tool that lets you define both cloud and on-premise resources in human-readable configuration files.
It can manage both low-level(Networking, Storage) & high-level components(DNS, SaaS).
It can be used to manage single application and also an entire datacenter.

What it isn’t?

It is not a configuration management tool that focuses on single application configuration but Terraform focuses on higher-level abstraction like managing the entire environment

Terraform Editions

Open Source

A free to download tool that you interact with CLI.

Terraform Cloud

SaaS application that runs in a secure remote environment and stores states and secrets.
Connects to VCS and new commits triggers Terraform automatically.
It has Rich UI and RBAC controls.

Terraform Enterprise

Self-hosted distribution of Terraform Cloud. It offers enterprises a private instance of the Terraform Cloud application, with no resource limits and with additional enterprise-grade architectural features like audit logging and SAML single sign-on.

Terraform Jargons

Workspace

A workspace contains everything that Terraform needs to manage a given collection of infrastructure.
It links to VCS and has Variables, Credentials, Secrets, State versions, and Run History.

Backend

A backend defines where Terraform stores its state data files.

terraform {
  cloud {
    organization = "sampleorg"

    workspaces {
      name = "dev-workspace"
    }
  }
}

Provider

Provides abstraction to APIs that connect to the infrastructure hosts.
Without providers, Terraform can’t manage any kind of infrastructure.
Every provider makes a list of resources and data types available for use in the Terraform code.

Registry

The Terraform Registry is an interactive resource for discovering a wide selection of integrations (providers), configuration packages (modules), and security rules (policies) for use with Terraform.

HCL (HashiCorp Configuration Language)

HCL is a declarative and domain-specific language.

State

Terraform is a stateful application, meaning it keeps track of everything it builds in the infrastructure.
State is used by Terraform to map real-world resources to your configuration.

How Terraform works?

Terraform creates and manages resources on cloud platforms and other services through their APIs.
Providers enable Terraform to work with virtually any platform or service with an accessible API.
The core Terraform workflow consists of three stages:

Write:

Defining resources

Plan:

An execution plan that describes the infra terraform will create/update/destroy based on existing infra and your configurations.

Apply:

Performing the operations in the correct order by respecting resource dependencies. Can be automatic or manual.

Configuration Language

Terraform supports HCL, JSON, CRDS and CDK. CDK allows you to use familiar programming languages to define and provision infrastructure. It also lets you leverage the power of your existing toolchain for testing, dependency management, etc. In this blog we will focus on HCL and how to configure Providers, Resources and Modules.

HCL

Provider

Terraform Cloud and Terraform Enterprise install providers as part of every run.

Provider configuration

Provider configurations belong in the root module of a Terraform configuration. Child modules receive their provider configurations from the root module.

provider "aws" {
  alias  = "us_east_1"
  region = "us-east-1"
}

Provider Requirements

Each Terraform module must declare which providers it requires, so that Terraform can install and use them.

terraform {
  required_providers {
    mycloud = {
      source  = "mycorp/mycloud"
      version = "~> 1.0"
    }
  }
}

Variables, Outputs, and Locals

In comparison with traditional programming languages,

Input variables are like function arguments.
Output values are like function return values.
Local values are like a function’s temporary local variables.

variable "image_id" {
  type = string
}

output "instance_ip_addr" {
  value = aws_instance.server.private_ip
}

locals {
  common_tags = {
    Stack       = local.stack
    Environment = local.environment
  }
}

Resource

Each resource block describes one or more infrastructure objects, such as virtual networks, compute instances, or higher-level components such as DNS records.

resource "aws_athena_workgroup" "default" {
  name = "default"
  tags = locals.common_tags

  configuration {
    enforce_workgroup_configuration    = true
    publish_cloudwatch_metrics_enabled = true

    result_configuration {
      output_location = "s3://some-bucket/path"

      encryption_configuration {
        encryption_option = "SSE_S3"
      }
    }
  }
}

Data Sources

Data sources allow Terraform to use information defined outside of Terraform.

data "aws_vpc" "default" {
  id = "vpc-0aexzk638sp5h9u7v"
}

Modules

Containers for multiple resources that are used together.

Root Module

Every Terraform configuration has at least one module, known as its root module, which consists of the resources defined in the .tf files in the main working directory.

Child Modules

A module that has been called by another module is often referred to as a child module.

Published Modules

In addition to modules from the local filesystem, Terraform can load modules from a public or private registry.

module "example_glue_job" {
  source            = "github.com/komminar/terraform-aws-glue-job?ref=v0.1.0"
  name              = "example-glue-job"
  max_retries       = 1
  number_of_workers = 2
  schedule          = "cron(0 12 * * ? *)"
  script_location   = "S3://example-bucket/location/script.py"
  trigger_type      = "SCHEDULED"
  worker_type       = "Standard"

  default_arguments = {
    "--VAR1" = "some value"
  }

  tags = {
    Environment = "development"
    Stack       = "glue"
  }
}

You can find some of my AWS Terraform modules here.

Standards

Use count only for enabling/disabling resource creation and use for_each for loops.
Variables are ordered A-Z, but tags always comes at last.
Multi-line list or map should be in new line.

resource "aws_sagemaker_domain" "default" {
  domain_name             = var.name
  app_network_access_type = var.app_network_access_type
  auth_mode               = "IAM"
  subnet_ids              = var.subnet_ids
  vpc_id                  = var.vpc_id
  tags                    = var.tags

  default_user_settings {
    execution_role  = var.role_arn != null ? var.role_arn : aws_iam_role.default[0].arn
    security_groups = var.security_groups

    jupyter_server_app_settings {
      lifecycle_config_arns = [aws_sagemaker_studio_lifecycle_config.jupyter.arn]

      default_resource_spec {
        instance_type        = "system"
        lifecycle_config_arn = aws_sagemaker_studio_lifecycle_config.jupyter.arn
      }
    }

    kernel_gateway_app_settings {
      lifecycle_config_arns = [aws_sagemaker_studio_lifecycle_config.kernel.arn]

      default_resource_spec {
        instance_type        = "system"
        lifecycle_config_arn = aws_sagemaker_studio_lifecycle_config.kernel.arn
      }
    }
  }
}

There are some linters already available for you to use

https://github.com/terraform-linters/tflint

https://github.com/terraform-linters/tflint-ruleset-terraform

https://github.com/terraform-linters/tflint-ruleset-aws