DEV Community: prakhyatkarri

From a Unified Bronze Layer to Multiple Silver Layers: Streamlining Data Transformation in Databricks Unity Catalog

prakhyatkarri — Sun, 20 Oct 2024 21:59:58 +0000

Introduction

In modern data engineering workflows, managing raw data and its transformations is crucial for providing clean, reliable, and structured data to analytics teams and downstream applications. Databricks Unity Catalog provides a unified data governance model that simplifies and secures data access across the entire organization. This guide will walk through a comprehensive process for ingesting raw data into a shared bronze layer and transforming it into multiple silver layers in a Unity Catalog setup, with data stored in catalog-managed tables instead of traditional file paths. We will use examples and data flow diagrams to illustrate the concepts.

Overview of the Layers

Raw Layer: Unstructured or semi-structured data as it arrives from source systems (e.g., logs, IoT feeds, application data).
Bronze Layer: A landing area where raw data is cleaned and minimally processed. This layer maintains data fidelity while addressing obvious data quality issues.
Silver Layer: Curated data for specific use cases. Data from the bronze layer is cleaned, structured, and aggregated into independent tables optimized for downstream use (e.g., analytics, reporting, machine learning).
Data Flow Diagram

+-------------------+          +-------------------+          +-------------------+
|    Raw Layer      |   --->   |   Bronze Layer    |   --->   |  Silver Layers    |
| (External Data)   |          | (Cleaned Data)    |          | (Curated Data)    |
+-------------------+          +-------------------+          +-------------------+

                                       |
                           +-----------+-----------+
                           |                       |
               +-------------------+     +-------------------+     +-------------------+
               |  Bronze to Silver 1|     |  Bronze to Silver 2|     |  Bronze to Silver n|
               | (Specific Use Case)|     | (Specific Use Case)|     | (Specific Use Case)|
               +-------------------+     +-------------------+     +-------------------+

Step-by-Step Guide

1. Ingesting Data into the Raw Layer

The Raw Layer typically represents data coming from various external sources such as JSON, CSV, or logs, and is unprocessed. This data is brought into Databricks either through batch or streaming jobs. You could use Databricks Auto Loader for efficient file ingestion from cloud storage systems like Azure Data Lake Storage (ADLS).

Here’s an example of a batch ingestion job using PySpark:

from pyspark.sql import SparkSession

# Create a Spark session
spark = SparkSession.builder.appName("RawToBronzeIngestion").getOrCreate()

# Define source and target paths for raw data
raw_data_path = "abfss://raw@storage_account.dfs.core.windows.net/dataset/"
bronze_table = "catalog_name.schema_name.bronze_table"

# Load the raw data
raw_df = spark.read.option("header", True).csv(raw_data_path)

# Write the raw data into the bronze layer
raw_df.write.format("delta").mode("overwrite").saveAsTable(bronze_table)

Key Points:

Source: Data is ingested from external systems or cloud storage.
Target: The raw data is ingested directly into a Unity Catalog table in the bronze layer.
Table Management: Unity Catalog manages the metadata for the tables, providing security and governance.

2. Bronze Layer: Initial Cleansing and Normalization

The Bronze Layer is a lightly processed version of the raw data. The data is stored in Unity Catalog tables, allowing for basic cleansing and structural changes (e.g., handling nulls, normalizing data types). This layer acts as a source of truth and an archive of the original data with minor cleaning.

Let’s clean the raw data and store it in the bronze table:

from pyspark.sql.functions import col

# Clean and transform raw data
bronze_df = raw_df \
    .dropDuplicates() \
    .filter(col("important_column").isNotNull()) \
    .withColumn("processed_at", current_timestamp())

# Write the cleaned data into the bronze table
bronze_df.write.format("delta").mode("overwrite").saveAsTable(bronze_table)

Key Steps:

Data Cleansing: Remove duplicates, null values, and other obvious quality issues.
Column Enrichment: Add processing metadata such as timestamps for auditing.
Governance: Data in the bronze layer is managed and versioned with Delta Lake, enhancing traceability.

3. Transforming Bronze Data into Multiple Silver Layers

The Silver Layer represents clean, structured, and optimized datasets for specific downstream use cases. These could include separate tables for different business units or functional areas. The goal is to provide ready-to-use data for analytics, machine learning, or reporting.

Each silver table is designed with specific business logic in mind. Below, we walk through two different transformations from the bronze layer into silver layers.

Silver Layer 1: Customer Data Cleansing and Aggregation

For example, if you have customer data, you may want to clean and aggregate customer transactions for reporting purposes.

# Load the bronze data
bronze_df = spark.table(bronze_table)

# Transform and cleanse data for the silver layer
silver_customer_df = bronze_df \
    .filter(col("transaction_type") == "purchase") \
    .groupBy("customer_id") \
    .agg(sum("transaction_amount").alias("total_purchase"))

# Write the curated data to the silver layer
silver_customer_table = "catalog_name.schema_name.silver_customers"
silver_customer_df.write.format("delta").mode("overwrite").saveAsTable(silver_customer_table)

Silver Layer 2: Product Data Transformation

You might have product data that needs to be structured for inventory tracking:

# Filter and transform product-related data
silver_product_df = bronze_df \
    .filter(col("category").isNotNull()) \
    .select("product_id", "category", "inventory_status")

# Write the product data to another silver table
silver_product_table = "catalog_name.schema_name.silver_products"
silver_product_df.write.format("delta").mode("overwrite").saveAsTable(silver_product_table)

Key Points for Silver Layer:

Business Logic: Each silver table represents a different subset of the bronze data, filtered and transformed to serve specific use cases.
Table Partitioning: Silver tables can be partitioned for performance optimization based on frequently queried columns (e.g., date, customer region).
Storage Optimization: Data stored in the silver layer can be aggregated, cleaned, and formatted for efficient querying.

4. Managing Unity Catalog Tables

All data at each stage (raw, bronze, silver) is managed as Unity Catalog tables. This provides centralized control over access permissions, schema evolution, and lineage tracking. You can also enforce data masking or row-level security policies based on user roles.

To view and manage the tables:

-- List all Unity Catalog tables in a specific schema
SHOW TABLES IN catalog_name.schema_name;

5. Scheduling and Automation of Pipelines

Data pipelines for moving from raw to bronze and from bronze to silver can be automated using Databricks Workflows or Azure Data Factory. This ensures the ingestion and transformation processes are consistent and run on predefined schedules.

Here’s a sample setup of a Databricks job to automate this pipeline:

# Job Configuration (Pseudo Code)
job = {
    "name": "BronzeToSilverPipeline",
    "tasks": [
        {
            "task_key": "ingest_raw_data",
            "notebook_path": "/Repos/IngestRawData",
            "schedule": "0 2 * * *"  # Daily at 2 AM
        },
        {
            "task_key": "transform_to_silver",
            "notebook_path": "/Repos/TransformToSilver",
            "depends_on": "ingest_raw_data"
        }
    ]
}

This ensures each pipeline step runs in sequence and that data is consistently updated from raw to bronze to silver layers.

Conclusion

Ingesting and transforming data within Databricks using Unity Catalog offers a scalable and governed approach for managing data pipelines. By following this shared bronze layer and independent silver layers architecture, organizations can ensure data reliability, performance, and security. Leveraging Delta Lake and Unity Catalog features like access control and auditing ensures compliance while providing the flexibility to transform and prepare data for diverse use cases.

This architecture can be expanded further with additional layers (e.g., gold layer for final analytics) and extended with machine learning models. By managing all data in catalog-based tables, you create a unified environment with centralized governance and control.

Some reference links to help you build data pipelines using Databricks, Delta lake, Pyspark and Unity Catalog.

Mastering Terraform: Best Practices for Scalable, Secure, and Reliable Infrastructure as Code

prakhyatkarri — Fri, 08 Dec 2023 03:29:17 +0000

1. Organize Your Code into Modules:
Structuring your Terraform code into reusable modules promotes maintainability and reusability.

Example:

# main.tf

module "web_server" {
  source = "./modules/web_server"

  instance_count = 3
  // Other module-specific variables
}

# modules/web_server/main.tf

variable "instance_count" {
  description = "Number of web server instances"
}

resource "aws_instance" "web_server" {
  count = var.instance_count

  // Instance configuration
}

2. Use Variables for Configurability:
Leverage variables to make your configurations more flexible and easier to reuse.

Example:

# main.tf

variable "region" {
  default     = "us-west-2"
  description = "AWS region"
}

provider "aws" {
  region = var.region
}

3. Separate Environments with Workspaces:
Use workspaces to manage multiple environments (e.g., development, staging, production) with the same configuration.

Example:

terraform workspace new dev
terraform workspace new prod

4. Lock State for Collaboration:
Use remote state storage and locking to enable collaboration among team members.

Example (backend configuration):

# backend.tf

backend "s3" {
  bucket         = "my-terraform-backend"
  key            = "terraform.tfstate"
  region         = "us-east-1"
  encrypt        = true
  dynamodb_table = "terraform-lock-table"
}

5. Version Your Modules:
Version your modules to ensure reproducibility and avoid unexpected changes.

Example:

# main.tf

module "web_server" {
  source = "git::https://github.com/example/web_server.git?ref=v1.0.0"

  // Module-specific variables
}

6. Use Dependency Pinning:
Pin the versions of providers and modules to avoid unexpected changes.

Example (provider version pinning):

# main.tf

provider "aws" {
  version = "~> 3.0"
  // Other provider configuration
}

7. Avoid Hardcoding Sensitive Information:
Use environment variables or input variables to avoid hardcoding sensitive information like API keys.

Example:

# main.tf

provider "aws" {
  region = var.aws_region
  access_key = var.aws_access_key
  secret_key = var.aws_secret_key
}

8. Enable Detailed Logging for Troubleshooting:
Enable detailed logging during development and troubleshooting.

Example:

# main.tf

provider "aws" {
  // Other provider configuration
  // ...

  // Enable detailed logging
  terraform {
    log = {
      format = "json"
    }
  }
}

9. Use Terraform Modules Registry:
Leverage the Terraform Module Registry for discovering and sharing modules.

Example (using a module from the registry):

# main.tf

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.81.0"

  // Module-specific variables
}

10. Regularly Update Terraform and Providers:

Keep Terraform and your providers up to date to benefit from new features, bug fixes, and security updates.

Example (updating Terraform):

terraform version
terraform init -upgrade

11. Document Your Code:
Add comments and documentation to your Terraform code to explain complex configurations and provide context for future maintainers.

Example:

# main.tf

# Provisioning an S3 bucket for storing static website content
resource "aws_s3_bucket" "static_website" {
  # Bucket configuration
  bucket = "my-static-website"
  acl    = "public-read"
}

12. Use Data Sources for Information Retrieval:
Utilize Terraform data sources to fetch information from existing resources in your infrastructure.

Example (AWS VPC data source):

# main.tf

data "aws_vpcs" "all_vpcs" {}

output "vpc_ids" {
  value = data.aws_vpcs.all_vpcs.ids
}

13. Apply Least Privilege Principle:
Follow the principle of least privilege when configuring provider credentials. Avoid using overly permissive IAM roles or accounts.

Example (AWS IAM Role Policy):

# main.tf

provider "aws" {
  // Other provider configuration
  // ...

  assume_role {
    role_arn = "arn:aws:iam::ACCOUNT_ID:role/terraform"
  }
}

14. Test Your Infrastructure Code:
Implement automated testing for your Terraform code using tools like Terratest to catch issues early in the development process.

Example (Terratest):

// test/main_test.go

package test

import (
  "testing"
  "github.com/gruntwork-io/terratest/modules/terraform"
  "github.com/stretchr/testify/assert"
)

func TestTerraformExample(t *testing.T) {
  terraformOptions := &terraform.Options{
    // Set Terraform variables and other options
    TerraformDir: "../examples/simple",
  }

  // Run `terraform init` and `terraform apply`
  terraform.InitAndApply(t, terraformOptions)

  // Validate the result
  result := terraform.Plan(t, terraformOptions)
  assert.True(t, len(result.Diff.Modules) == 0)
}

15. Implement Remote Backend Locking:
Use a remote backend with locking (e.g., AWS S3 with DynamoDB) to prevent concurrent modifications and ensure state consistency.

Example (Remote backend configuration):

# backend.tf

backend "s3" {
  bucket         = "my-terraform-backend"
  key            = "terraform.tfstate"
  region         = "us-east-1"
  encrypt        = true
  dynamodb_table = "terraform-lock-table"
}

16. Monitor and Audit Changes:
Integrate Terraform with monitoring and auditing tools to track changes and monitor the state of your infrastructure.

Example (AWS CloudTrail integration):

# main.tf

resource "aws_cloudtrail" "example" {
  name                          = "example-cloudtrail"
  s3_bucket_name                = "example-cloudtrail-bucket"
  is_multi_region_trail         = true
  enable_log_file_validation    = true
  include_global_service_events = true
}

17. Version Control Your Infrastructure Code:
Store your Terraform code in version control systems (e.g., Git) to track changes, collaborate, and roll back to previous configurations.

Example (Git usage):

git init
git add .
git commit -m "Initial Terraform configuration"

18. Separate Immutable and Mutable Resources:
Distinguish between resources that are immutable (e.g., infrastructure components) and mutable (e.g., application configurations) to facilitate controlled updates.

Example:

# main.tf

// Immutable infrastructure
resource "aws_instance" "web_server" {
  // ...
}

// Mutable application configuration
resource "aws_s3_bucket" "app_config" {
  // ...
}

19. Regularly Review and Refactor Code:
Perform regular code reviews and refactor your Terraform code to improve readability, maintainability, and adherence to best practices.

20. Consider Using Terraform Cloud or Enterprise:
For larger teams or enterprise-scale projects, consider using Terraform Cloud or Enterprise for collaboration, version control, and additional features.

These additional best practices can enhance the security, reliability, and scalability of your Terraform configurations. Always adapt best practices based on your specific project requirements and team workflows.

21. Implement Resource Tagging:
Apply consistent tagging to your resources for better organization, cost tracking, and management.

Example:

# main.tf

resource "aws_instance" "example" {
  // ...

  tags = {
    Name        = "web-server"
    Environment = "production"
    Project     = "my-project"
  }
}

22. Backup and Version Your State File:
Regularly back up your Terraform state file, and consider versioning it. This ensures recovery options and allows you to roll back to a known state if needed.

23. Use Conditional Logic Sparingly:
Limit the use of conditional logic in your Terraform configurations. Complexity can make it harder to understand and maintain.

Example (Conditional Resource Creation):

# main.tf

resource "aws_instance" "example" {
  count = var.create_instance ? 1 : 0
  // ...
}

24. Handle Secrets Securely:
Use a dedicated secret management tool or platform (e.g., HashiCorp Vault) to store and retrieve sensitive information.

25. Keep Terraform Versions Consistent:
Ensure all team members are using the same Terraform version to prevent compatibility issues.

Example (Terraform Version Constraint in Code):

# main.tf

terraform {
  required_version = ">= 0.15, < 0.16"
}

26. Consider Using Terraform Modules for Workflows:
Break down complex workflows into reusable modules, making it easier to understand, test, and maintain.

Example (Module for Multi-Tier Application):

# main.tf

module "web_app" {
  source = "./modules/web_app"

  // Module-specific variables
}

27. Use Terraform's Built-in Functions:
Leverage Terraform's built-in functions for string manipulation, formatting, and other common tasks.

Example:

# main.tf

locals {
  environment_name = "dev"
}

resource "aws_s3_bucket" "example" {
  bucket = "my-${local.environment_name}-bucket"
  // ...
}

28. Implement a Review and Approval Process:
Set up a process for reviewing and approving Terraform changes before applying them to production environments.

29. Monitor Terraform Operations:
Use monitoring tools to track Terraform operations and identify issues or performance bottlenecks.

30. Document External Dependencies:
Clearly document external dependencies, such as external scripts or manual steps, required for the Terraform configuration to work.

Example (External Script Dependency):

# main.tf

provisioner "local-exec" {
  command = "./setup_script.sh"
}

31. Review and Implement Provider Security Best Practices:
Follow security best practices provided by your cloud providers to secure access and configurations.

Example (AWS Security Group):

# main.tf

resource "aws_security_group" "example" {
  // Security group configuration
  // ...

  ingress {
    from_port = 22
    to_port   = 22
    protocol  = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

32. Plan and Apply in Separate Steps:
Use separate steps for planning (terraform plan) and applying (terraform apply) to avoid accidental changes.

terraform plan -out=tfplan
terraform apply tfplan

33. Consider Using Terraform Cloud Workspaces:
For remote collaboration and management, consider using Terraform Cloud workspaces.

Example (Terraform Cloud Configuration):

# main.tf

terraform {
  backend "remote" {
    organization = "your-org"
    workspaces = {
      name = "example-workspace"
    }
  }
}

34. Automate Approval Workflows:
Integrate Terraform changes with approval workflows, such as using Terraform Cloud's Sentinel policies, to ensure changes are reviewed and approved before being applied.

35. Use State Data Outputs Wisely:
Be cautious with exposing sensitive information through state data outputs. Avoid outputting sensitive data unless necessary, and use data sources to retrieve information when possible.

Example (Outputting Sensitive Information):

# main.tf

resource "aws_instance" "example" {
  // ...
}

output "instance_ip" {
  value = aws_instance.example.private_ip
}

36. Implement Resource Naming Conventions:
Establish and adhere to naming conventions for resources to maintain consistency and clarity across your infrastructure.

Example:

# main.tf

resource "aws_instance" "web_server" {
  // ...
  tags = {
    Name        = "web-${var.environment}-instance"
    Environment = var.environment
  }
}

37. Set Resource Dependencies Explicitly:
Clearly define dependencies between resources using the depends_on attribute to ensure proper resource creation order.

Example:

# main.tf

resource "aws_security_group" "example" {
  // ...

  depends_on = [
    aws_vpc.example,
    aws_subnet.example,
  ]
}

38. Regularly Review Provider Release Notes:
Stay informed about provider updates by regularly reviewing release notes. This helps you leverage new features and stay aware of any changes that might affect your configurations.

39. Limit Blast Radius with Terraform Workspaces:
Use Terraform workspaces to isolate environments and limit the potential impact of changes across different stages (e.g., development, staging, production).

40. Avoid Hardcoded Resource IDs:
Avoid hardcoding resource IDs when referencing existing resources. Instead, use data sources to dynamically retrieve the required information.

Example (Hardcoded Security Group ID):

# main.tf

resource "aws_instance" "example" {
  // ...

  vpc_security_group_ids = ["sg-0123456789abcdef0"]
}

Example (Using Data Source):

# main.tf

data "aws_security_group" "example" {
  name = "example-security-group"
}

resource "aws_instance" "example" {
  // ...

  vpc_security_group_ids = [data.aws_security_group.example.id]
}

41. Handle Terraform State Locks Effectively:
Ensure effective handling of state locks, especially in scenarios where multiple team members or automation scripts might be interacting with the same state.

42. Use Resource Meta-Arguments Carefully:
Be cautious when using meta-arguments like count, for_each, or lifecycle blocks. They can significantly impact resource behavior and dependencies.

43. Implement Rollback Strategies:
Establish rollback strategies in case of unsuccessful Terraform deployments. This may involve creating backups, validating changes in a staging environment first, or using canary deployments.

44. Use Terraform Import Judiciously:
Use terraform import cautiously and understand its limitations. Manually importing resources into Terraform can lead to challenges in maintaining and updating configurations.

45. Consider Using Terraform Sentinel Policies:
Implement Sentinel policies in Terraform Cloud to enforce compliance and security standards within your infrastructure.

Example (Sentinel Policy File):

# sentinel.hcl

import "tfplan"

main = rule {
  all tfplan.resources.aws_instance as _, instances {
    instances.attributes.instance_type is "t2.micro"
  }
}

These additional best practices can contribute to the overall efficiency, security, and reliability of your Terraform configurations. As always, adapt them to fit the specific needs and scale of your projects.

Unlocking the Power of Infrastructure as Code: An In-Depth Introduction to Terraform for Modern DevOps

prakhyatkarri — Fri, 08 Dec 2023 03:00:30 +0000

What is Terraform?

Terraform is an open-source Infrastructure as Code (IaC) tool developed by HashiCorp. It enables users to define and provision infrastructure using a declarative configuration language. With Terraform, you can manage infrastructure resources such as virtual machines, networks, and storage across various cloud providers and on-premises environments.

Key features of Terraform include:

Declarative Syntax: Terraform uses a declarative language to define infrastructure as code. Users specify what resources they want, and Terraform ensures the desired state is achieved.
Multi-Cloud Support: Terraform supports multiple cloud providers, including AWS, Azure, Google Cloud Platform, and others. It also works with on-premises and hybrid cloud environments.
Immutable Infrastructure: Terraform follows the immutable infrastructure paradigm, where infrastructure changes are made by creating entirely new resources rather than modifying existing ones.
State Management: Terraform maintains a state file that keeps track of the current state of the infrastructure. This file helps Terraform understand what changes need to be applied and in what order.
Resource Graph: Terraform builds a dependency graph of all resources defined in the configuration, enabling it to determine the correct order for resource provisioning.

Getting Started with Terraform

Installation

To use Terraform, you need to install it on your local machine. You can download the latest version from the official website: Terraform Downloads.

Basic Concepts

Providers
Providers are plugins that enable Terraform to interact with various infrastructure platforms. Each provider corresponds to a specific cloud or service. For example, there are providers for AWS, Azure, and Google Cloud.
Resources
Resources are the building blocks of your infrastructure. They represent the components you want to create, such as virtual machines, networks, or storage buckets.
Variables
Variables allow you to parameterize your configurations. They make it easy to reuse code and create more flexible and dynamic infrastructure.
Outputs
Outputs define values that are exposed after the infrastructure is provisioned. These values can be useful for other Terraform configurations or scripts.
Modules
Modules are reusable sets of Terraform configurations. They enable you to encapsulate and share infrastructure components.

Terraform Configuration Language

Terraform uses HashiCorp Configuration Language (HCL) for writing configuration files. Here's a simple example of a Terraform configuration file that creates an AWS EC2 instance:

# main.tf

provider "aws" {
  region = "us-west-2"
}

resource "aws_instance" "example" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"
}

In this example:

provider "aws" specifies the AWS provider and the region.
resource "aws_instance" defines an EC2 instance, specifying the Amazon Machine Image (AMI) and instance type.
Terraform Commands
terraform init: Initializes a Terraform working directory, downloading the necessary providers and modules.

terraform init

terraform plan: Creates an execution plan, showing the changes Terraform will make to your infrastructure.

terraform plan

terraform apply: Applies the changes described in the Terraform configuration.

terraform apply

terraform destroy: Destroys the infrastructure defined in the Terraform configuration.

terraform destroy

Managing State

Terraform uses a state file to keep track of the infrastructure it manages. By default, the state is stored locally in a file named terraform.tfstate. For production use, it is recommended to use remote state storage for collaboration and better state management.

Example: AWS S3 Bucket

Let's create a simple Terraform configuration to provision an AWS S3 bucket.

# main.tf

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "example" {
  bucket = "my-terraform-bucket"
  acl    = "private"
}

To apply this configuration, run:

terraform init
terraform apply

Terraform will prompt you to confirm the changes before applying. Once confirmed, it will create the specified S3 bucket.

Example: Azure Virtual Machine

Here's an example of provisioning an Azure Virtual Machine using Terraform.

# main.tf

provider "azurerm" {
  features = {}
}

resource "azurerm_resource_group" "example" {
  name     = "myResourceGroup"
  location = "East US"
}

resource "azurerm_virtual_network" "example" {
  name                = "myVnet"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
}

resource "azurerm_subnet" "example" {
  name                 = "mySubnet"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.1.0/24"]
}

resource "azurerm_network_interface" "example" {
  name                = "myNIC"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name

  ip_configuration {
    name                          = "myNicConfiguration"
    subnet_id                     = azurerm_subnet.example.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_virtual_machine" "example" {
  name                  = "myVM"
  location              = azurerm_resource_group.example.location
  resource_group_name   = azurerm_resource_group.example.name
  network_interface_ids = [azurerm_network_interface.example.id]
  vm_size               = "Standard_DS1_v2"

  storage_image_reference {
    publisher = "MicrosoftWindowsServer"
    offer     = "WindowsServer"
    sku       = "2019-Datacenter"
    version   = "latest"
  }

  storage_os_disk {
    name              = "myOsDisk"
    caching           = "ReadWrite"
    create_option     = "FromImage"
    managed_disk_type = "Premium_LRS"
  }

  os_profile {
    computer_name  = "myVM"
    admin_username = "adminuser"
    admin_password = "Password1234!"
  }

  os_profile_windows_config {
    provision_vmagent = true
  }
}

To apply this configuration, run:

terraform init
terraform apply

This example creates an Azure Virtual Machine with associated resources.

Conclusion

Terraform is a powerful tool for managing infrastructure as code, providing a consistent and reproducible way to create, modify, and destroy infrastructure across different cloud providers. This guide covers the basics of Terraform, from installation and configuration to creating and managing resources. As you become more familiar with Terraform, you can explore advanced features, such as modules, remote state, and conditionals, to further enhance your infrastructure deployment and management workflows.

Completed Hacktoberfest 2020!

prakhyatkarri — Sat, 17 Oct 2020 11:44:56 +0000

What I Learned From Hacktoberfest

This is my second year participating in this Hackathon. Last year, this event helped me in getting exposed to Open Source culture and collaboration. I believe this year it taught me to try to deliver quality over quantity. I have invited some of my friends to participate in this event this year and I believe they felt the same.

I know there are many people who might be participating in this event for a Tee or Goodies. If you're one of them, go for it. But, once you reach your target 4 PRs, try to focus on high quality contributions where Projects really need your help. These contributions should focus on solving issues rather than just counting towards Hacktoberfest qualification. These contributions add value to both projects and your mileage in promoting Open Source collaboration. It can be just 1 PR, but it could bring utmost satisfaction when accepted by Maintainer. Enough said. Happy Hacking!

DEV Community: prakhyatkarri

From a Unified Bronze Layer to Multiple Silver Layers: Streamlining Data Transformation in Databricks Unity Catalog

Introduction

Overview of the Layers

Step-by-Step Guide

1. Ingesting Data into the Raw Layer

Key Points:

2. Bronze Layer: Initial Cleansing and Normalization

Key Steps:

3. Transforming Bronze Data into Multiple Silver Layers

Silver Layer 1: Customer Data Cleansing and Aggregation

Silver Layer 2: Product Data Transformation

Key Points for Silver Layer:

4. Managing Unity Catalog Tables

5. Scheduling and Automation of Pipelines

Conclusion

Unity Catalog Overview

Delta Lake Guide

Auto Loader for Data Ingestion

PySpark for Data Transformations

Databricks Jobs and Workflows

Azure Data Lake Integration

Mastering Terraform: Best Practices for Scalable, Secure, and Reliable Infrastructure as Code

Unlocking the Power of Infrastructure as Code: An In-Depth Introduction to Terraform for Modern DevOps

What is Terraform?

Getting Started with Terraform

Installation

Basic Concepts

Terraform Configuration Language

Managing State

Example: AWS S3 Bucket

Example: Azure Virtual Machine

Conclusion

Completed Hacktoberfest 2020!

What I Learned From Hacktoberfest