DEV Community: Kunal Das

Setting Up a Comprehensive Python Build Validation Pipeline in Azure DevOps

Kunal Das — Thu, 03 Oct 2024 16:49:00 +0000

Setting Up a Comprehensive Python Build Validation Pipeline in Azure DevOps

Hey there,
Almost everyone who deploys python code wants to have it thoroughly checked and follow proper standards but sometimes setting up a code quality pipeline in your CI-CD process can be cumbersome due to various reasons.

Mainly,

You have to setup for each individual pipelines which can be an issue since many of us might have multiple repo multiple project and we need to have it standardize across projects to make sure it just works without tinkering in every other repo
There are plenty of tools and finding the right set of tools and also making sure a certain set of tools work together without giving any issues
Publishing a output also matters which will help the developers easily see what is lacking and they can start improving on the same.

Python developers and DevOps enthusiasts! Today, we're going to walk through setting up a robust build validation pipeline for your Python projects using Azure DevOps. This guide will help you ensure code quality, maintain consistency across multiple repositories, and streamline your development process - all without relying on external tools.

I was tasked not long ago to find a FREE solution without any external tool which should have below features

Must work with multiple repo.
Must provide meaningful output.
Should improve the overall Code Quality
Developers can run locally as well before publishing to the testing tool
All code in the main branch must follow the best in class industry standards

While I started developing the build validation pipeline ( Azure DevOps term basically means successful run of a pipeline to merge the pull request to a particular branch) I started looking for coding standards mainly for python I stumbled upon many articles and was in a bit of confusion which one to follow, but respecting the popularity and unanimous use of PEP-8 made the decision much easier.

Ref :https://peps.python.org/pep-0008/

Ok on top of a pep-8 validator I thought of adding isort which is another great tool to validate if your imports are correctly sorted and on top of this I added a few things extra like BLACK
if we look at the black defination we see

"Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. You will save time and mental energy for more important matters."

Ref : https://pypi.org/project/black/

So from code quality part I ended up using three tools or validators

Black (code formatter)
isort (import sorter)
Flake8 (linter)

I also added pytest and code coverage checks on this pipeline just to make sure developers are diligent to write unit test cases before they push the code to main branch

pytest (testing framework)
Code coverage analysis

The steps here are for Azure Devops Pipeline but it can be easily replicated to githuba ctions or any other ci-cd tools with very minimal to minor tweaking

So let's look at the pipeline that I have been telling till now

Step-by-Step Setup

Step 1: Create the Azure Pipeline YAML File

First, create a file named azure-pipelines.yml in the root of your repository. This file will define our pipeline.

trigger:
  - none

variables:
  - name: sourcePath
    value: 'sourcecode'
  - name: coverageThreshold
    value: 80
  - group: Tokens
  - name: CodeDirectory
    value: '$(System.DefaultWorkingDirectory)'

pool:
  vm../images/image: 'ubuntu-latest'

jobs:
- job: BuildTestAndAnalyze
  displayName: '🚀 Build, Test, and Analyze'
  steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.9'
    displayName: '🐍 Use Python 3.9'

  # More steps will be added here

This sets up the basic structure of our pipeline, specifying the Python version and defining some variables we'll use later.

Now if you see ths issue we need this build validation pipeline in multiple repositories which is a bit tricky since we all know if we add this yaml to a particular repo then it will only work on that and not any other repo instead I wanted it to be a seperate but clone the repo for which it get's triggered in the pull request.

The solution? 
look at he below step :)

Step 2: Clone the PR Repository

Add this step to clone the PR repository:

  - bash: |
      echo "🚀 Initiating repository clone process..."
      REPO_URL=$(System.PullRequest.SourceRepositoryURI)
      AUTH_URL=$(echo $REPO_URL | sed "s|cyncly-engineering@|$(Code_Read_PAT)@|")
      FULL_BRANCH=$(System.PullRequest.SourceBranch)
      BRANCH=$(echo $FULL_BRANCH | sed 's|^refs/heads/||')
      CLONE_DIR=$(System.DefaultWorkingDirectory)/cloned_repo

      git clone --branch "$BRANCH" "$AUTH_URL" "$CLONE_DIR"

      if [ $? -eq 0 ]; then
        echo "✅ Git clone successful"
        echo "##vso[task.setvariable variable=CodeDirectory]$CLONE_DIR"
      else
        echo "❌ Git clone failed"
        exit 1
      fi
    displayName: '🔄 Clone PR Repository'

This step clones the PR repository, allowing us to work with the latest code.

this is a bit tricky spart and for other ci-cd tool we need to modify this the most

Step 3: Install Dependencies

Next, let's install the project dependencies and our code quality tools:

  - script: |
      echo "🔧 Setting up Python environment..."
      python -m pip install --upgrade pip
      if [ -f $(sourcePath)/requirements.txt ]; then
        pip install -r $(sourcePath)/requirements.txt
      fi
    workingDirectory: $(CodeDirectory)
    displayName: '📦 Install Project Dependencies'

  - script: |
      echo "🛠️ Installing code quality and testing tools..."
      pip install black flake8 isort pytest pytest-cov httpx
    displayName: '🛠️ Install Code Quality and Testing Tools'

Step 4: Run Code Quality Checks

Now, let's add steps to run our code quality checks:

  - script: |
      echo "🖌️ Running Black formatter check..."
      black --check --line-length 79 $(sourcePath) 
    displayName: '🖌️ Run Black'
    workingDirectory: $(CodeDirectory)

  - script: |
      echo "🔀 Running isort import sorter check..."
      isort --check-only $(sourcePath)
    displayName: '🔀 Run isort'
    workingDirectory: $(CodeDirectory)

  - script: |
      echo "🔍 Running Flake8 linter..."
      flake8 $(sourcePath)
    displayName: '🔍 Run Flake8'
    workingDirectory: $(CodeDirectory)

These steps will check your code formatting, import sorting, and linting.

Step 5: Run Tests and Generate Coverage Report

Add this step to run tests and generate a coverage report:

  - script: |
      echo "🧪 Running unit tests and generating coverage report..."
      pytest --cov=$(sourcePath) --cov-report=xml --cov-report=html ./tests
    displayName: '🧪 Run Tests with pytest'
    workingDirectory: $(CodeDirectory)

Step 6: Check Code Coverage Threshold

Let's ensure our code meets a minimum coverage threshold:

  - script: |
      echo "📊 Checking code coverage threshold..."
      coverage_percentage=$(python -c "import xml.etree.ElementTree as ET; tree = ET.parse('coverage.xml'); root = tree.getroot(); print(float(root.attrib['line-rate']) * 100)")
      echo "Current code coverage: $coverage_percentage%"
      echo "Coverage threshold: $COVERAGE_THRESHOLD%"
      if (( $(echo "$coverage_percentage < $COVERAGE_THRESHOLD" | bc -l) )); then
        echo "❌ ##vso[task.logissue type=error]Code coverage ($coverage_percentage%) is below the threshold of $COVERAGE_THRESHOLD%"
        exit 1
      else
        echo "✅ Code coverage ($coverage_percentage%) meets or exceeds the threshold of $COVERAGE_THRESHOLD%"
      fi
    displayName: '📊 Check Code Coverage Threshold'
    workingDirectory: $(CodeDirectory)
    env:
      COVERAGE_THRESHOLD: $(coverageThreshold)

Step 7: Publish Coverage Report

Finally, let's publish our coverage report:

  - task: PublishCodeCoverageResults@1
    inputs:
      codeCoverageTool: 'cobertura'
      summaryFileLocation: '$(CodeDirectory)/coverage.xml'
      reportDirectory: '$(CodeDirectory)/htmlcov'
    displayName: '📈 Publish Coverage Report'

  - publish: $(CodeDirectory)/htmlcov
    artifact: CodeCoverageReport
    displayName: '📦 Publish Coverage Report as Artifact'

Setting Up Local Development Environment

To ensure developers can run these checks locally, add the following instructions to your project's README:

Prerequisites

Python 3.9
pip (Python package manager)

Setting Up Your Environment

Clone the repository:

   git clone <repository-url>
   cd <repository-name>

Create a virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

Install dependencies:

   pip install -r sourcecode/requirements.txt
   pip install black flake8 isort pytest pytest-cov httpx

Running Checks Locally

To ensure your code passes the pipeline checks, run these commands from the project root:

Black (code formatting):

   black --check --line-length 79 sourcecode

isort (import sorting):

   isort --check-only sourcecode

Flake8 (linting):

   flake8 sourcecode

pytest (unit tests and coverage):

   pytest --cov=sourcecode --cov-report=xml --cov-report=html ./tests

Setting Up Pre-commit Hooks

To catch issues before they even make it to the pipeline, set up pre-commit hooks:

Install pre-commit:

   pip install pre-commit

Create a .pre-commit-config.yaml file in your project root:

   repos:
   - repo: https://github.com/psf/black
     rev: 22.3.0
     hooks:
     - id: black
       args: [--line-length=79]
   - repo: https://github.com/PyCQA/isort
     rev: 5.10.1
     hooks:
     - id: isort
   - repo: https://github.com/PyCQA/flake8
     rev: 4.0.1
     hooks:
     - id: flake8

Install the hooks:

   pre-commit install

Configuring isort

To ensure consistent import sorting that aligns with Black's formatting, create a file named .isort.cfg in the sourcecode/ directory:

[settings]
profile = black
multi_line_output = 3
include_trailing_comma = True
force_grid_wrap = 0
use_parentheses = True
ensure_newline_before_comments = True
line_length = 79

ok Now how can we setup this pipeline to make sure any of our code get's scanned properly before it goes to the main branch?

First we need a PAT token to clone any repo across azure devops organisation so to get that first then set up the pipeline as a branch policy.

Step 1: Create a Personal Access Token (PAT)

To clone repositories across your Azure DevOps organization, you'll need a PAT with appropriate permissions. Here's how to create one:

Sign in to your Azure DevOps organization (https://dev.azure.com/{your-organization}).
In the top right corner, click on your profile picture and select "Personal access tokens". ![alt text]../images/image.png)
Click on "New Token".
Give your token a name (e.g., "Build Validation Pipeline").
Set the organization to your Azure DevOps organization.
For expiration, choose an appropriate timeframe (e.g., 1 year).
Under "Scopes", select "Custom defined" and then check the following permissions:
- Code (Read & Write)
- Build (Read & Execute)
Click "Create".
Copy the generated token and store it securely. You won't be able to see it again.

Step 2: Add the PAT as a Pipeline Variable

Now that you have a PAT, you need to add it as a variable in your pipeline:

In Azure DevOps, go to "Pipelines" > "Library".
Click on "+ Variable group".
Name the group (e.g., "BuildValidationTokens").
Add a new variable:
- Name: Code_Read_PAT
- Value: Paste your PAT here
Check the "Keep this value secret" box.
Click "Save".

Step 3: Update Your Pipeline YAML

Update your azure-pipelines.yml to use the variable group:

variables:
  - group: BuildValidationTokens
  # ... other variables ...

# ... rest of your pipeline configuration ...

Step 4: Create a Branch Policy

To ensure this pipeline runs on all pull requests to your main branch:

Go to your Azure DevOps project.
Navigate to "Repos" > "Branches".
Find your main branch (often named main or master).
Click the ellipsis (...) next to the branch name and select "Branch policies".
Under "Build Validation", click "+ Add build policy".
Select your build validation pipeline.
Set "Trigger" to "Automatic".
Set "Policy requirement" to "Required".
Set "Build expiration" as per your preference (e.g., "Immediately").
Click "Save".

Step 5: Configure Branch Protection

To prevent direct pushes to the main branch:

Still in the branch policies page for your main branch.
Under "Require a minimum number of reviewers", check the box.
Set the minimum number of reviewers (e.g., 1 or 2).
Optionally, check "Allow requestors to approve their own changes" based on your team's preferences.
Click "Save".

Step 6: Update Repository Settings

Ensure your repository settings align with your new policy:

Go to "Project settings" > "Repositories".
Select your repository.
Under "Policies":
- Check "Require a minimum number of reviewers".
- Set "Minimum number of reviewers" (should match your branch policy).
- Check "Check for linked work items".
- Check "Check for comment resolution".
- Optionally, check other policies as per your team's needs.
Click "Save".

Step 7: Educate Your Team

Finally, make sure your team understands the new process:

Create documentation explaining the new pipeline and its checks.
Highlight the importance of running checks locally before creating a pull request.
Explain how to interpret and act on the pipeline results.
Encourage the use of pre-commit hooks for catching issues early.

By following these steps, you've now set up a robust system where:

All code changes must go through a pull request.
Each pull request automatically triggers your build validation pipeline.
The pipeline checks code formatting, linting, tests, and coverage.
Pull requests cannot be merged until the pipeline passes and required reviews are completed.

Now in the repo where you have added the build validation just try to merge the code into main branch
you will the build validation pipeline will be automatically added in queue and you can see the logs from the pipeline

After that if we go into the pipeline one step back there will be two important things

on top left there will be a tab called code coverage
upon clicking that you will be able to view line by line coverage report of each of the files

This setup ensures that all code going into your main branch meets your quality standards. Remember to periodically review and update your pipeline as your project evolves and new best practices emerge.

Conclusion

By following these steps, you'll have a robust build validation pipeline that ensures code quality across your Python projects. This setup:

Enforces consistent code style
Catches potential bugs early
Encourages comprehensive testing
Provides clear feedback to developers

Remember, the key to success is continuous improvement. Don't hesitate to iterate on this pipeline as you learn what works best for your team and projects. Happy coding!

It takes lot of time to write up and share with comminity if you think this can benifit you do share the word

Creating Azure Web App CI/CD with Terraform and Azure DevOps

Kunal Das — Sun, 21 Jan 2024 12:38:01 +0000

Step-by-Step Guide: Creating Azure Web App CI/CD with Terraform and Azure DevOps

Reach at : https://heylink.me/kunaldas

Introduction:

In today’s fast-paced development environment, implementing Continuous Integration and Continuous Deployment (CI/CD) is crucial for efficient software delivery. In this tutorial, we will walk through the process of setting up a CI/CD pipeline for an Azure web app using Terraform and Azure DevOps. By following these steps, you can automate the deployment process and ensure high-quality code reaches your production environment.

Prerequisites:

Before we begin, make sure you have the following prerequisites in place:

- An Azure DevOps account

- Access to an Azure subscription

- Basic knowledge of Terraform and Docker

Step 1: Creating Infrastructure with Terraform

To ensure a consistent and repeatable infrastructure setup, we will use Terraform. Terraform allows us to define our infrastructure as code and manage it through version control. Follow these steps to create the necessary infrastructure for your Azure web app:

1. Set up a new Azure DevOps project.

2. Create a new repository within your project and clone it to your local machine.

3. Create a folder structure for your Terraform code, such as “infra/environment1” and “infra/environment2” for two environments.

4. Within each environment folder, create a main.tf file and define the necessary Azure resources, such as resource groups, app services, and databases. Use Terraform modules to ensure a modular and reusable infrastructure setup.

5. Define input variables in a separate variable.tf file to make your infrastructure code dynamic and easily configurable.

6. Initialize Terraform within each environment folder by running the terraform init\ command. This will download the required providers and modules.

7. Create a build pipeline in Azure DevOps to trigger the Terraform deployment. Use the “Terraform CLI” task to execute Terraform commands, passing the appropriate input variables for each environment.

Step 2: CI Steps

In this step, we will set up Continuous Integration (CI) to ensure code quality and generate artifacts for deployment.

1. Configure your source code repository in Azure DevOps to trigger the CI pipeline on code changes.

2. Set up a build pipeline in Azure DevOps with the following steps:

a. Use a task to install the required dependencies for your application, such as Python packages.

b. Run unit tests using your preferred testing framework (e.g., pytest) to ensure code functionality.

c. Use linters (flake8, black) and formatters (isort) to enforce code style consistency.

d. Generate a code coverage report using coverage.py to track the percentage of code covered by tests.

e. Integrate with SonarQube for static code analysis and maintain code quality.

f. Build a Docker image containing your application code and dependencies.

g. Push the Docker image to an Azure Container Registry (ACR) for later use in deployment.

Step 3: CD — Continuous Deployment

Now that we have our infrastructure in place and a reliable CI process, we can proceed to set up Continuous Deployment (CD) to automate the deployment process.

1. Determine the deployment strategy based on your requirements. For example:

— Two environments (dev and prod) can follow a simple deployment flow where the code is deployed to dev first and then promoted to prod after successful testing.

— Three environments (dev, test, and prod) can follow a more complex deployment flow where code moves through each environment sequentially.

2. Create release pipelines in Azure DevOps for each environment, using the appropriate deployment strategy.

3. Configure post-deployment tests specific to each environment to ensure the deployed application is functioning as expected.

4. Add performance testing steps after the deployment to the dev environment to validate system

performance under real-world conditions. Use tools like Apache JMeter or Azure Application Insights to conduct performance tests.

Certainly! Below are the code snippets for each step, following a generic parameterized approach and best practices:

Step 1: Creating Infrastructure with Terraform

1. Set up Azure DevOps pipeline:

— Create a new pipeline in Azure DevOps and select your repository.

— Choose the appropriate trigger for your pipeline, such as a repository branch or pull request.

2. Configure Terraform backend:

— Create an Azure Storage Account to store Terraform state.

— Add the following code to your main.tf\ file within each environment folder:

terraform {
 backend “azurerm” {
 storage_account_name = “<storage_account_name>”
 container_name = “<container_name>”
 key = “<environment>.tfstate”
 }
}

3. Define variables:

variable "environment" {
 type = string
 description = "Environment name"
}
variable "resource_group_name" {
 type = string
 description = "Name of the resource group"
}
# Add more variables as per your infrastructure requirements

— Create a variables.tf\ file within each environment folder to define input variables. Here’s an example:

4. Create infrastructure resources:

resource "azurerm_resource_group" "rg" {
 name = var.resource_group_name
 location = "West US"
}
resource "azurerm_app_service_plan" "app_service_plan" {
 name = "${var.environment}-app-service-plan"
 location = azurerm_resource_group.rg.location
 resource_group_name = azurerm_resource_group.rg.name
 kind = "Linux"
 reserved = true
sku {
 tier = "Basic"
 size = "B1"
 }
}
# Add more resources as per your infrastructure requiremen

Add the following code to create a resource group and app service plan in the main.tf\ file within each environment folder:

- task: UsePythonVersion@0
 inputs:
 versionSpec: '3.x'
 addToPath: true
- task: TerraformInstaller@0
 inputs:
 terraformVersion: 'latest'
- task: TerraformCLI@0
 inputs:
 command: 'init'
 workingDirectory: 'infra/environment1'
- task: TerraformCLI@0
 inputs:
 command: 'apply'
 workingDirectory: 'infra/environment1'
 environmentServiceNameAzureRM: '<connection_name>'
 commandOptions: '-auto-approve'

5. Execute Terraform commands in Azure DevOps pipeline:

Add the following steps to your Azure DevOps pipeline YAML file:

Step 2: CI Steps

1. Configure CI pipeline triggers and variables:

— Define your pipeline triggers and variables in the Azure DevOps YAML file, as per your requirements.

2. Set up build steps:

— Use appropriate task names and commands based on your specific needs. Here’s an example:

- task: UsePythonVersion@0
 inputs:
 versionSpec: '3.x'
 addToPath: true
- task: PythonScript@0
 inputs:
 scriptSource: 'filePath'
 scriptPath: 'scripts/install_dependencies.py'
- task: PythonScript@0
 inputs:
 scriptSource: 'filePath'
 scriptPath: 'scripts/run_unit_tests.py'
- task: CmdLine@2
 inputs:
 script: |
 pip install coverage
 coverage run - source=src -m pytest
 coverage xml -o coverage.xml
- task: SonarQubePrepare@4
 inputs:
 SonarQube: '<SonarQube_connection_name>'
 scannerMode: 'CLI'
 cliProjectKey: '<project_key>'
 cliProjectName: '<project_name>'
 cliSources: 'src'
 extraProperties: |
 sonar.coverage.reportPaths=coverage.x

3. Build and push Docker image:

— Use the appropriate task names and commands based on your specific needs. Here’s an example:

- task: Docker@2
 inputs:
 containerRegistry: '<container_registry_connection_name>'
 repository: '<image_repository_name>'
 command: 'buildAndPush'
 Dockerfile: 'Dockerfile'
 tags: '$(Build.BuildId)'

Step 3: CD — Continuous Deployment

1. Determine deployment strategy and set up release pipelines:

— Based on your deployment strategy, define the release pipelines in Azure DevOps with appropriate stages and triggers.

2. Configure post-deployment tests:

— Use the appropriate tasks and scripts to run post-deployment tests. Here’s an example:

- task: PythonScript@0  
inputs:  
scriptSource: ‘filePath’  
scriptPath: ‘scripts/post\_deployment\_tests.py’  
arguments: ‘-e $(environment)’

- task: PublishTestResults@2  
inputs:  
testResultsFormat: ‘JUnit’  
testResultsFiles: ‘\*\*/test-results.xml’  
searchFolder: ‘$(System.DefaultWorkingDirectory)’

3. Perform performance testing:

— Use the appropriate tasks and scripts to perform performance testing. Here’s an example using Apache JMeter:

- task: DownloadFile@1  
inputs:  
sourceUrl: ‘[https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.4.1.tgz'](https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.4.1.tgz')  
targetPath: ‘$(Pipeline.Workspace)/jmeter/apache-jmeter-5.4.1.tgz’

- task: ExtractFiles@1  
inputs:  
archiveFilePatterns: ‘$(Pipeline.Workspace)/jmeter/apache-jmeter-5.4.1.tgz’  
destinationFolder: ‘$(Pipeline.Workspace)/jmeter/’

- task: CmdLine@2  
inputs:  
script: |  
$(Pipeline.Workspace)/jmeter/apache-jmeter-5.4.1/bin/jmeter -n -t $(Pipeline.Workspace)/jmeter/performance\_test.jmx -l $(Pipeline.Workspace)/jmeter/results.jtl

Note: The provided code snippets serve as examples and may require modifications based on your specific application, environment, and tools being used.

Remember to update the connection names, resource names, and specific command arguments according to your project setup and requirements.

Read my blogs:

Connect with Me:

Streamlining Synapse CI/CD & Dedicated SQL pool with Azure DevOps

Kunal Das — Sun, 21 Jan 2024 12:36:58 +0000

Streamlining Synapse CI/CD & Dedicated SQL pool with Azure DevOps: Best Practices and Implementation Tips

Reach at : https://heylink.me/kunaldas

Streamlining Synapse CI/CD & Dedicated SQL pool with Azure DevOps: Best Practices and Implementation Tips
- What is Synapse Analytics?
- what is the need for CI-CD for synapse analytics?
- What are the different components of Synapse Analytics?
- why it is hard to implement ci cd for synapse analytics?
- Let’s understand the CI -CD Approach:
- Continuous Integration :
- let’s look at the high-level workflow of the things we have done so far!
- CI-CD for dedicated SQL pool
- First, let’s setup the repository for the SQL pool
- IMPLEMENTING THE RELEASE PIPELINE TO DEPLOY THE ABOVE-GENERATED BUILD ARTIFACTS

It is crucial to have a simplified and effective process for developing, testing, and implementing solutions as data and analytics become more and more important for enterprises. Microsoft’s cloud-based analytics solution, Synapse Analytics, has strong data warehousing, machine learning, and integration capabilities. However, without the proper equipment and knowledge, building a Continuous Integration and Continuous Deployment (CI/CD) procedure for Synapse can be difficult. Teams can automate the deployment of Synapse solutions thanks to Azure DevOps’ complete collection of CI/CD pipeline capabilities. Using Azure DevOps to streamline Synapse CI/CD, we will examine best practices and implementation advice in this article.

What is Synapse Analytics?

Microsoft created the cloud-based analytics solution known as Synapse Analytics. It is a platform that gives businesses the ability to ingest, prepare, manage, and provide data for their urgent needs in business intelligence and machine learning. Data integration, big data, data warehousing, and AI are just a few of the tools and services that Synapse Analytics integrates into a single workspace to provide a seamless end-to-end data analytics experience. The platform provides cutting-edge functions like code-free or code-first data integration, big data processing based on SQL and Spark, machine learning, and Power BI reporting capabilities. Synapse Analytics offers a single, end-to-end development and deployment experience while allowing customers to work with their current tools, languages, and frameworks.

what is the need for CI-CD for synapse analytics?

First of all, Synapse Analytics is a cloud-based analytics service that is frequently updated and added to with new capabilities. The danger of potential bugs and other problems is reduced by a well-defined CI/CD process, which makes sure that the most recent changes are integrated and tested in the current system before being deployed to production.

Second, a well-defined CI/CD procedure that guarantees seamless cooperation and effective deployment is crucial because Synapse Analytics frequently involves numerous developers working on various project components.

Lastly, A well-defined CI/CD approach also ensures that the pipeline is tested and verified at every level, resulting in greater quality, more stable solutions, and a quicker time to market. Synapse Analytics frequently entails complicated data processing and integration.

Overall, teams can manage the development and deployment of solutions more effectively while maintaining high quality and lowering the risk of errors and downtime by adopting a CI/CD process for Synapse Analytics with technologies like Azure DevOps.

What are the different components of Synapse Analytics?

A cloud-based analytics service called Synapse Analytics consists of a number of parts that work together to offer a whole analytics solution. Synapse Analytics’ principal parts are:

Synapse Studio: A web-based integrated development environment (IDE) called Synapse Studio offers a centralized workspace for creating, overseeing, and maintaining Synapse applications. It consists of a number of technologies and services, including big data, machine learning, data warehousing, and data integration.

Synapse SQL: Users of Synapse SQL can execute SQL queries against both organized and unstructured data using this distributed SQL engine. It supports serverless and provisioned models, enables petabyte-scale data warehousing, and supports both.

Synapse Spark: Synapse Spark is a big data processing engine that enables customers to use Apache Spark to process massive amounts of data. Big data analytics, offers a high-performance computing environment that supports both batch and real-time data processing.

Synapse Pipelines: Users can design, orchestrate, and monitor data pipelines using the data integration service known as Synapse Pipelines. It can integrate data from numerous sources and supports both code-free and code-first data integration scenarios.

Synapse Serverless: Users can analyze data using SQL queries using Synapse Serverless, a serverless SQL pool, without needing to set up or maintain a specific cluster.

Synapse Dedicated SQL Pool: A dedicated cluster for high-performance data warehousing and analytics is offered via the provided SQL pool known as Synapse.

Synapse Notebooks: Users can work with code and data in a group setting using Synapse Notebooks, a collaborative notebook environment. Python, Scala, and .NET are just a few of the many programming languages it supports.

why it is hard to implement ci cd for synapse analytics?

For a number of reasons, implementing a Continuous Integration and Continuous Deployment (CI/CD) procedure for Synapse Analytics can be difficult.

First off, Synapse Analytics is a sophisticated cloud-based analytics service that combines a variety of parts and offerings, each with specific needs and dependencies. It may be challenging to develop a simplified and effective CI/CD pipeline that manages all the many components of the service due to its complexity.

Second, it can be difficult to develop a testing and deployment procedure that is quick and successful since Synapse Analytics frequently requires processing and handling massive volumes of data. It can take a lot of time to test and validate data pipelines, and handling enormous amounts of data makes it challenging to set up a testing environment that precisely mimics the production environment.

Thirdly, it can be difficult to make sure that everyone is using the most recent code and data because Synapse Analytics is frequently utilized by numerous developers and teams. Version control problems and other issues may result from this, which may slow down the development and deployment process.

Last but not least, For Synapse Analytics one needs experience in a variety of fields, including data integration, warehousing, big data, and machine learning, in order to establish a CI/CD process. Teams may find it difficult to locate the necessary skills and materials to build a reliable and effective CI/CD pipeline as a result.

Overall, For Synapse Analytics one needs to carefully prepare the implementation of a CI/CD process and have a thorough understanding of all the different parts and services involved. To build a streamlined and effective pipeline that can handle the complexity of Synapse Analytics, it’s crucial to collaborate with professionals in data engineering, DevOps, and cloud computing.

Let’s understand the CI -CD Approach:

Resource Groups: Here we have two resource groups, In how enterprise we may have multiple resource groups and multiple synapse workspaces so to deploy that you can use my terraform guide which can be done very easily.
Artifacts: for artifacts deployment, I have used the ARM template deployment approach and deployed all the artifacts in that way.

The most important thing is if you have multiple artifacts linked services then you have to manually edit the ARM templates before deployment for it to work.

3. Dedicated SQL pool: The dedicated SQL pool scripts and stored procedures, views, etc also can be automated using CI-CD but I have used a secondary pipeline for the same, which makes sense as the development of the two will be asynchronous and have a different lifecycle.

Continuous Integration :

So for the example let’s think that we have two synapse workspace which represents two environments.

Now log into the dev synapse workspace by going to the portal and clicking on the open synapse studio dialog box

Once you are in dev synapse studio go to Settings and then GIT configuration.

then click on ‘configure’

select GITHUB or AZURE DEVOPS in the repository type

Then proceed to select the collaboration branch and publish branch.

The collaboration branch and publish branch are two crucial ideas in Synapse Studio that are connected to source control and versioning of Synapse artifacts.

The collaboration branch where numerous developers can work together on a Single workspace is the collaboration branch. Usually, Synapse artifacts are developed, tested, and validated using this branch. The collaboration branch can be updated by developers after they create and modify Synapse items like pipelines, notebooks, data flows, and SQL scripts. When you establish a new workspace in Synapse Studio, the collaboration branch, which is the default branch, is automatically created.

The publish branch however does a different thing, it publishes the ARM template into that branch which contains everything in the artifact be it pipelines, notebooks, and Linked services detail everything!

Once you are done it will look something like this,

Now try to commit all and then publish

This will start generating the ARM templates and save them in the repo mentioned.

If you go to the repo you will see a folder showing your synapse workspace name which will contain two ARM templates.

the build and release concept is kind of this.

So we are using three branch policy

Develop branch: This branch would be the primary branch for developers to work on Synapse artifacts, including pipelines, notebooks, and data flows. Each developer would create their own feature branch from the develop branch, where they would make and commit their changes. When their feature is complete and tested, they would submit a pull request to merge their changes into the develop branch. The development branch would contain the latest tested changes from all the feature branches.

Feature branch: Each developer would create their own feature branch from the develop branch, where they would make and commit their changes. Feature branches are typically named after the feature or issue that they are addressing. Once a developer has completed their changes and testing, they would submit a pull request to merge their changes into the develop branch.

Main branch: The main branch would be the publish branch where you would deploy your Synapse artifacts to a live environment. The main branch would only contain the latest, tested changes that have been approved and merged from the develop branch.

In this setup, developers would use the develop branch to collaborate and integrate their changes before publishing to the main branch. This enables developers to work in parallel while maintaining a single source of truth for the Synapse artifacts. The use of feature branches ensures that changes are isolated and tested before being integrated into the develop branch. Finally, the main branch would contain only the tested and approved changes that are ready for deployment. By using this branch policy, you can help ensure that changes to your Synapse artifacts are properly tested and validated, and that version control is maintained throughout the development process.

Create a pipeline with the below code

# The pipeline copies files from workspace publish branch and publishes artifact
# @author  Kunal Das
# @version 1.0
# @since   10-11-2022
name: 
    -synapse-CI-low
trigger:
- workspace_publish

pool:
  vmImage: ubuntu-22.04

steps:

- task: CopyFiles@2
  displayName: 'Copy Files to Build Pipeline directory'
  inputs:
    SourceFolder: 'synapse-dev'
    TargetFolder: '$(Build.ArtifactStagingDirectory)/ARM'

- task: PublishPipelineArtifact@1
  displayName: 'Publish Pipeline Artifact'
  inputs:
    targetPath: '$(Build.ArtifactStagingDirectory)'
    artifact: drop

This is a simplified YAML pipeline for Synapse CI/CD that has been named ‘synapse-CI-low’. The pipeline triggers whenever a Synapse workspace is published. The pipeline runs on an Ubuntu 22.04 virtual machine.

The first step in the pipeline is to copy files from a folder called ‘synapse-dev’ to a directory called ‘ARM’ in the build pipeline. This step is accomplished using the ‘CopyFiles’ task.

The second and final step in the pipeline is to publish the Synapse artifacts as a pipeline artifacts. This step is accomplished using the ‘PublishPipelineArtifact’ task. The target path for the artifacts is set to the build artifact staging directory, and the artifact is named ‘drop’.

This pipeline is a simple example of a Synapse CI/CD pipeline that copies Synapse artifacts from a source directory and publishes them as a pipeline artifact. This pipeline can be further expanded to include additional steps for building, testing, and deploying the Synapse artifacts.

Now to deploy the ARM template one extension is required, which can be added from the marketplace.

To install the extension follow these steps:

Log in to your Azure DevOps organization and navigate to the Extensions tab in the left-hand menu.
Click on Browse Marketplace to access the Visual Studio Marketplace.
Search for the extension you want to install by typing the name of the extension in the search bar.
Click on the extension from the list of results to open the extension page.
Click on the Get it free button.
Select the Azure DevOps organization where you want to install the extension and review the terms and conditions.
Click on the Install button to install the extension in your Azure DevOps organization.

Now for release, we need to add the main step:

which is workspace deployment, for that we add the below task

steps:
- task: AzureSynapseWorkspace.synapsecicd-deploy.synapse-deploy.Synapse workspace deployment@2
  displayName: 'Synpase deployment task for workspace: synapse-qa'
  inputs:
    TemplateFile: '$(System.DefaultWorkingDirectory)/_Synapse-CI-pipeline/drop/ARM/TemplateForWorkspace.json'
    ParametersFile: '$(System.DefaultWorkingDirectory)/_Synapse-CI-pipeline/drop/ARM/TemplateParametersForWorkspace.json'
    azureSubscription: 'syn-sp'
    ResourceGroupName: 'synapseQA-RG'
    TargetWorkspaceName: 'synapse-qa'
    OverrideArmParameters: '-LS_AKV_QA_properties_typeProperties_baseUrl <Synapse QA AKV URL>'
    FailOnMissingOverrides: true

This is a YAML code snippet for deploying a Synapse workspace using the Synapse Workspace Deployment task in Azure DevOps. The code contains the following steps:

The task is defined with the name ‘Synapse deployment task for workspace: synapse-qa’ and the version number of the Synapse Workspace Deployment task is set to 2.
The template and parameter files for the Synapse workspace are specified. The template file is named ‘TemplateForWorkspace.json’ and the parameters file is named ‘TemplateParametersForWorkspace.json’. These files are located in the default working directory of the pipeline.
The Azure subscription to be used for the deployment is specified as ‘syn-sp’.
The resource group name where the Synapse workspace will be deployed is set to ‘synapseQA-RG’.
The target Synapse workspace name is set to ‘synapse-qa’.
The override ARM parameters are specified using the parameter name and the new value. In this case, the parameter name is ‘-LS_AKV_QA_properties_typeProperties_baseUrl’ and the new value is 'QA-AKV-URL'.
The ‘FailOnMissingOverrides’ option is set to true, which means that the deployment will fail if any of the specified override parameters are not found.

In summary, this code snippet deploys a Synapse workspace to the specified Azure subscription, resource group, and Synapse workspace name. The override ARM parameters allow you to customize the deployment settings for the workspace.

if you notice when we deploy the ARM template we need to change somethings like

1 . Sql pool name

2. Linked Services details

3. Spark pool name

If you have the same name for all these then you may not have to tinker with the ARM template but for others before running the above deployment step we need to modify these.

steps:
- task: PythonScript@0
  displayName: 'Change SQL Pool'
  inputs:
    scriptSource: inline
    script: |
     search_text = 'synapsesqlpooldev'
     replace_text = 'synapsesqlpoolqa'

     with open(r'$(System.DefaultWorkingDirectory)/_Synapse-CI-pipeline/drop/ARM/TemplateForWorkspace.json', 'r') as file:
        data = file.read()
        data = data.replace(search_text, replace_text)
     with open(r'$(System.DefaultWorkingDirectory)/_Synapse-CI-pipeline/drop/ARM/TemplateForWorkspace.json', 'w') as file:
        file.write(data)
     print("SQL pool changed")

This is a YAML code snippet for a PythonScript task in Azure DevOps. The task changes the name of a SQL pool in an ARM template file for a Synapse workspace deployment. The code contains the following steps:

The task is defined with the name ‘Change SQL Pool’ and the task version is set to 0.
The script source is set to ‘inline’, which means that the script is written in the YAML file directly.
The Python script is defined, which replaces the search text ‘synapsesqlpooldev’ with the replacement text ‘synapsesqlpoolqa’ in the ARM template file.
The ARM template file is read and its contents are stored in the ‘data’ variable.
The ‘replace’ method is used to replace the search text with the replacement text.
The modified data is written back to the ARM template file.
A message ‘SQL pool changed’ is printed to the console.

In summary, this code snippet uses a Python script to modify the ARM template file for a Synapse workspace deployment by changing the name of a SQL pool from ‘synapsesqlpooldev’ to ‘synapsesqlpoolqa’. The modified ARM template file is used in the subsequent tasks for deploying the Synapse workspace.

The same kind of scripts can use for Spark pool and linked services as well.

If you want to use Powershell here is the command

Let us change the linked Azure KeyVault for the same.

(Get-Content -path $(System.DefaultWorkingDirectory)/_Synapse-CI-pipeline/drop/ARM/TemplateForWorkspace.json) -replace 'LS_AKV_DEV','LS_AKV_QA' | Set-Content -Path $(System.DefaultWorkingDirectory)/_Synapse-CI-pipeline/drop/ARM/TemplateForWorkspace.json


(Get-Content -path $(System.DefaultWorkingDirectory)/_Synapse-CI-pipeline/drop/ARM/TemplateParametersForWorkspace.json) -replace 'LS_AKV_DEV','LS_AKV_QA' | Set-Content -Path $(System.DefaultWorkingDirectory)/_Synapse-CI-pipeline/drop/ARM/TemplateParametersForWorkspace.json

The first command replaces the string ‘LS_AKV_DEV’ with ‘LS_AKV_QA’ in the contents of the ARM template file named ‘TemplateForWorkspace.json’.
The second command replaces the same string in the contents of the ARM template parameter file named ‘TemplateParametersForWorkspace.json’.
The ‘-path’ parameter specifies the file path for each file, which is set to the default working directory in the build pipeline.
The ‘-replace’ parameter is used to search for the specified string and replace it with the replacement string in each file.
The ‘|’ (pipe) character is used to send the modified contents to the ‘Set-Content’ command, which writes the modified contents back to the same file.
The modified ARM template files are used in the subsequent tasks for deploying the Synapse workspace.

In summary, this PowerShell script code uses the ‘Get-Content’ and ‘Set-Content’ commands to modify the contents of two ARM template files for a Synapse workspace deployment in Azure DevOps. The ‘replace’ parameter is used to replace a string in each file, and the modified files are used in the subsequent deployment tasks.

Now if you have some triggers in the synapse workspace it will not deploy as before deployment triggers must be turned off, to achieve this we can add the below task just before deployment

steps:
- task: AzureSynapseWorkspace.synapsecicd-deploy.toggle-trigger.toggle-triggers-dev@2
  displayName: 'Toggle Azure Synapse Triggers'
  inputs:
    azureSubscription: 'syn-sp'
    ResourceGroupName: 'Synapseqa-RG'
    WorkspaceName: 'synapse-qa'
    ToggleOn: false

This pipeline step uses the Azure Synapse CI/CD extension to turn off triggers for a Synapse workspace. The task is named “Toggle Azure Synapse Triggers” and the version being used is 2. The task has several inputs including:

azureSubscription: The Azure subscription that contains the Synapse workspace.
ResourceGroupName: The name of the resource group where the Synapse workspace is located.
WorkspaceName: The name of the Synapse workspace where the triggers will be turned off.
ToggleOn: A boolean value indicating whether to turn on or off the Synapse triggers. In this case, it is set to false, meaning the triggers will be turned off.

Once you do all these steps you are finally ready to deploy the synapse workspace!!!

let’s look at the high-level workflow of the things we have done so far!

Developers create or modify Synapse artifacts, such as notebooks, and pipelines, in the development environments.
The changes are pushed to a source control repository, such as Azure DevOps Repos, GitHub, or Bitbucket.
A build pipeline is triggered, which compiles the changes, and creates an artifact, such as an ARM template.
The artifact is published to a release pipeline, which deploys it to the Synapse workspace.
The deployment process may involve creating or updating Synapse artifacts, deploying resources to Azure, and configuring the Synapse workspace.
After the deployment is completed, the triggers for Synapse pipelines, notebooks, and triggers may need to be toggled on or off.
If any issues are found during testing, the pipeline may be rolled back or the code may be fixed and re-deployed.

But this whole CI-CD pipeline will take care of the spark notebooks pipelines and additionally the Linked services and IRs as well but one major thing is the Stored procedures views etc from the dedicated SQL pool, we need some way to publish these changes can be taken care of by using a second Pipeline,

CI-CD for dedicated SQL pool

Let’s look at the high-level workflow

Sure, here’s a detailed explanation of the CI/CD workflow for a dedicated SQL pool using Azure Data Studio:

Connect to dedicated SQL pool: First, connect to the dedicated SQL pool using Azure Data Studio. This can be done by selecting the dedicated SQL pool in the Object Explorer and providing the necessary login credentials.
Create a project: Once connected, create a new project in Azure Data Studio by selecting File -> New Project. Choose the appropriate project type based on the requirements of the dedicated SQL pool.
Add source control: Add the project to a source control repository such as Git by selecting View -> Source Control and following the prompts to initialize the repository and commit the project.
Build pipeline: Create a build pipeline in Azure DevOps that takes the *.sqlproj file from the source control repository and builds a DACPAC file using the vsbuild Task. This task compiles the SQL scripts and T-SQL code into a single package that can be deployed to the dedicated SQL pool.
Release pipeline: Create a release pipeline in Azure DevOps that deploys the DACPAC file to the next environment, such as the QA environment. This can be done using the dacpac Deploy Task. This task deploys the DACPAC file to the specified SQL Server instance or dedicated SQL pool, and can also handle any necessary pre- or post-deployment scripts.

Here is the complete workflow diagram of the same.

First, let’s setup the repository for the SQL pool

Create or use an existing Azure DevOps project.
Click on Repos from the sidebar.
Click on New Repository and fill in the necessary details, such as the repo name. In this case, the repo name used in this example is “Synapse_SQLpool”. Click on Create.
Configure Azure Data Studio (ADS) to connect to the above-created repository. Open ADS and select the New Connection option from the welcome page.

5. A new pop-up window will come, here put all the details required for the connection.

6. Click on connect and then wait for the browser window to come on screen, here u can normally log in with your azure credentials and after successful login, you will be redirected to the data studio.

7. At this point if you click on the Connections side pane on the left you will definitely see the connection sub menu there,

Now select the server connection from here and then right-click and select ‘create project from database’ option,

Keep the settings as shown in the below image but select the folder and project name appropriately.

Once you are done with it you will see a project from the left side pane

Now the only step is to add the project to the Azure repo, there are plenty of ways we can do that, but here inside Azure Data studio itself, you can do the same by going into the left pane source control option.
Click on the three-dot menu and then select the add remote option,

Here u just have to put the azure repo link, if it fails somehow open the terminal by pressing “ctrl + `” and then run the git init command which will initialize the directory,
Once u put the azure repo link whatever changes are there in your project can be pushed to the azure repo easily.
Verify the remote repo by the command ‘git remote -v’
The Sync option can also be used as it will automatically pull and then push the repo from the azure repo to the local system,

Open the azure repo and navigate to your repo and see the files updated,

Now we have to create a build pipeline for that,

go to pipeline and a new pipeline,

select Azure repo and locate your repository.

The third step would be just selecting the starter pipeline option.

Then in the yaml file paste the below code, `yaml # @Author : Kunal Das # Date : 30-12-2022

variables:
poolstagingarea: $(Build.ArtifactStagingDirectory)\poolstaging
BuildConfiguration: release
SQLPoolartifactname: AzureSQLPool
SQLPooldacpacfile: $(System.ArtifactsDirectory)\$(SQLPoolartifactname)\synapseSQLpoolDEV.sqlproj

trigger:

main

pool:
vmImage: windows-2019

stages:

stage: Pooldacpac displayName: 'Build dacpac'

jobs:
- job: 'Builddacpac'
displayName: 'Build SQL Pool dacpac'

  steps:
  - task: VSBuild@1
    displayName: 'Builds the dacpac'
    inputs:
      solution: synapseSQLpoolDEV.sqlproj
      configuration: $(BuildConfiguration)


  - task: PublishBuildArtifacts@1
    displayName: 'Publishes dacpac as an artifact'
    # Publishes the dacpac as part of an artifact within Azure DevOps
    inputs:
      PathtoPublish: 'bin\$(BuildConfiguration)'
      ArtifactName: $(SQLPoolartifactname)
      publishLocation: 'Container'

And Done! The build pipeline is ready, you may notice I have given the main branch as a trigger at the top so every time someone updates the main branch it triggers the main pipeline.
Save and run the pipeline, you can see all jobs running perfectly!!!

Click on the pipeline and see the job artifact in this case the dacpac file has been published.

IMPLEMENTING THE RELEASE PIPELINE TO DEPLOY THE ABOVE-GENERATED BUILD ARTIFACTS

Now the release pipeline can be created easily by going to pipeline -> release and then selecting create new release pipeline.

Go to the pipeline and add just one task “SQL dacpac deployment”

This task just points to the QA or the next environment and then just takes the dacpac file to deploy it in the next environments.

Then locate and select the dacpac file.

Pass an additional argument in the field Additional SqlPackage.exe Arguments as /p: BlockOnPossibleDataLoss=false
Just save and run the pipeline.
The job will run and deploy the changes to the next environment,

In conclusion, the effectiveness and collaboration of data engineering and BI projects can be significantly enhanced by connecting Azure DevOps with Azure Synapse Analytics workspace. You may connect your Azure DevOps project with your Synapse workspace by following the instructions in this article, which will also help you optimize your CI/CD pipeline for better data asset distribution and maintenance. To ensure a seamless and successful implementation, it is crucial to test carefully and make any necessary adjustments. Your firm can benefit from a smooth DevOps and data analytics process with proper design and implementation.

Read my blogs:

Connect with Me:

A comprehensive list of Azure Best Practices

Kunal Das — Sun, 21 Jan 2024 12:36:57 +0000

Azure Best Practices

Reach at : https://heylink.me/kunaldas

Azure Best Practices
- Security:
- Enable Azure AD Conditional Access and enforce MFA
- Disable RDP/SSH from the Internet
- Management:
- Operations:
- Cost Optimization:
- Services:
- Database
- AppService:
- Conclusion:
- Credits:

Azure best practices assist businesses in making the most of Azure resources to create and maintain scalable, cost-efficient, and secure solutions on the Microsoft Azure cloud. Having the correct Azure best practices can make or ruin a firm because Azure serves as the foundation of many modern organizations. I’ve covered the fundamental best practices that every Azure administrator needs to be aware of in this document. I’ve also provided advice on how to create a secure, reliable, and effective Azure infrastructure. These best practices must be followed in tandem because none of them by themselves can adequately secure the systems. You must select the best security alternatives based on your surroundings and demands, as is always the case.

Security:

the most important thing before anything else is security, The recommendations below can help assure more robust Azure security, but they cannot serve as a complete substitute. The top practices that I believe will help you strengthen and safeguard your Azure are listed below.

Enable Azure AD Conditional Access and enforce MFA

Azure AD Conditional Access assists administrators in enabling users to be productive whenever and wherever they choose while simultaneously safeguarding the assets of the company. Automating access control based on security, business, and compliance requirements are made possible via conditional access. Access to data and applications is protected by the addition of a crucial security layer provided by Azure AD Multi-Factor Authentication. To design an Azure Active Directory conditional access deployment and evaluate deployment considerations for Azure AD Multi-Factor Authentication, go to these Azure documentation sites (MFA).

Disable RDP/SSH from the Internet

Use JIT and Azure Bastion Do not expose RDP and SSH access over the Internet in order to provide remote access to Windows and Linux VMs in Azure from anywhere in the world without compromising security. To provide safe remote access to Azure virtual machines, use one of these techniques:

✓ Enable site-to-site VPN or ExpressRoute connections for Just-In-Time (JIT) VM access:

JIT offers time-limited VM access through RDP and SSH, which lessens the exposure to brute force assaults. In essence, Network Security Groups (NSGs) lock off RDP and SSH ports and only permit authorized users to access them for a predetermined time. Users can request access to JIT VMs using Azure AD and role-based Access Control (RBAC) permissions.

✓ Configure Azure Bastion inside your virtual network.

Direct RDP/SSH communication to your VMs over TLS from the Azure interface is made possible by Azure Bastion. The PaaS service Azure Bastion does away with the requirement for VMs to have public IP addresses, agents, or specialized client software.

Secure privileged access with Azure AD PIM

Azure AD Privileged Identity Management enables management, control, and monitoring of access to vital resources in Azure, Microsoft 365, and Intune (PIM). Admins must activate or elevate their privilege through PIM in order to use it for a brief period of time.

Management:

Use Resource Groups; Tag individual resources:

Create a resource group strategy that meets your company’s needs, then plan, create, and implement it. Resource groups, which are logical collections of Azure resources, containerize related resources in a group for administration simplicity, security, and cost tracking for your workloads.

A best practice resource group strategy should include the following: Group resources by

Environment: prod, dev, uat,stg,perf,sit

Application: BI,DWH

Business Unit: ML,DS

Follow a well-defined naming standard

Include resource type, application or business unit, environment, Azure region, and consecutive entity number etc.

For example, dev-ause-asy-01 means Azure Synapse Workspace in Australia South East Region which is in DEV environment.

Leverage Resource tagging

Tag all resources inside resource groups to communicate valuable information to your teams, discover resources, and manage costs and it is easy to delete resources as well.

Operations:

Use Azure Advisor:

Azure Advisor offers individualised, practical advice for cost, security, dependability, operational excellence, and performance. Organizations can optimise their deployments with Azure Advisor in accordance with Microsoft best practises. These suggestions are based on Microsoft best practises that are successful for most businesses and were compiled using resource configuration analysis and usage telemetry in your Azure tenant.

Cost Optimization:

Use reserved instances:

Reserved instances are useful in certain circumstances, such as when you frequently employ the same VM size across several VMs. Domain controllers operating on Azure are an excellent example. On these VMs, three-year reserved instances offer savings of up to 72%.

Delete unneeded resources:

After VMs are decommissioned, orphan resources are frequently forgotten about and left in a tenant. These resources are pricey! Common examples of these Azure orphan resources include network cards and OS discs. Fortunately, the Azure Portal assists in reminding administrators to delete unnecessary resources at provisioning and deletion time.

Services:

In this section let us see in detail the most used resources and the best way to use those,

➢ Virtual Machine

✓ Enforce multi-factor authentication (MFA) and complex passwords. MFA can help limit the threat of compromised credentials. Complex passwords help reduce the effectiveness of brute-force password attacks. ✓ Use just-in-time (JIT) virtual machine access. JIT access works with NSGs and the Azure firewall and helps you layer in role-based access controls (RBAC) and time-bindings on access to virtual machines.

✓ Have a patch process in place. If you’re not patching your workloads, all your other efforts may be for nothing. A single unpatched vulnerability can lead to a breach. A patch process to keep your operating systems and applications up to date helps you mitigate this risk.

✓ Lock down administrative ports. Unless necessary, restrict access to SSH, RDP, WinRM, and other administrative ports.

✓ Use the Azure firewall and network security groups (NGSs) to limit access to workloads. Consistent with the principle of least privilege, use NSGs and the Azure firewall to restrict workload access.

✓ Apply the Latest OS Patches Ensure that the latest OS patches available for Microsoft Azure virtual machines are applied.

✓ Approved Azure Machine Image in Use Ensure that all your Azure virtual machine instances are launched from approved machine images only.

✓ Enable Auto-Shutdown Configure your Microsoft Azure virtual machines to automatically shut down on a daily basis.

➢ Storage

✓ Restrict database and storage access. UseFirewalls and access controls to limit what level of access users, devices, and services have to your databases and storage blobs.

✓ Leverage auditing. Turn on auditing for your Azure databases. Doing so enables you to gain visibility into all database changes.

✓ Configure threat detection for Azure SQL. If you use Azure SQL, activating threat detection helps you identify security issues faster and limit dwell time.

✓ Set log alerts in Azure Monitor. It isn’t enough to simply log events. Make sure you are alerting against security-related events in Azure Monitor so you can remediate issues quickly (and automatically when possible).

✓ Enable Azure Defender for your storage accounts. Azure Defender provides you with hardening and securing your Azure storage accounts.

✓ Use soft deletes. Soft deletes help you ensure data is still retrievable (for 14 days) in the event a malicious actor (or user error) leads to data you wanted to keep — getting deleted.

✓ Use shared access signatures (SAS). SAS enables you to implement granular access controls and time limits on client access to data.

✓ Disable Anonymous Access to Blob Containers Ensure that anonymous access to blob containers is disabled within your Azure Storage account.

✓ Disable public access to storage accounts with blob containers Ensure that public access to blob containers is disabled for your Azure storage accounts to override any ACL configurations allowing access.

➢ Network

✓ Encrypt data in transit. As we mentioned in the encryption and data security section: encryption of data in transit (and at rest) is a must. Leverage modern encryption protocols for all network traffic.

✓ Implement zero trust. By default, network policies should deny access unless there is an explicit allow rule.

✓ Limit open ports and Internet-facing endpoints. Unless there is a well-defined business reason for a port to be open or workload to be Internet-facing, don’t let it happen.

✓ Monitor device access. Monitoring access to your workloads and devices (e.g. using a SIEM or Azure Monitor) helps you proactively detect threats

✓ Segment your networks. Logical network segmentation can help improve visibility, make your networks easier to manage and limit east-west movement in the event of a breach.

✓ Check for NSG Flow Log Retention Period Ensure that the Network Security Group (NSG) flow log retention period is greater than or equal to 90 days.

✓ Check for Network Security Groups with Port Ranges Ensure there are no network security groups with a range of ports opened to allow incoming traffic.

✓ Enable DDoS Standard Protection for Virtual Networks Ensure that DDoS standard protection is enabled for production Azure virtual networks.

✓ Monitor Network Security Group Configuration Changes Network security group changes have been detected in your Microsoft Azure cloud account.

✓ Review Network Interfaces with IP Forwarding Enabled Ensure that the Azure network interfaces with IP forwarding enabled are regularly reviewed.

➢ Database

➢ Cosmos DB:

✓ Enable Advanced Threat Protection Ensure that Advanced Threat Protection is enabled for all Microsoft Azure Cosmos DB accounts.

✓ Enable Automatic Failover Enable automatic failover for Microsoft Azure Cosmos DB accounts.

✓ Restrict Default Network Access for Azure Cosmos DB Accounts Ensure that default network access (i.e. public access) is denied within your Azure Cosmos DB accounts configuration.

➢ SQL:

✓ Advanced Data Security for SQL Servers Ensure that Advanced Data Security (ADS) is enabled at the Azure SQL database server level.

✓ Check for Publicly Accessible SQL Servers Ensure that Azure SQL database servers are accessible via private endpoints only.

✓ Check for Sufficient Point in Time Restore (PITR) Backup Retention Period Ensure there is a sufficient PITR backup retention period configured for Azure SQL databases.

✓ Check for Unrestricted SQL Database Access Ensure that no SQL databases allow unrestricted inbound access from 0.0.0.0/0 (any IP address).

✓ Configure “AuditActionGroup” for SQL Server Auditing Ensure that “AuditActionGroup” property is well configured at the Azure SQL database server level.

✓ Configure Emails for Vulnerability Assessment Scan Reports and Alerts Ensure that “Send scan reports to” setting is configured for SQL database servers.

✓ Detect Create, Update, and Delete SQL Server Firewall Rule Events SQL Server firewall rule changes have been detected in your Microsoft Azure cloud account.

✓ Enable All Types of Threat Detection on SQL Servers Enable all types of threat detection for your Microsoft Azure SQL database servers.

✓ Enable Auditing for SQL Servers Ensure that database auditing is enabled at the Azure SQL database server level.

✓ Enable Auto-Failover Groups Ensure that your Azure SQL database servers are configured to use auto-failover groups.

✓ Enable Automatic Tuning for SQL Database Servers Ensure that Automatic Tuning feature is enabled for Microsoft Azure SQL database servers.

✓ Enable Transparent Data Encryption for SQL Databases Ensure that Transparent Data Encryption (TDE) is enabled for every Azure SQL database.

✓ Enable Vulnerability Assessment Email Notifications for Admins and Subscription Owners Ensure that the Vulnerability Assessment setting “Also send email notification to admins and subscription owners” is enabled. Enable Vulnerability Assessment Periodic Recurring Scans Ensure that the Vulnerability Assessment Periodic Recurring Scans setting is enabled for SQL database servers.

✓ Enable Vulnerability Assessment for Microsoft SQL Servers Ensure that Vulnerability Assessment is enabled for Microsoft SQL database servers.

✓ SQL Auditing Retention Ensure that SQL database auditing has a sufficient log data retention period configured.

✓ Use Azure Active Directory Admin for SQL Authentication Ensure that an Azure Active Directory (AAD) admin is configured for SQL authentication.

✓ Use BYOK for Transparent Data Encryption Use Bring Your Own Key (BYOK) support for Transparent Data Encryption (TDE).

➢ PostgreSQL

✓ Check for PostgreSQL Log Retention Period Ensure that PostgreSQL database servers have a sufficient log retention period configured.

✓ Check for PostgreSQL Major Version Ensure that PostgreSQL database servers are using the latest major version of PostgreSQL database.

✓ Enable “CONNECTION_THROTTLING” Parameter for PostgreSQL Servers Ensure that “connection_throttling” parameter is set to “ON” within your Azure PostgreSQL server settings.

✓ Enable “LOG_CHECKPOINTS” Parameter for PostgreSQL Servers Enable “log_checkpoints” parameter for your Microsoft Azure PostgreSQL database servers.

✓ Enable “LOG_CONNECTIONS” Parameter for PostgreSQL Servers Enable “log_connections” parameter for your Microsoft Azure PostgreSQL database servers.

✓ Enable “LOG_DISCONNECTIONS” Parameter for PostgreSQL Servers Enable “log_disconnections” parameter for your Microsoft Azure PostgreSQL database servers.

✓ Enable “LOG_DURATION” Parameter for PostgreSQL Servers Enable “log_duration” parameter on your Microsoft Azure PostgreSQL database servers.

✓ Enable “log_checkpoints” Parameter for PostgreSQL Flexible Servers Enable “log_checkpoints” parameter for your Microsoft Azure PostgreSQL flexible database servers.

✓ Enable In-Transit Encryption for PostgreSQL Database Servers Ensure that in-transit encryption is enabled for your Azure PostgreSQL database servers.

✓ Enable Infrastructure Double Encryption for Single Servers Ensure that infrastructure double encryption is enabled for Single Server Azure PostgreSQL database servers.

✓ Use Azure Active Directory Admin for PostgreSQL Authentication Ensure that an Azure Active Directory (AAD) admin is configured for PostgreSQL authentication.

➢ MySQL:

✓ Configure TLS Version for MySQL Flexible Database Servers Ensure that the ‘tls_version’ parameter is set to a minimum of ‘TLSV1.2’ for all MySQL flexible database servers.

✓ Enable In-Transit Encryption for MySQL Servers Ensure that in-transit encryption is enabled for your Azure MySQL database servers.

➢ AppService:

✓ Check for Latest Version of .NET Framework Enable HTTP to HTTPS redirects for your Microsoft Azure App Service web applications.

✓ Check for Latest Version of Java Ensure that Azure App Service web applications are using the latest stable version of Java.

✓ Check for Latest Version of PHP Ensure that Azure App Service web applications are using the latest version of PHP.

✓ Check for Latest Version of Python Ensure that Azure App Service web applications are using the latest version of Python.

✓ Check for Sufficient Backup Retention Period Ensure there is a sufficient backup retention period configured for Azure App Services applications.

✓ Check for TLS Protocol Latest Version Ensure that Azure App Service web applications are using the latest version of TLS encryption.

✓ Disable Plain FTP Deployment Ensure that FTP access is disabled for your Azure App Services web applications.

✓ Disable Remote Debugging Disable Remote Debugging feature for your Microsoft Azure App Services web applications.

✓ Enable Always On Ensure that your Azure App Services web applications stay loaded all the time by enabling the Always On feature.

✓ Enable App Service Authentication Ensure that App Service Authentication is enabled within your Microsoft Azure cloud account.

✓ Enable Application Insights Ensure that Azure App Services applications are configured to use Application Insights feature.

✓ Enable Automated Backups Ensure that all your Azure App Services applications are using the Backup and Restore feature.

✓ Enable FTPS-Only Access Enable FTPS-only access for your Microsoft Azure App Services web applications.

✓ Enable HTTP/2 Ensure that Azure App Service web applications are using the latest stable version of HTTP.

✓ Enable HTTPS-Only Traffic Enable HTTP to HTTPS redirects for your Microsoft Azure App Service web applications.

✓ Enable Incoming Client Certificates Ensure that Azure App Service web applications are using incoming client certificates.

✓ Enable Registration with Azure Active Directory Ensure that registration with Azure Active Directory is enabled for Azure App Service applications.

✓ Use Key Vaults to Store App Service Application Secrets Ensure that Azure Key Vaults are used to store App Service application secrets.

➢ KeyVault

✓ App Tier Customer-Managed Key In Use Ensure that a Customer-Managed Key is created for your Azure cloud application tier.

✓ Check for Allowed Certificate Key Types Ensure that Azure Key Vault certificates are using the appropriate key type(s).

✓ Check for Azure Key Vault Keys Expiration Date Ensure that your Azure Key Vault encryption keys are renewed prior to their expiration date.

✓ Check for Azure Key Vault Secrets Expiration Date Ensure that your Azure Key Vault secrets are renewed prior to their expiration date.

✓ Check for Certificate Minimum Key Size Ensure that Azure Key Vault RSA certificates are using the appropriate key size.

✓ Check for Key Vault Full Administrator Permissions Ensure that no Azure user, group or application has full permissions to access and manage Key Vaults.

✓ Check for Sufficient Certificate Auto-Renewal Period Ensure there is a sufficient period configured for the SSL certificates auto-renewal.

✓ Database Tier Customer-Managed Key In Use Ensure that a Customer-Managed Key is created for your Microsoft Azure cloud database tier.

✓ Enable AuditEvent Logging for Azure Key Vaults Ensure that AuditEvent logging is enabled for your Microsoft Azure Key Vaults.

✓ Enable Certificate Transparency Ensure that certificate transparency is enabled for all your Azure Key Vault certificates.

✓ Enable Key Vault Recoverability Ensure that your Microsoft Azure Key Vault instances are recoverable.

✓ Enable SSL Certificate Auto-Renewal Ensure that Auto-Renewal feature is enabled for your Azure Key Vault SSL certificates.

✓ Enable Trusted Microsoft Services for Key Vault Access Allow trusted Microsoft services to access your Azure Key Vault resources (i.e. encryption keys, secrets and certificates).

✓ Restrict Default Network Access for Azure Key Vaults Ensure that default network access (i.e. public access) rule is set to “Deny” within your Azure Key Vaults configuration.

✓ Set Azure Secret Key Expiration Ensure that an expiration date is set for all your Microsoft Azure secret keys.

✓ Set Encryption Key Expiration Ensure that an expiration date is configured for all your Microsoft Azure encryption keys.

✓ Web Tier Customer-Managed Key In Use Ensure that a Customer-Managed Key is created for your Microsoft Azure cloud web tier

Conclusion:

There are numerous Azure features and services that require ongoing maintenance in terms of security. There are countless ways to attack a system, and poorly protected systems are the ones that hackers most frequently target. By keeping a few simple things in mind, you can strengthen your network considerably. With some investment and your work, you can make your Azure secure and robust by using a variety of Azure services.

Credits:

https://learn.microsoft.com/en-us/azure/security/fundamentals/best-practices-and-patterns

Read my blogs:

Connect with Me:

Mastering Private Repositories in Enterprise with GitHub

Kunal Das — Sun, 21 Jan 2024 12:36:56 +0000

Mastering Private Repositories in Enterprise with GitHub: A Comprehensive Guide

Reach at : https://heylink.me/kunaldas

Mastering Private Repositories in Enterprise with GitHub: A Comprehensive Guide
- Getting Started with GitHub and Git
- Securing Your Connection with SSH
- Additional Tips and Techniques
- Leverage GitHub’s Issue Tracker
- 2. Use Branching Strategically
- 3. Take Advantage of GitHub Actions
- 4. Protect Sensitive Information with .gitignore
- 5. Stay Informed with Webhooks

In the modern era of software development, understanding how to effectively use tools like GitHub is crucial. This guide is designed to help you navigate the world of GitHub, specifically focusing on working with private repositories in an enterprise setting. By the end of this guide, you’ll have a solid foundation in GitHub, git, and SSH, empowering you to manage your codebase efficiently and securely.

Getting Started with GitHub and Git

Create Your GitHub Account

Start your journey by setting up a GitHub account. GitHub is a web-based hosting service for version control and is a key player in the open-source community. Having a GitHub account opens up a world of opportunities for collaboration and project management.

https://github.com/join

2. Install Git

Git is the backbone of GitHub. It’s a distributed version control system that allows multiple people to work on a project without overwriting each other’s changes. Download and install git on your local machine to start leveraging its power.

https://git-scm.com/book/en/v2/Getting-Started-Installing-Git

3. Configure Git

Personalize your git setup by adding your username and email. This information will be associated with any commits you make. Open your terminal or shell and type:

git config --global user.name "Your name here"git config --global user.email "your_email@example.com"

To enhance your git experience, enable colored output in the terminal and set your preferred editor. This can make navigating git responses easier and ensure you’re comfortable when git opens an editor for you:

git config - global color.ui  
git config - global core.editor

Securing Your Connection with SSH

Establish SSH Connection

Security is paramount when working with code, especially in an enterprise setting. SSH, or Secure Shell, is a protocol that provides a secure channel between your local machine and GitHub. You can follow this comprehensive guide for setting up password-less logins. GitHub also provides a detailed guide for generating SSH keys.

Check if you have the files ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub. If not, create these public/private keys:

ssh-keygen -t rsa -C "your_email@example.com"

Now to Copy your public key and get ready to paste it into your GitHub account follow the below steps.

Update SSH Key in GitHub Account

Navigate to your GitHub Account Settings and click on “SSH Keys”. Add a new SSH key with a label (like “Vs Code”) and paste the public key you copied earlier.

To verify your setup, type the following in your terminal:

ssh -T git@github.com

Hi username! You’ve successfully authenticated, but GitHub does not provide shell access.

If you see the above message, congratulations! You’re all set to start working with private repositories in an enterprise setting.

Additional Tips and Techniques

Leverage GitHub’s Issue Tracker

GitHub’s issue tracker is a powerful tool for managing tasks, bugs, and feature requests. Each issue provides a platform for discussion, allowing team members to communicate about the task at hand.

2. Use Branching Strategically

Branching is a core feature of git that allows you to work on different features or bugs in isolation. Developing a good branching strategy can help keep your codebase organized and make the development process smoother.

3. Take Advantage of GitHub Actions

GitHub Actions is a CI/CD service provided by GitHub. It allows you to automate tasks like building, testing, and deploying your code. This can save you time and help ensure consistent quality in your codebase.

4. Protect Sensitive Information with .gitignore

The .gitignore file allows you to specify files or directories that git should ignore. This is crucial for keeping sensitive information, like API keys or configuration files with passwords, out of your codebase.

5. Stay Informed with Webhooks

Webhooks allow you to set up automatic notifications when specific events occur in your repository. This can help keep you informed about the state of your project and respond quickly to changes.

By mastering these tools and techniques, you’ll be well-equipped to manage private repositories in an enterprise setting. Whether you’re a seasoned developer or just starting out, GitHub offers a wealth of features to streamline your workflow and enhance collaboration.

Read my blogs:

Connect with Me:

Exploring Azure Storage Services

Kunal Das — Sun, 21 Jan 2024 12:36:06 +0000

Exploring Azure Storage Services 🌐🗄️

Azure, Microsoft's cloud computing service, offers a range of storage solutions designed to meet the diverse needs of modern businesses. In this guide, we'll delve into the various Azure storage services, exploring their unique features and use cases. Whether you're a developer, a data scientist, or an IT professional, understanding these services can enhance your cloud strategy.

Reach at : https://heylink.me/kunaldas

Exploring Azure Storage Services 🌐🗄️
- Table of Contents
- Introduction to Azure Storage
- Azure Blob Storage
- What is Blob Storage? 🤔
- Key Features 🌟
- Use Cases 🛠️
- Azure File Storage
- What is File Storage? 📂
- Key Features 🌟
- Use Cases 🛠️
- Azure Queue Storage
- What is Queue Storage? 📨
- Key Features 🌟
- Use Cases 🛠️
- Azure Table Storage
- What is Table Storage? 📊
- Key Features 🌟
- Use Cases 🛠️
- Azure Disk Storage
- What is Disk Storage? 💽
- Key Features 🌟
- Use Cases 🛠️
- Choosing the Right Azure Storage Service
- Conclusion
- Read my blogs:
- Connect with Me:

Introduction to Azure Storage

Azure Storage is a cloud service at the core of Azure. It offers scalable, durable, and secure storage options for a variety of data types. In this guide, we will explore the different Azure Storage services, understand their functionalities, and see how they can be applied in real-world scenarios.

Azure Blob Storage

What is Blob Storage? 🤔

Blob Storage is Azure's object storage solution for the cloud. It is optimized for storing massive amounts of unstructured data, such as text or binary data.

Key Features 🌟

Scalability: Easily handles massive amounts of data.
Security: Advanced security and encryption features.
Accessibility: Data is accessible from anywhere in the world over HTTP or HTTPS.

Use Cases 🛠️

Storing Images and Videos: Ideal for storing media files for websites and applications.
Data Backup: Efficient for backing up data and disaster recovery solutions.
Big Data Analysis: Can store large datasets for analytics purposes.

Azure File Storage

What is File Storage? 📂

Azure File Storage offers fully managed file shares in the cloud using the standard SMB protocol.

Key Features 🌟

SMB Protocol: Uses the standard SMB 3.0 protocol.
Shared Access: Files can be shared across applications and virtual machines.

Use Cases 🛠️

Lift and Shift: Ideal for migrating on-premises applications that rely on file shares to Azure.
Hybrid Storage: Works well for scenarios requiring both on-premises and cloud storage.

Azure Queue Storage

What is Queue Storage? 📨

Queue Storage provides a messaging queue for reliable messaging between application components.

Key Features 🌟

Decoupling: Helps in decoupling application components.
Scalability: Scales to handle a large number of messages.

Use Cases 🛠️

Message Processing: Perfect for asynchronous message processing in applications.
Task Scheduling: Useful in task scheduling scenarios.

Azure Table Storage

What is Table Storage? 📊

Table Storage is a NoSQL data store for semi-structured data.

Key Features 🌟

NoSQL: Ideal for storing large volumes of non-relational data.
Flexible: Easy to adapt to changing data requirements.

Use Cases 🛠️

Storing User Data: Great for storing user data for web applications.
Address Books: Suitable for storing address books, user profiles, etc.

Azure Disk Storage

What is Disk Storage? 💽

Azure Disk Storage offers high-performance, durable block storage for Azure Virtual Machines.

Key Features 🌟

High Performance: Designed for I/O-intensive workloads.
Data Durability: Provides persistent data storage.

Use Cases 🛠️

Virtual Machines: Ideal for databases and other high-performance applications running on VMs.

Choosing the Right Azure Storage Service

When selecting an Azure Storage service, consider factors like data type, access patterns, and scalability requirements. Each service is tailored to specific scenarios, so understanding your application's needs is crucial.

Conclusion

Azure's diverse storage options offer flexible, scalable, and secure solutions to meet the ever-evolving data storage needs of businesses. By understanding the strengths and applications of each service, you can make informed decisions about your cloud storage strategy.

Read my blogs:

Connect with Me:

Seamless Integration and Deployment of Azure Data Factory by using Azure DevOps

Kunal Das — Sun, 21 Jan 2024 12:36:05 +0000

Seamless Integration and Deployment of Azure Data Factory by using Azure DevOps

Reach at : https://heylink.me/kunaldas

Seamless Integration and Deployment of Azure Data Factory by using Azure DevOps
- Introduction
- Embracing Source Control in ADF
- Advantages of Git Integration with Azure Data Factory
- Architecture Flow diagram
- Connecting to a Git Repository
- Implementing the Pipeline Template

Introduction

Azure Data Factory (ADF) offers a robust platform for data integration and transformation, and when combined with continuous integration and delivery (CI/CD), it becomes a powerhouse. CI/CD in ADF context is the seamless transition of data pipelines across different environments like development, testing, and production. ADF leverages Azure Resource Manager (ARM) templates to encapsulate the configurations of its entities, such as pipelines, datasets, and data flows. There are primarily two recommended ways to transition a data factory across environments:

Leveraging Azure Pipelines for an automated deployment.
Manually uploading an ARM template through the Data Factory user interface integrated with Azure Resource Manager.

Embracing Source Control in ADF

Azure Data Factory’s integration with ARM templates facilitates the deployment of pipelines. Notably, there’s a distinct ADF Publish branch and a collaboration branch.

Steps for Integrating Source Control with Branching Strategy:

Initialize the Git Repository: Start by initializing a single Git repository. This repository will house multiple ADF configurations, each tailored for specific pipelines.
Branching Blueprint: Designate a unique branch for every ADF, which will act as a collaboration branch. This setup will lead to the creation of distinct folders under the adf_publish branch. It's crucial to have separate branches for development to allow individual feature development, testing, and deployment.Important to note that we may not use this branch as we will use automated ARM tempalte publishing methood.
Integrate Development Branches with Source Control: Only link the development branches with source control. This ensures continuous validation and checking of the code during its development phase. Keeping UAT/Production deployments separate ensures a clear demarcation between development and deployment phases.
Pipeline Deployment: Utilize the ARM templates produced by ADF to deploy your pipelines, ensuring a uniform deployment process.
Final Integration: Post thorough testing, merge the feature branches with the collaboration branch. The final version in the collaboration branch should be the one deployed to production.

Advantages of Git Integration with Azure Data Factory

1. Enhanced Source Control: As ADF tasks become increasingly critical, it’s essential to:

Seamlessly track and audit changes.
Effortlessly revert unwanted modifications.

2. Flexible Drafting: Unlike direct authoring, which mandates validation for every save, Git integration allows:

Drafting or partial saves.
Incremental modifications without validation, ensuring only thoroughly tested changes are published.

3. Collaborative Environment & Role-Based Access: Git facilitates:

Collaborative code reviews.
Differentiated permissions, dictating who can edit via Git and who has publishing rights.

4. Streamlined CI/CD Process: Git aids in:

Automating release pipelines upon changes.
Customizing ARM template properties for cleaner configuration management.

5. Boosted Performance: ADFs integrated with Git are significantly faster, loading up to a whopping 10 times quicker due to efficient resource downloading.

It’s worth noting that direct authoring in the Azure Data Factory UI becomes disabled once a Git repository is integrated. However, modifications made via PowerShell or SDK are directly published to the Data Factory service, bypassing Git.

Architecture Flow diagram

Connecting to a Git Repository

Configuration using Management Hub:

Navigate to the management hub within the ADF UI.
Under the Source control section, select Git configuration.
If no repository is linked, click on Configure.

When setting up Git in the Azure Portal, certain settings like project name and repository name need to be manually inputted.

Azure Repos Settings:

The configuration pane will display various Azure Repos code repository settings. These settings are essential to apply the CI-CD template. For instance:

Repository Type: Specifies the type of the Azure Repos code repository.
Azure Active Directory: Your Azure AD tenant name.
Azure Repos Organization: Your Azure Repos organization name.
Project Name: Your Azure Repos project name.
Repository Name: Your Azure Repos code repository name.
Collaboration Branch: Your Azure Repos collaboration branch used for publishing.
Publish Branch: The branch where ARM templates related to publishing are stored.
Root Folder: Your root folder in your Azure Repos collaboration branch.

Ensure you enable the option in the management hub to include global parameters in the ARM template if you have declared any global parameters.

# Parameter Definitions: Allows configuration without diving deep into the script.
parameters:
- name: envTarget
  displayName: 'Deployment Environment'
  type: string
  values:
  - Stage
  - Prod

- name: azureDFName
  displayName: 'Azure Data Factory Identifier'
  type: string

- name: gitADFPath
  displayName: 'Git Path for ADF Publishing'
  type: string

- name: azureRegion
  displayName: 'Azure Deployment Region'
  type: string

- name: azureResourceGroup
  displayName: 'Resource Group in Azure'
  type: string

- name: azureSubID
  displayName: 'Azure Subscription Identifier'
  type: string

- name: azureRMConnectionName
  displayName: 'Azure Resource Manager Connection Identifier'
  type: string

- name: sourceDFName
  displayName: 'Source Data Factory for ARM Template'
  type: string

- name: targetDFName
  displayName: 'Target Data Factory for Deployment'
  type: string

- name: modifyGlobalParams
  displayName: 'Modify Global Parameters'
  type: boolean
  default: false

# Build Phase: Validate and Create ARM templates for Data Factory using npm.
stages:
- stage: Construct
  displayName: 'Compile and Confirm'
  jobs:
  - job: CompileAndCheck
    displayName: 'Compile and Confirm Azure Data Factory'
    pool:
      vmImage: 'ubuntu-latest'

    steps:
      # Set up Node.js for npm tasks.
      - task: NodeTool@0
        inputs:
          versionSpec: '14.x'
        displayName: 'Set up Node.js'

      # Set up required npm packages for Data Factory.
      - task: Npm@1
        inputs:
          command: 'install'
          verbose: true
          workingDir: '$(Build.Repository.LocalPath)/data-factory/'
        displayName: 'Set up npm modules'

      # Confirm Data Factory setup.
      - task: Npm@1
        inputs:
          command: 'custom'
          workingDir: '$(Build.Repository.LocalPath)/data-factory/'
          customCommand: 'run build validate $(Build.Repository.LocalPath)/data-factory /subscriptions/$(azureSubID)/resourceGroups/$(azureResourceGroup)/providers/Microsoft.DataFactory/factories/$(azureDFName)'
        displayName: 'Confirm Data Factory Setup'

      # Create ARM template for Data Factory.
      - task: Npm@1
        inputs:
          command: 'custom'
          workingDir: '$(Build.Repository.LocalPath)/data-factory/'
          customCommand: 'run build export $(Build.Repository.LocalPath)/data-factory /subscriptions/$(azureSubID)/resourceGroups/$(azureResourceGroup)/providers/Microsoft.DataFactory/factories/$(azureDFName) "ArmTemplate"'
        displayName: 'Create ARM template'

      # Share the created ARM template for later stages.
      - task: PublishPipelineArtifact@1
        inputs:
          targetPath: '$(Build.Repository.LocalPath)/data-factory/ArmTemplate'
          artifact: 'ArmTemplateArtifact'
          publishLocation: 'pipeline'
        displayName: 'Share ARM template'

# Deployment Phase: Deploy the Data Factory using ARM template.
- stage: DeployPhase
  jobs:
  - deployment: DeployToTarget
    displayName: 'Deploy to ${{ parameters.envTarget }} | ADF: ${{ parameters.azureDFName }}'
    dependsOn: Construct
    condition: succeeded()
    environment: ${{ parameters.envTarget }}
    pool:
      vmImage: 'ubuntu-latest'
    strategy:
      runOnce:
        deploy:
          steps:
            # Skip repo checkout for faster deployment.
            - checkout: none

            # Retrieve the ARM template from the build phase.
            - task: DownloadPipelineArtifact@2
              inputs:
                buildType: 'current'
                artifactName: 'ArmTemplateArtifact'
                targetPath: '$(Pipeline.Workspace)'
              displayName: "Retrieve ARM template"

            # Optionally modify global parameters if needed.
            - ${{ if eq(parameters.modifyGlobalParams, true) }}:
              - task: AzurePowerShell@5
                displayName: '(Optional) Modify Global Parameters'
                inputs:
                  azureSubscription: ${{ parameters.azureRMConnectionName }}
                  azurePowerShellVersion: 'LatestVersion'
                  ScriptType: 'FilePath'
                  ScriptPath: '$(Pipeline.Workspace)/GlobalParametersUpdateScript.ps1'
                  ScriptArguments: '-globalParametersFilePath "$(Pipeline.Workspace)/*_GlobalParameters.json" -resourceGroupName "${{ parameters.azureResourceGroup }}" -dataFactoryName "${{ parameters.sourceDFName }}"'

            # Deactivate ADF Triggers after deployment.
            - task: toggle-adf-trigger@2
              inputs:
                azureSubscription: ${{ parameters.azureRMConnectionName }}
                ResourceGroupName: ${{ parameters.azureResourceGroup }}
                DatafactoryName: ${{ parameters.targetDFName }}
                TriggerStatus: 'stop'

            # Deploy using the ARM template. Override source ADF name with target ADF name.
            - task: AzureResourceManagerTemplateDeployment@3
              displayName: 'Deploy using ARM Template'
              inputs:
                azureResourceManagerConnection: ${{ parameters.azureRMConnectionName }}
                subscriptionId: ${{ parameters.azureSubID }}
                resourceGroupName: ${{ parameters.azureResourceGroup }}
                location: ${{ parameters.azureRegion }}
                csmFile: '$(Pipeline.Workspace)/ARMTemplateForFactory.json'
                csmParametersFile: '$(Pipeline.Workspace)/ARMTemplateParametersForFactory.json'
                overrideParameters: '-factoryName "${{ parameters.targetDFName }}"'

            # Activate ADF Triggers after deployment.
            - task: toggle-adf-trigger@2
              inputs:
                azureSubscription: ${{ parameters.azureRMConnectionName }}
                ResourceGroupName: ${{ parameters.azureResourceGroup }}
                DatafactoryName: ${{ parameters.targetDFName }}
                TriggerStatus: 'start'

Implementing the Pipeline Template

This guide provides a comprehensive walkthrough on setting up and utilizing the Azure Data Factory CI/CD pipeline as defined in the YAML file. The pipeline streamlines the build and deployment of ADF artifacts to designated environments.

Prerequisites:

Azure Data Factory Instance: An active ADF instance in Azure.
Azure DevOps: The YAML is tailored for Azure DevOps. Ensure you have an active Azure DevOps organization and project.
Azure DevOps Agent: The pipeline employs the ubuntu-latest VM image.
Node.js: The initial stage requires Node.js.
Azure Subscription: Necessary permissions on your Azure subscription are required.

To download necessery package for the npm, keep package.json file in the parent directory

{
    "scripts":{
        "build":"node node_modules/@microsoft/azure-data-factory-utilities/lib/index"
    },
    "dependencies":{
        "@microsoft/azure-data-factory-utilities":"^1.0.0"
    }
}

Repository Structure:

The ADF code should be housed in a folder named data-factory-directory at the root of your repository.

Setup & Execution:

Azure Service Connection: Establish a service connection in Azure DevOps linked to your Azure subscription.
GlobalParametersUpdateScript.ps1: If the modifyGlobalParams flag is set to true, ensure a PowerShell script named GlobalParamsScript.ps1 is present at the root of your repository.
Using the Pipeline: Upload the YAML file to your Azure DevOps repository, create a new pipeline, fill in the parameters, trigger the pipeline, and monitor the build and deployment.

Running the Pipeline

Push some changes to developmentbranch. and the pipeline will get triggered automatically

Once the pipeline finished see the artifact that got published.

Similarly once the pipeline gets approval for deployment, it will deploy the updated template to production,

Modifications & Best Practices:

Global Parameter Update: Adjust the logic in the optional global parameter update task if needed.
Additional Tasks: Insert any extra tasks within the steps section of the stages.
Permissions: Not every team member should have update permissions. Implement a system where only a select few can publish to the Data Factory.
Using Azure Key Vault: For security, store connection strings or passwords in Azure Key Vault or use managed identity authentication for ADF Linked Services.

In conclusion, integrating CI/CD with Azure Data Factory not only streamlines the deployment process but also enhances collaboration, auditing, and overall efficiency. With the right setup and best practices, teams can ensure seamless and error-free deployments.

Read my blogs:

Connect with Me:

Mastering Kubernetes A Deep Dive into Cluster Management Tools Every DevOps Engineer Should Know

Kunal Das — Sun, 21 Jan 2024 12:36:04 +0000

Mastering Kubernetes: A Deep Dive into Cluster Management Tools Every DevOps Engineer Should Know

Reach at : https://heylink.me/kunaldas

Mastering Kubernetes: A Deep Dive into Cluster Management Tools Every DevOps Engineer Should Know
- 1. Kustomizer
- 2. k9s
- 3. Kudo
- 4. node-problem-detector
- 5. k0s
- 6. Helm
- 7. ClusterPedia
- 8. KEDA
- 9. kubectl snapshot
- 10. Cert-manager
- 11. Prometheus
- 12. Metalk8s
- 13. Kube-ops-view

Kubernetes, often abbreviated as K8s, has revolutionized the world of container orchestration. As its popularity grows, so does the ecosystem around it, offering a myriad of tools designed to simplify, enhance, and optimize the Kubernetes experience. For DevOps engineers, understanding these tools is paramount. In this comprehensive guide, we’ll explore some of the most promising cluster management tools that can elevate your Kubernetes game.

1. Kustomizer

Kustomizer is not just another configuration management tool. It stands out by allowing Kubernetes native applications to be customized without the need for templates. This means you can manage variations of Kubernetes YAML configurations without diving into complex templating engines.

Learn more about Kustomizer

2. k9s

Imagine having a bird’s-eye view of your Kubernetes clusters right from your terminal. k9s offers a terminal-based UI that provides real-time insights into cluster activities and resources. It’s like having a dashboard but with the power of the command line.

Explore k9s

3. Kudo

Building Kubernetes Operators can be challenging. Enter KUDO — the Kubernetes Universal Declarative Operator. It’s an open-source toolkit that simplifies the creation of Operators using a declarative approach, making the management of stateful applications on Kubernetes a breeze.

Discover Kudo

4. node-problem-detector

Ensuring the health of your nodes is crucial. The node-problem-detector tool identifies common node issues, bridging the gap between the kernel and cluster management layers. It’s like having a health check-up for your nodes.

Check out node-problem-detector

5. k0s

In environments where resources are limited, such as edge computing or IoT, k0s shines. It’s a lightweight, certified Kubernetes distribution tailored for such scenarios, ensuring you don’t compromise on performance.

Dive into k0s

6. Helm

Helm is often dubbed the “package manager for Kubernetes.” It allows users to define, install, and upgrade even the most complex Kubernetes applications using charts, which are packages of pre-configured Kubernetes resources.

Discover Helm

7. ClusterPedia

Searching and managing resources across clusters can be daunting. ClusterPedia, a unified search platform for Kubernetes clusters, streamlines this process, making resource management efficient and hassle-free.

Learn about ClusterPedia (Note: Link to be updated when available)

8. KEDA

Event-driven architectures are gaining traction. KEDA (Kubernetes-based Event-Driven Autoscaling) is a component that brings event-driven autoscaling to your Kubernetes applications, ensuring they scale based on real-time demand.

Explore KEDA

9. kubectl snapshot

Documentation and debugging are made easier with kubectl snapshot. This tool captures the current state of a Kubernetes cluster, providing a snapshot that can be analyzed or shared.

Discover kubectl snapshot (Note: Link to be updated when available)

10. Cert-manager

Security is paramount. Cert-manager steps in as a native Kubernetes certificate management controller, automating the issuance and renewal of certificates from various sources, ensuring your applications remain secure.

Check out Cert-manager

11. Prometheus

Monitoring is crucial in a Kubernetes environment. Prometheus, an open-source monitoring and alerting toolkit, integrates seamlessly with Kubernetes, providing insights into your clusters and applications.

Learn more about Prometheus

12. Metalk8s

For those focusing on long-term on-prem deployments, especially on metal machines, Metalk8s is a go-to Kubernetes distribution. It’s opinionated, ensuring stability and performance in such specific scenarios.

Dive into Metalk8s

13. Kube-ops-view

Visual representation can simplify complex operations. Kube-ops-view offers a read-only system dashboard for multiple K8s clusters, providing a graphical overview of cluster operations.

Explore Kube-ops-view

Conclusion: The Kubernetes ecosystem is vast and ever-evolving. For DevOps engineers, staying updated with the right tools can make the difference between a smoothly running cluster and a chaotic environment. This guide provides a starting point, but always ensure to evaluate tools based on your unique needs.

Comment down the tools you use I shall be updating the list whenever I find something cool.

keep it followed !!

Read my blogs:

Connect with Me:

COST ESTIMATION FOR INFRASTRUCTURE

Kunal Das — Sun, 21 Jan 2024 12:36:03 +0000

COST ESTIMATION FOR INFRASTRUCTURE

Reach at : https://heylink.me/kunaldas

COST ESTIMATION FOR INFRASTRUCTURE
- 1. Design Goals
- 1.1. Cost estimation
- 1.2. Options available
- 1.3. Available tools
- 2. Infracost
- 2.1. Overview
- 2.2. Advantages
- 2.3 Limitations
- 2.4 Use cases
- 2.5 Impact analysis
- 3. Workflow
- 3.1 Basic workflow
- 3.2 DevOps workflow
- 3.2.1 Azure DevOps
- 3.2.2 Jenkins
- 1. Implementation
- 4.1 Steps for Azure DevOps
- 4.1.1 Installing extension
- 4.1.2 Making the repo ready
- 4.1.3 Adding Infracost code in pipeline
- 4.1.1 Additional settings
- 4.1.2 Creating a Pull Request
- 4.1.3 Viewing the dashboard
- 4.1.1 Dashboard use cases
- 1. Navigation
- 5.1 PR cost difference
- 5.1 Detailed Cost estimation
- 6. Diagram
- 6.1 New Branch
- 6.2 Making changes
- 6.3 Pull Request
- 6.4 Release
- 1. Scope for improvement
- 2. Cost of Implementation

1. Design Goals

1.1. Cost estimation

Cost estimation is the process of predicting the cost of a project, product, or service. It involves identifying all the resources that will be needed and determining the associated costs to arrive at a total estimate. Cost estimation is a key part of project management and helps stakeholders understand the financial implications of their decisions. It can also help organizations make informed decisions about whether to pursue a project or service and how to allocate budgets for the best return on investment. There are many factors that can affect the cost of a project, including the complexity of the work, the skills and expertise of the team, and the availability of resources.

1.2. Options available

There are several options for cost estimation in the cloud:

ü Cost calculators: Most cloud providers offer cost calculators that allow you to estimate the cost of various cloud resources based on your usage patterns and requirements. These calculators can help you compare the cost of different options and choose the most cost-effective solution.

ü Cost optimization tools: Many cloud providers offer tools and services to help organizations optimize their resource usage and costs. These tools can help identify cost drivers, forecast future costs, and recommend optimization strategies.

ü Pricing plans: Many cloud providers offer a variety of pricing plans that offer different levels of resources and support at different price points. Carefully reviewing and comparing these plans can help you choose the one that best meets your needs and budget.

ü Cost monitoring and management tools: There are a variety of tools and services available that can help you monitor your cloud resource usage and costs in real-time. These tools can alert you to unexpected cost spikes and help you identify opportunities for cost optimization.

ü Negotiating with vendors: Some cloud providers may be willing to negotiate pricing or offer discounts for large or long-term commitments. It may be worth negotiating with your provider to see if you can get a better deal on your cloud resources.

ü Resource optimization: Identifying and eliminating unnecessary or underutilized resources can help reduce your overall cloud costs. This could include retiring or decommissioning old resources, optimizing resource allocation, and consolidating resources where possible.

1.3. Available tools

There are several tools available for cost estimation:

ü Cost calculators: Many cloud providers offer cost calculators that allow you to estimate the cost of various cloud resources based on your usage patterns and requirements. These calculators can help you compare the cost of different options and choose the most cost-effective solution. E.g. Total Cost of Ownership (TCO) Calculator in AWS, The Azure Pricing Calculator, Google Cloud Pricing Calculator

ü Project management software: Many project management software platforms offer cost estimation and budgeting tools to help organizations plan and track project costs.

ü Specialized cost estimation software: There are also specialized cost estimation software tools that are specifically designed for cost estimation. These tools often offer advanced features and capabilities, such as the ability to create detailed cost breakdowns and incorporate risk analysis.

ü Infracost: Infracost is a tool that helps users optimize their costs when using Terraform for infrastructure management. It provides real-time cost estimates for infrastructure resources and suggests ways to reduce costs through optimization.

ü Teracost: TeraCost is a cloud cost management and optimization platform that helps organizations optimize their cloud costs by providing visibility into resource usage and costs, identifying opportunities for optimization, and recommending actions to reduce costs. It offers a range of features, including cost forecasting, budget tracking, and resource optimization recommendations, and is designed to work with a variety of cloud providers, including Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP).

2. Infracost

2.1. Overview

Infracost is a tool that is specifically designed to optimize costs in Terraform environments. It works by analysing the resources that are being created in a Terraform configuration and providing recommendations for cost optimization. This can include suggestions for using lower cost resource types, resizing resources to a more cost-effective size, or identifying resources that are no longer needed and can be safely deleted. Infracost also provides real-time cost estimates for the resources in a Terraform configuration, which can help teams make more informed decisions about the cost of their infrastructure. Overall, Infracost is a powerful tool that can help teams save money on their infrastructure costs by identifying opportunities for cost optimization and providing actionable recommendations for implementing those optimizations.

2.2. Advantages

Infracost is a tool that can be used for cost optimization in Terraform, a popular Infrastructure as Code (IaC) tool. It provides several advantages over other cost optimization tools available for Terraform:

ü Real-time cost estimation: Infracost provides real-time cost estimates for all the resources in your Terraform code, so you can make informed decisions about your infrastructure costs.

ü Integration with Terraform: Infracost integrates seamlessly with Terraform, so you can see the cost estimates for your infrastructure as you write your code. This helps you avoid costly mistakes and optimize your infrastructure costs from the start.

ü Custom pricing: Infracost allows you to set custom pricing for your resources, so you can better reflect your organization’s negotiated rates or your unique usage patterns.

ü Resource filtering: Infracost allows you to filter resources by tag, type, and name, making it easier to focus on specific resources and optimize their costs.

ü Resource grouping: Infracost groups similar resources together, making it easier to compare costs and identify opportunities for optimization.

ü Easy to use: Infracost is easy to use and requires minimal setup, so you can start optimizing your infrastructure costs quickly and easily.

Overall, Infracost provides a comprehensive and user-friendly solution for cost optimization in Terraform, making it an excellent choice for organizations looking to optimize their infrastructure costs.

2.3 Limitations

Infracost is a useful tool for gaining visibility into the cost of your infrastructure and optimizing your spend on cloud resources. However, it does have some limitations to consider:

ü Limited to supported cloud providers: Infracost currently supports only a limited number of cloud providers, including AWS, Azure, and Google Cloud. If you are using a different cloud provider or a combination of multiple providers, Infracost may not be able to provide cost estimates for your infrastructure.

ü Dependent on accurate resource pricing: Infracost relies on accurate resource pricing information from the supported cloud providers. If the pricing information is incorrect or out of date, Infracost cost estimates may not be accurate.

ü Limited to resources supported by the cloud providers: Infracost can only provide cost estimates for resources that are supported by the cloud providers it integrates with. If you are using custom resources or resources from third-party providers, Infracost may not be able to provide cost estimates for them.

ü Requires manual integration with infrastructure as code: While Infracost integrates with popular infrastructure as code tools such as Terraform and CloudFormation, it requires manual integration. This means you will need to set up Infracost and configure it to work with your infrastructure as code tools, which can be time-consuming and require some technical expertise.

ü Limited to infrastructure costs: Infracost only provides estimates for the cost of your infrastructure resources, such as compute instances and storage. It does not include other costs such as data transfer fees or licensing costs for third-party software.

ü Limited to current infrastructure: Infracost only provides cost estimates for your current infrastructure. It does not allow you to compare the cost of different infrastructure configurations or predict the cost of future infrastructure changes.

Overall, while Infracost is a useful tool for gaining visibility into the cost of your infrastructure and optimizing your spend on cloud resources, it is limited in its scope and may not be suitable for all organizations or infrastructure configurations.

2.4 Use cases

Some possible use cases for Infracost include:

ü Identifying and reducing over-provisioned resources: Infracost can help you identify resources that are over-provisioned, which means they have more capacity than is necessary to meet your workload demands. By reducing the capacity of these resources, you can save on your cloud costs.

ü Optimizing resource usage: Infracost can help you optimize resource usage by identifying underutilized resources and suggesting ways to improve their utilization. For example, if you have a virtual machine that is only running at 20% capacity, Infracost may suggest moving workloads to that machine to better utilize its capacity.

ü Estimating costs for infrastructure changes: Infracost can be used to estimate the costs of making changes to your infrastructure, such as adding or removing resources. This can help you plan for changes and ensure that they are cost-effective.

ü Analysing cloud cost trends: Infracost can provide detailed cost breakdowns and graphs that allow you to understand your cloud costs over time and identify trends that may impact your budget. This can help you plan and make decisions to optimize your costs.

ü Auditing and compliance: Infracost can be used to audit your infrastructure and ensure that it complies with company policies or regulatory requirements. It can also help you identify resources that may be overpriced or misconfigured, allowing you to take corrective action.

2.5 Impact analysis

Infracost is a tool that allows users to perform cost analysis on their infrastructure-as-code (IaC) resources. This means that users can see the estimated cost of their resources before they are deployed, allowing them to make informed decisions about their infrastructure and optimize for cost. One key use case for Infracost is in the development phase, where users can use it to identify and address potential cost issues before they become a problem. It can also be used in the operations phase to monitor the ongoing cost of infrastructure and identify opportunities for optimization. By providing visibility into the cost of infrastructure, Infracost can help organizations make informed decisions about how to allocate their resources and stay within budget. Overall, the use of Infracost can have a significant impact on the cost efficiency of an organization’s infrastructure, helping to reduce waste and ensure that resources are being used effectively.

3. Workflow

3.1 Basic workflow

ü The workflow of infracost involves the following steps:

ü Install the infracost CLI tool: Infracost can be installed as a standalone CLI tool or as a Terraform plugin. To install the CLI tool, you can use the following command:

curl -sfL https://raw.githubusercontent.com/infracost/infracost/master/scripts/install.sh | sh

ü Initialize infracost: Once you have installed the infracost CLI tool, you can initialize it by running the following command in your Terraform directory:

infracost init

ü This will create a default configuration file (infracost.yml) in your Terraform directory, which you can customize as per your needs.

ü Add infracost to your Terraform workflow: You can add infracost to your Terraform workflow by using the infracost command before or after running terraform plan or terraform apply. For example, you can run the following command to generate a cost estimate before applying your changes:

infracost plan

ü Alternatively, you can run the following command to generate a cost estimate after applying your changes:

infracost apply

ü Review the cost estimate: Infracost will generate a cost estimate in the form of a table, which will show you the estimated cost of each resource in your Terraform configuration. You can review the cost estimate and make any necessary changes to your configuration to optimize costs.

ü Repeat the process: You can repeat the process of running infracost before or after applying your changes as many times as you like, until you are satisfied with the cost estimate.

ü Use infracost with version control: You can also use infracost with version control systems like Git to track changes to your cost estimates over time. This can be useful for comparing the costs of different versions of your Terraform configuration and for identifying cost-saving opportunities.

ü Use infracost with CI/CD: You can also integrate infracost with your CI/CD pipeline to automate the process of generating and reviewing cost estimates. This can be especially useful if you have a large number of Terraform configurations that need to be regularly reviewed for cost optimization.

3.2 DevOps workflow

3.2.1 Azure DevOps

ü The workflow for using infracost with Azure DevOps involves integrating infracost into your Azure DevOps pipeline. This can be done by adding the infracost task to your pipeline and configuring it to run at the appropriate stage.

ü To begin, you will need to install the infracost extension from the Azure DevOps marketplace. Next, you will need to create a pipeline and add the infracost task to it. In the task configuration, you will need to specify the path to your Terraform configuration files, as well as any additional arguments that you want to pass to infracost.

ü Once the task is configured, you can run your pipeline as usual. As part of the pipeline execution, infracost will analyze your Terraform configuration and provide cost estimates for the resources that are being created. This information will be displayed in the pipeline output, allowing you to easily see the cost implications of your changes.

ü In addition to running infracost as part of your pipeline, you can also use it to create pull request checks. This allows you to enforce cost limits on pull requests, ensuring that any changes that are made do not exceed a certain budget.

ü Overall, integrating infracost into your Azure DevOps workflow can help you better understand the cost implications of your infrastructure changes, and ensure that you are making cost-effective decisions as you develop and deploy your applications.

3.2.2 Jenkins

The workflow for using infracost with Jenkins typically involves the following steps:

ü Install the infracost plugin in Jenkins. This can be done through the Jenkins Plugin Manager.

ü Configure the infracost plugin in the Jenkins Global Configuration page. This includes specifying the AWS access keys, the Terraform version, and the desired frequency for cost scans.

ü In your Jenkins job, add a build step to run infracost. This can be done by selecting “Infracost” from the Add Build Step dropdown menu.

ü In the infracost build step, specify the path to your Terraform configuration files and any additional command line options.

ü Run the Jenkins job to trigger the infracost scan. The infracost plugin will scan your Terraform configuration files and generate a report on the estimated cost of your infrastructure.

ü View the infracost report in the Jenkins build output. The report will contain a breakdown of the cost estimates for each resource, as well as a total estimate for the entire infrastructure.

ü Optionally, you can configure infracost to fail the Jenkins build if the estimated cost exceeds a certain threshold. This can help prevent overspending on infrastructure costs.

ü By integrating infracost into your Jenkins workflow, you can easily track the cost of your infrastructure and make informed decisions about resource allocation and optimization.

Apart from all Infracost supports multiple CI/CD platforms shown below,

1. Implementation

4.1 Steps for Azure DevOps

4.1.1 Installing extension

ü Go to your organization setting page in Azure Devops

https://dev.azure.com/{org\_name}/\_settings/extensions

ü Go to the marketplace link and install Infracost extension

https://marketplace.visualstudio.com/items?itemName=Infracost.infracost-tasks

ü Go to the Infracost Dashboard URL and login with your account

https://dashboard.infracost.io/

4.1.2 Making the repo ready

ü Upon login go to organization settings and copy the API and keep it in safe place for later use

ü If you do not already have a Terraform file, create one or coy from my repo,

https://github.com/Kunaldastiger/Terraform-Sample/blob/main/Virtual-machines/MicrosoftWindowsServer.tf

ü Apart from this you must have a pipeline yaml file or a pipeline created in classic fashion,

code for the same can be found here,

https://github.com/Kunaldastiger/Terraform-Sample/blob/main/azure-deployment.yaml

ü Now run the pipeline and check if resource is getting created or not, needless to say, you also need to have a service connection to azure.

If you do not have a service connection kindly follow below steps,

Ø Sign in to Azure DevOps and navigate to the Project Settings page.

Ø In the left menu, under “Pipelines,” select “Service Connections.”

Ø Click the “New Service Connection” button.

Ø In the “Add new service connection” dialog, select “Azure Resource Manager” from the “Type” dropdown menu.

Ø Give the service connection a name and click “Next.”

Ø In the “Authorize Azure Resource Manager” dialog, select the Azure subscription you want to use and click “Authorize.”

Ø After the authorization process is complete, click “Finish” to create the service connection.

You can now use this service connection in your Azure DevOps pipelines to access resources in your Azure subscription. For example, you could use it to deploy an application to an Azure web app, create or delete Azure resources, or run Azure Resource Manager templates.

4.1.3 Adding Infracost code in pipeline

Create a new pipeline, selecting

Azure Repos Git when prompted in the “Connect” stage
Select the appropriate repo you wish to integrate Infracost with in the “Select” stage
Choose “Starter Pipeline” in the “Configure” stage
Replace the Starter Pipeline yaml with the following:

# The Azure Pipelines docs (https://docs.microsoft.com/en-us/azure/devops/pipelines/process/tasks) describe other options.
6. # Running on pull requests to `master` (or your default branch) is a good default.
7. pr:
8.   - main
9. 
10. variables:
11.   - name: TF_ROOT
12.     value: Terraform
13.   # If you use private modules you'll need this env variable to use 
14.   # the same ssh-agent socket value across all steps. 
15.   - name: SSH_AUTH_SOCK
16.     value: /tmp/ssh_agent.sock
17.   # This instructs the CLI to send cost estimates to Infracost Cloud. Our SaaS product
18.   #   complements the open source CLI by giving teams advanced visibility and controls.
19.   #   The cost estimates are transmitted in JSON format and do not contain any cloud 
20.   #   credentials or secrets (see https://infracost.io/docs/faq/ for more information).
21.   - name: INFRACOST_ENABLE_CLOUD
22.     value: true
23.   # If you're using Terraform Cloud/Enterprise and have variables stored on there
24.   # you can specify the following to automatically retrieve the variables:
25.   # env:
26.   # - name: INFRACOST_TERRAFORM_CLOUD_TOKEN
27.   #   value: $(tfcToken)
28.   # - name: INFRACOST_TERRAFORM_CLOUD_HOST
29.   #   value: app.terraform.io # Change this if you're using Terraform Enterprise
30. 
31. jobs:
32.   - job: infracost
33.     displayName: Run Infracost
34.     pool:
35.       vmImage: ubuntu-latest
36. 
37.     steps:
38.       # If you use private modules, add a base 64 encoded secret
39.       # called gitSshKeyBase64 with your private key, so Infracost can access
40.       # private repositories (similar to how Terraform/Terragrunt does).
41.       # - bash: |
42.       #     ssh-agent -a $(SSH_AUTH_SOCK)
43.       #     mkdir -p ~/.ssh
44.       #     echo "$(echo $GIT_SSH_KEY_BASE_64 | base64 -d)" | tr -d '\r' | ssh-add -
45.       #     # Update this to github.com, gitlab.com, bitbucket.org, ssh.dev.azure.com or your source control server's domain
46.       #     ssh-keyscan ssh.dev.azure.com >> ~/.ssh/known_hosts
47.       #   displayName: Add GIT_SSH_KEY
48.       #   env:
49.       #     GIT_SSH_KEY_BASE_64: $(gitSshKeyBase64)
50. 
51.       # Install the Infracost CLI, see https://github.com/infracost/infracost-azure-devops#infracostsetup
52.       # for other inputs such as version, and pricingApiEndpoint (for self-hosted users).
53.       - task: InfracostSetup@1
54.         displayName: Setup Infracost
55.         inputs:
56.           apiKey: $(infracostApiKey)
57. 
58.       # Clone the base branch of the pull request (e.g. main/master) into a temp directory.
59.       # - bash: |
60.       #     branch=$(System.PullRequest.TargetBranch)
61.       #     branch=${branch#refs/heads/}
62.       #     # Try adding the following to git clone if you're having issues cloning a private repo: --config http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)"
63.       - task: Bash@3
64.         inputs:
65.               targetType: 'inline'
66.               script: 'git clone @dev.azure.com/{orgname}/{project_name}/{repo_path}">https://$(PATToken)@dev.azure.com/{orgname}/{project_name}/{repo_path} --branch=main  /tmp/base'
67.       #   displayName: Checkout base branch
68. 
69.       # Generate an Infracost cost estimate baseline from the comparison branch, so that Infracost can compare the cost difference.
70.       - bash: |
71.           infracost breakdown --path=$(TF_ROOT) \
72.                               --format=json \
73.                               --out-file=/tmp/infracost-base.json
74.         displayName: Generate Infracost cost estimate baseline
75. 
76.       # Generate an Infracost diff and save it to a JSON file.
77.       - bash: |
78.           infracost diff --path=$(TF_ROOT) \
79.                          --format=json \
80.                          --compare-to=/tmp/infracost-base.json \
81.                          --out-file=/tmp/infracost.json
82.         displayName: Generate Infracost diff
83. 
84.       # Posts a comment to the PR using the 'update' behavior.
85.       # This creates a single comment and updates it. The "quietest" option.
86.       # The other valid behaviors are:
87.       #   delete-and-new - Delete previous comments and create a new one.
88.       #   new - Create a new cost estimate comment on every push.
89.       # See https://www.infracost.io/docs/features/cli_commands/#comment-on-pull-requests for other options.
90.       - bash: |
91.            infracost comment azure-repos --path=/tmp/infracost.json \
92.                                          --azure-access-token=$(System.AccessToken) \
93.                                          --pull-request=$(System.PullRequest.PullRequestId) \
94.                                          --repo-url= '@dev.azure.com/kunaldas0966/test_project/_git/terracost-sample'">https://$(PATToken)@dev.azure.com/kunaldas0966/test_project/_git/terracost-sample' \
95.                                          --behavior=update
96.         displayName: Post Infracost comment

In the pipeline, variables add the below variables

Ø To do the same,

Sign into Azure DevOps and navigate to your project.

Ø In the left menu, under “Pipelines,” select “Pipelines.”

Ø Select the pipeline you want to add a variable to and click the “Edit” button.

Ø In the pipeline editor, click the “Variables” tab.

Ø Click the “Add” button to add a new variable.

Ø In the “Add variable” dialog, enter a name and value for the variable. You can also specify whether the variable is secret and should be masked in the log output.

Ø Click “Save” to add the variable to the pipeline.

You can now use the pipeline variable in your pipeline tasks by using the syntax $(variableName). You can also use pipeline variables to control the flow of your pipeline, by using conditions or branching.

4.1.1 Additional settings

ü Enable pull request build triggers. Without this, Azure Pipelines do not trigger builds with the pull request ID, thus comments cannot be posted by the integration.

ü From your Azure DevOps organization, click on your project > Project Settings > Repositories

ü Select the repository that your created the pipeline for in step 1

ü Select the Policies tab and under the Branch Policies select on your default branch (master or main)

ü Scroll to Build Validation and click + sign to add one if you don’t have one already

ü Set your ‘Build pipeline’ to the pipeline you created in step 1, leave ‘Path filter’ blank, set ‘Trigger’ to Automatic, and ‘Policy requirement’ to Optional (you can also use Required but we don’t recommend it).

ü Enable Azure Pipelines to post pull request comments

ü From your Azure DevOps organization, click on your project > Project Settings > Repositories > your repository.

ü Click on the Securities tab, scroll down to Users and click on the ‘[project name] Build Service ([org name])’ user, and set the ‘Contribute to pull requests’ to Allow.

If you are using github instead of azure repo the code for the same is available at below link,

https://marketplace.visualstudio.com/items?itemName=Infracost.infracost-tasks&targetId=10277548-b72b-4200-99f2-565bf446c70d&utm_source=vstsproduct&utm_medium=ExtHubManageList

4.1.2 Creating a Pull Request

Ø Not you have to create a new branch and make some changes in the new branch to that the resource cost may increase or decrease,

To create a new branch and do a pull request in an Azure Repo, you’ll need to follow these steps:

Ø Navigate to the Azure Repos page in Azure DevOps.

Ø Select the repository that you want to create the branch and pull request in.

Ø On the repository page, click on the “Branches” tab.

Ø Click on the “New branch” button.

Ø Enter a name for the new branch and select the branch that you want to base the new branch on.

Ø Click on the “Create branch” button to create the new branch.

Once the new branch has been created, you can switch to it and make the necessary changes. When you’re ready to submit the changes for review, you can follow the steps above to create a pull request.

Ø Navigate to the Azure Repos page in Azure DevOps.

Ø Select the repository that you want to create the pull request in.

Ø On the repository page, click on the “Pull requests” tab.

Ø Click on the “New pull request” button.

Ø Select the source and target branches for the pull request. The source branch is the branch that contains the changes that you want to merge, and the target branch is the branch that you want to merge the changes into.

Ø Review the changes that will be included in the pull request. You can use the “Files changed” tab to view the individual changes and make any necessary adjustments.

Ø Add a title and optional description for the pull request.

Ø If you want to request a review from specific people or teams, you can use the “Reviewers” field to add them.

Ø When you’re ready to create the pull request, click on the “Create pull request” button.

After this you should see your pipeline starts running as the trigger is set as main pull request!

4.1.3 Viewing the dashboard

ü Log in to https://dashboard.infracost.io/ and select the repo ,

You should be able to see similar kind of dashboard, there are many things to go through in the GUI,

The Infracost dashboard is a web-based tool that provides an overview of the cost of your infrastructure resources. It displays information about the total cost of your resources, as well as detailed information about each resource and its cost.

The dashboard is divided into several sections:

Ø Overview: This section provides a summary of your infrastructure costs, including the total cost of your resources and the breakdown of costs by resource type.

Ø Resources: This section lists all of the infrastructure resources in your project, along with their cost, resource type, and resource ID. You can use this section to view the cost of individual resources and identify opportunities for cost optimization.

Ø Billing: This section provides information about your billing history, including the total amount billed, the billing period, and the date of the most recent bill.

Ø Costs by resource type: This section displays a breakdown of your costs by resource type, showing you which types of resources are the most expensive.

Ø Costs by tag: This section displays a breakdown of your costs by tag, allowing you to see which resources are associated with a particular tag and their costs.

Ø Cost trends: This section displays a chart showing the trend in your infrastructure costs over time. You can use this chart to identify trends in your costs and identify opportunities for cost optimization.

4.1.1 Dashboard use cases

There are several things that you can do from the Infracost dashboard to manage and optimize the cost of your infrastructure resources:

Ø View a summary of your infrastructure costs: The dashboard provides an overview of your total infrastructure costs, as well as a breakdown of costs by resource type. This can help you understand the overall cost of your infrastructure and identify areas where you might be able to save money.

Ø View detailed information about individual resources: The dashboard provides detailed information about each infrastructure resource, including its cost, resource type, and resource ID. This can help you understand the cost of individual resources and identify opportunities for cost optimization.

Ø View your billing history: The dashboard includes a section on billing that provides information about your total amount billed, the billing period, and the date of the most recent bill. This can help you understand your billing history and track changes in your costs over time.

Ø View costs by resource type and tag: The dashboard provides a breakdown of costs by resource type and tag, which can help you understand which types of resources the most expensive and which resources are are associated with a particular tag.

Ø View cost trends over time: The dashboard includes a chart showing the trend in your infrastructure costs over time. This can help you identify trends in your costs and identify opportunities for cost optimization.

1. Navigation

5.1 PR cost difference

ü Click on the repo and select the repository you are using in the Infracost dashboard,

Here, you will see all the price difference before you choose to merge the PR.

5.1 Detailed Cost estimation

Click to see the full cost estimate and it will open the detailed cost estimates in a separate

ü Here in the diff output tab you can see the difference that will occur if you merge the changes,

ü Click on the Table output to see all the involved cost,

ü Under each resource you can see detailed cost report of the individual items,

ü You may also choose to export the data in CSV format by clicking on export data.

6. Diagram

I have prepared a simple flow diagram that is very easy to understand, let’s go through each block one by one.

6.1 New Branch

ü Here basically we are creating a new branch so that, we can start making changes to existing infrastructure, creating a new branch in a version control system like Azure Repos allows you to work on a copy of your codebase and make changes to it without affecting the main branch (often called the “master” branch). This is useful when you want to make changes to your infrastructure, as it allows you to test and validate your changes before merging them into the main branch.

ü By creating a new branch, you can make changes to your infrastructure in a safe and isolated environment, without worrying about breaking anything in the main branch. This can help you avoid disruptions to your infrastructure and ensure that any changes you make are well-tested and ready for production.

6.2 Making changes

ü When you make changes to your codebase in a feature branch, you are working on a copy of the code that is separate from the main branch (often called the “master” branch). This allows you to make changes and test them without affecting the main branch, which can be useful when you are developing new features or making changes to your infrastructure. So you can just pull the code to your IDE like VS code and start making changes in the TF file, once done you can then push it to the dev/feature branch,

6.3 Pull Request

ü Create a pull request: When you’re ready to merge your changes into the main branch, you’ll need to create a pull request. This will allow other members of your team to review your changes and decide whether to merge them into the main branch.

ü Here once you create a pull request the pipeline will start running,

ü Once pipeline run finishes you can go to Infracost dashboard and view the cost estimation.

6.4 Release

If satisfied with the cost estimation then merge the changes and as soon as it gets merged to the main branch, it will trigger the terraform pipeline which will then create the desired resources.

1. Scope for improvement

The pipeline can be further automated to pass the Pull request automatically if certain conditions are met.

suppose if cost increases/decreases < 5% then the pipeline should read that and merge the code,

It needs further research and collaboration.

2. Cost of Implementation

Infracost is an open-source tool though it has some pricing options, the free version provides plenty of options to get the job done, the features offered in the free versions are:

ü Open source

ü Get cost breakdowns and diffs

ü CI/CD integrations (GitHub, GitLab, Bitbucket, Azure DevOps…)

ü Works with Terraform Cloud & Terragrunt

ü Use our hosted Cloud Pricing API or self-host

ü Community supported

For an annual fee of $30 some additionally functionalities can be availed,

ü In addition to Infracost Community:

ü Visibility across all changes, see pull requests that increase/decrease costs the most

ü Guardrails with custom messages/notifications/actions

ü Policies in pull requests (e.g. gp2 to gp3)

ü Custom price books and discounts

ü Jira integration

ü Custom reports

ü GitHub App integration with pull request status/metadata

ü Team management

ü SSO

Read my blogs:

Connect with Me:

How to publish from Release Pipeline in Azure DevOps

Kunal Das — Sun, 21 Jan 2024 12:36:02 +0000

How to publish from Release Pipeline in Azure DevOps

Reach at : https://heylink.me/kunaldas

How to publish from Release Pipeline in Azure DevOps
- Pushing the artifact in a git repo

In this section, I shall describe how you can get the artifact from the release pipeline,

Publishing any artifact from the build pipeline is pretty easy and there are plenty of tutorials available on the internet,

To publish an artifact from a build pipeline in Azure DevOps, you can use the Publish Build Artifacts task. Here's how you can do it:

In your build pipeline, add the Publish Build Artifacts task to a phase that runs after the build is completed.
In the Path to publish field, specify the path to the folder that contains the artifacts you want to publish. You can use a wildcard pattern to include multiple files and folders.
In the Artifact name field, enter a name for the artifact.
In the Artifact publish location field, choose whether you want to publish the artifact to the pipeline or to a file share.
If you are publishing to a file share, enter the path to the share in the Path to publish on the server field.
Save and run your pipeline.

That’s it! The Publish Build Artifacts task will publish the specified artifacts to the pipeline or file share you specified. You can then use the artifact in a release pipeline or download it from the Artifacts page in Azure DevOps.

- task: PublishPipelineArtifact@1
  displayName: 'Publish Pipeline Artifact'
  inputs:
    targetPath: '$(Build.ArtifactStagingDirectory)'
    artifact: drop

Now for any reason, you may want to get the same result from the release pipeline as well, but unfortunately, the PublishPipelineArtifact@1 task does not support in release pipeline and even if you run it you will get some error, to solve this you can do one thing,

Pushing the artifact in a git repo

in this way essentially you can save the copy of your updated artifacts in a separate folder inside your git repo.

let me show my scenario,

we are following the git flow branching strategy, so as you can see in each environment we need to save a copy of the updated ARM template .

How did I achieve this?

Well, let me guide you step by step.

First, give all the permission required so that the pipeline can access the repo and push to it!

Go to project settings → Repository → select your repo →security → project collection Administrators → contribute → ALLOW

This setting essentially allows pushing from the build pipeline to the repo.

also, you have to check one more setting,

for the classic pipeline, select the below option.

and for YAML, add the below code before the YAML code.

variables:
  system_accesstoken: $(System.AccessToken)

now add this bash task in the pipeline.

steps:
- bash: |
   git config --global user.email "azuredevops@microsoft.com"
   git config --global user.name "Azure DevOps"

   REPO="$(System.TeamFoundationCollectionUri)$(System.TeamProject)/_git/$(Build.Repository.Name)"
   EXTRAHEADER="Authorization: Bearer $(System.AccessToken)"
   git -c http.extraheader="$EXTRAHEADER" clone $REPO 
   cd $(Build.Repository.Name)

   mkdir 'QA-env'
   cd 'QA-env'

   cp '$(System.DefaultWorkingDirectory)/ARM/TemplateForWorkspace.json' .
   cp '$(System.DefaultWorkingDirectory)/ARM/TemplateParametersForWorkspace.json' .

   cd ..

   git add Template-QA/TemplateForWorkspace.json
   git add Template-QA/TemplateParametersForWorkspace.json


   MAINBRANCHNAME=main
   git config http.$REPO.extraHeader "$EXTRAHEADER"
   git commit -a -m "added QA json updated files"

   echo -- Merge $(Build.SourceBranchName) to $MAINBRANCHNAME --
   git fetch origin $(Build.SourceBranchName) --prune
   git merge origin/$(Build.SourceBranchName) -m "merge $(Build.SourceBranchName) to $MAINBRANCHNAME" --no-ff --allow-unrelated-histories



   git push origin $MAINBRANCHNAME
   git push origin --tags

  displayName: 'Bash Script'

If you are using a classic pipeline add a bash task

And add the below script as inline,

git config --global user.email "azuredevops@microsoft.com"
git config --global user.name "Azure DevOps"

REPO="$(System.TeamFoundationCollectionUri)$(System.TeamProject)/_git/$(Build.Repository.Name)"
EXTRAHEADER="Authorization: Bearer $(System.AccessToken)"
git -c http.extraheader="$EXTRAHEADER" clone $REPO 
cd $(Build.Repository.Name)

mkdir 'qa1-ause-asy-01'
cd 'qa1-ause-asy-01'

cp '$(System.DefaultWorkingDirectory)/_Synapse-CI-pipeline/drop/ARM/TemplateForWorkspace.json' .
cp '$(System.DefaultWorkingDirectory)/_Synapse-CI-pipeline/drop/ARM/TemplateParametersForWorkspace.json' .

cd ..

git add Template-QA/TemplateForWorkspace.json
git add Template-QA/TemplateParametersForWorkspace.json


MAINBRANCHNAME=main
git config http.$REPO.extraHeader "$EXTRAHEADER"
git commit -a -m "added QA json updated files"

echo -- Merge $(Build.SourceBranchName) to $MAINBRANCHNAME --
git fetch origin $(Build.SourceBranchName) --prune
git merge origin/$(Build.SourceBranchName) -m "merge $(Build.SourceBranchName) to $MAINBRANCHNAME" --no-ff --allow-unrelated-histories



git push origin $MAINBRANCHNAME
git push origin --tags

you can update the commit message according to your need,

In this script, the following actions are being performed:

The mkdir command is creating a new directory called QA-env.
The cd command is changing the current working directory to QA-env.
The cp command is copying the files TemplateForWorkspace.json and TemplateParametersForWorkspace.json from the ARM subdirectory of the default working directory to the current working directory (QA-env).
The cd .. command is changing the current working directory back to the parent directory.
The git add command is adding the files TemplateForWorkspace.json and TemplateParametersForWorkspace.json to the staging area in Git. This means that these files will be included in the next commit.

once done you will see the changes getting merged in the repo

That is it,

I guess you can try the steps and get back to me if any help is required!

Credit :

https://learn.microsoft.com/en-us/azure/devops/pipelines/release/?view=azure-devops

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/publish-build-artifacts-v1?view=azure-pipelines

https://stackoverflow.com/questions/52837980/how-to-allow-scripts-to-access-oauth-token-from-yaml-builds

https://chuvash.eu/2021/04/09/git-merge-develop-to-main-in-an-azure-devops-release/

Read my blogs:

Connect with Me:

The trick to not using a self-hosted agent in Azure DevOps

Kunal Das — Sun, 21 Jan 2024 12:36:01 +0000

The trick to not using a self-hosted agent in Azure DevOps

Reach at : https://heylink.me/kunaldas

Well, when you want to design the build and release pipeline for any software development the one thing that comes to mind is where to do the whole operation.

apparently, There are two ways of doing it,

The free and easy way is to use a Microsoft Hosted build agent,

But in some cases, we may need to use a Self-hosted build agent,

let’s see what Microsoft has to say about self-hosted build agents.

“An agent that you set up and manage on your own to run jobs is a self-hosted agent. You can use self-hosted agents in Azure Pipelines or Azure DevOps Server, formerly named Team Foundation Server (TFS). Self-hosted agents give you more control to install dependent software needed for your builds and deployments. Also, machine-level caches and configuration persist from run to run, which can boost speed.”

So, Self-hosted build agents do provide some benefits, but in my experience, some of these benefits can be achieved in Microsoft-hosted build agents as well.

For instance, if you have a relatively small build/release pipeline there’s no need to cache dependencies as it may give you a faster build time but do you really need it?

Now let’s see what a Microsoft hosted agent is as per them,

“If your pipelines are in Azure Pipelines, then you’ve got a convenient option to run your jobs using a Microsoft-hosted agent. With Microsoft-hosted agents, maintenance and upgrades are taken care of for you. Each time you run a pipeline, you get a fresh virtual machine for each job in the pipeline. The virtual machine is discarded after one job (which means any change that a job makes to the virtual machine file system, such as checking out code, will be unavailable to the next job). Microsoft-hosted agents can run jobs directly on the VM or in a container.”

So as you can see these agents are ephemeral by nature so we can not use them for something permanent right?

well, yes but there are some other wat to look at it.

In this article, I have a solution for a different problem, which is using a self-hosted build agent to overcome firewall restrictions set in different azure resources.

But first, let’s understand the problem.

Firewall restriction in a Synapse workspace

In the image, you can see public network access can be disabled. in that case, you will not be able to connect to this resource from the DevOps pipeline,

In some places, I have seen people using a self-hosted agent just for this purpose but actually, this is not required.

The process here is adding the IP address as a rule into the firewall

and you can do just that from the CI/CD pipeline itself,

Let’s look at how to achieve that.

I am taking the example of Synapse,

So just before the deployment task add an Azure CLI task in the pipeline.

and write the PowerShell script to add the build agent IP into the synapse workspace, needless to say, you need to give the service connection proper role to perform the action.

az synapse workspace firewall-rule create --name devops-build-agent-ip\
--workspace-name synapse-prod --resource-group synapse-prod-rg\
--start-ip-address  (Invoke-RestMethod http://ipinfo.io/json | Select -exp ip) \
--end-ip-address  (Invoke-RestMethod http://ipinfo.io/json | Select -exp ip)

If you like to use Yaml then here is the code!

steps:
- task: AzureCLI@2
  displayName: 'Add Build agent Ip to firewall'
  inputs:
    azureSubscription: 'sp-dev'
    scriptType: ps
    scriptLocation: inlineScript
    inlineScript: 'az synapse workspace firewall-rule create --name devops-build-agent-ip --workspace-name qa1-ause-asy-01 --resource-group QA1-AUSE-ASY-ARG-01 --start-ip-address  (Invoke-RestMethod http://ipinfo.io/json | Select -exp ip) --end-ip-address  (Invoke-RestMethod http://ipinfo.io/json | Select -exp ip)'

Now, as we can add the firewall rule we can delete the rule too, for that use the below code

az synapse workspace firewall-rule create --name devops-build-agent-ip\
--workspace-name synapse-prod --resource-group synapse-prod-rg --yes

and that is it.

once you run the deployment it will copy the IP to the build agent add it into the firewall rule then run the deployment and then delete it, but

Feel free to reach me incase of any issues!!

adios

Read my blogs:

Connect with Me:

Deploying Azure Data Factory, Azure Data bricks, Azure Data Lake storage & MySql DB using Terraform

Kunal Das — Thu, 18 Jan 2024 18:58:09 +0000

Deploying Azure Data Factory, Azure Data bricks, Azure Data Lake storage & MySql DB using Terraform

Reach at : https://heylink.me/kunaldas

Here I am going to share some terraform code to deploy ADF, ADLS, ADB, and several other necessary resources.

Deploying Azure Data Factory, Azure Data bricks, Azure Data Lake storage & MySql DB using Terraform
- Table of Contents
- Resource Group
- Azure Data Factory:
- Azure Data Bricks:
- Virtual network:
- Network Security group for ADB:
- Public subnet for Databricks:
- Private subnet for Databricks:
- Network security group for Public subnet:
- Network security group for Privatesubnet:
- Data Lake storage account:
- Storage account container:
- Storage Admin password:
- SQL server :
- SQL Database:

Let’s start with a resource group where we will store all the resources required.

Resource Group

data "azurerm_client_config" "Current" {}
resource "azurerm_resource_group" "RG" {
  name     = var.ResourceGroup.Name
  location = var.ResourceGroup.Location
}

points to note that we will fetch the RG name and RG location in the next resource declaration.

Azure Data Factory:

resource "azurerm_data_factory" "DataFactory" {
  name                = "DataFactory Name"
  location            = azurerm_resource_group.RG.location
  resource_group_name = azurerm_resource_group.RG.name

  identity {
    type = "SystemAssigned"
  }
}

Azure Data Bricks:

resource "azurerm_databricks_workspace" "Databricks" {
  location                      = azurerm_resource_group.RG.location
  name                          = "Databricks Name"
  resource_group_name           = azurerm_resource_group.RG.name
  managed_resource_group_name   = "Databricks Managed Resource Group"
  sku                           = "Databricks Sku"

  custom_parameters {
    no_public_ip        = true
    virtual_network_id  = azurerm_virtual_network.DatabricksVnet.id
    public_subnet_name  = azurerm_subnet.DatabricksSubnetPublic.name
    private_subnet_name = azurerm_subnet.DatabricksSubnetPrivate.name
  }

  depends_on = [
    azurerm_subnet_network_security_group_association.public,
    azurerm_subnet_network_security_group_association.private
  ]
}

Virtual network:

resource "azurerm_virtual_network" "DatabricksVnet" {
  name                     = "VNET NAME"
  resource_group_name      = azurerm_resource_group.RG.name
  location                 = azurerm_resource_group.RG.location
  address_space            = ["VNET CIDR"]
}

Network Security group for ADB:

resource "azurerm_network_security_group" "DatabricksNSG" {
  name                     = "VirtualNetwork NSG Name"
  resource_group_name      = azurerm_resource_group.RG.name
  location                 = azurerm_resource_group.RG.location
}

Public subnet for Databricks:

resource "azurerm_subnet" "DatabricksSubnetPublic" {
  name                 = "VirtualNetwork PublicSubnet Name"
  resource_group_name  = azurerm_resource_group.RG.name
  virtual_network_name = azurerm_virtual_network.DatabricksVnet.name
  address_prefixes     = ["VirtualNetwork PublicSubnet CIDR"]
  service_endpoints    = ["Microsoft.Storage"]

  delegation {
    name = "Microsoft.Databricks.workspaces"
    service_delegation {
      name = "Microsoft.Databricks/workspaces"
      actions = [
        "Microsoft.Network/virtualNetworks/subnets/join/action",
        "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
        "Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action"]
    }
  }
}

Private subnet for Databricks:

resource "azurerm_subnet" "DatabricksSubnetPrivate" {
  name                 = "VirtualNetwork PrivateSubnet Name"
  resource_group_name  = azurerm_resource_group.RG.name
  virtual_network_name = azurerm_virtual_network.DatabricksVnet.name
  address_prefixes     = ["VirtualNetwork PrivateSubnet CIDR"]

  delegation {
    name = "Microsoft.Databricks.workspaces"
    service_delegation {
      name = "Microsoft.Databricks/workspaces"
      actions = [
        "Microsoft.Network/virtualNetworks/subnets/join/action",
        "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
        "Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action"]
    }
  }
}

Network security group for Public subnet:

resource "azurerm_subnet_network_security_group_association" "public" {
  subnet_id                 = azurerm_subnet.DatabricksSubnetPublic.id
  network_security_group_id = azurerm_network_security_group.DatabricksNSG.id
}

Network security group for Privatesubnet:

resource "azurerm_subnet_network_security_group_association" "private" {
  subnet_id                 = azurerm_subnet.DatabricksSubnetPrivate.id
  network_security_group_id = azurerm_network_security_group.DatabricksNSG.id
}

Now as all the associated network configuration done let’s move to the DATA LAKE STORAGE account creation

Data Lake storage account:

resource "azurerm_storage_account" "DataLake" {
  name                     = "DataLake Name"
  resource_group_name      = azurerm_resource_group.RG.name
  location                 = azurerm_resource_group.RG.location
  account_tier             = "DataLake Tier"
  account_replication_type = "DataLake Replication"
  is_hns_enabled           = true
  min_tls_version          = "DataLake TLSVersion"

  network_rules {
    # bypass                     = "AzureServices"
    default_action             = "Allow"    
  }
}

Storage account container:

resource "azurerm_storage_container" "DataLakeContainer" {  
  for_each              = "DataLake Container"
  name                  = each.key
  storage_account_name  = azurerm_storage_account.DataLake.name
  container_access_type = "private"
}

Now, let us create SQL related resources

Storage Admin password:

resource "random_string" "SQLAdminPassword" {
  length      = 5
  special     = true
  min_upper   = 2
  min_numeric = 2
  min_special = 2
}

SQL server :

resource "azurerm_mssql_server" "SQLServer" {
  name                         = "SQLServer Name"
  resource_group_name          = azurerm_resource_group.RG.name
  location                     = azurerm_resource_group.RG.location
  version                      = "SQLServer Version"
  administrator_login          = "SQLServer AdministratorLogin"
  administrator_login_password = random_string.SQLAdminPassword.result
  minimum_tls_version          = "SQLServer  TLS Version"
}

SQL Database:

resource "azurerm_mssql_database" "SQLDatabase" {
  name           = "SQLDatabase Name"
  server_id      = azurerm_mssql_server.SQLServer.id
  collation      = "SQL_collation"
  max_size_gb    = "SQLDatabase MaxSizeGB"
  sku_name       = "SQLDatabase SKU"
  zone_redundant = "SQLDatabase ZoneRedundant"
}

This is a complete part by part snippets to create a running ADB ADF system, feel free to reach me in case any clarification required!

DEV Community: Kunal Das

Setting Up a Comprehensive Python Build Validation Pipeline in Azure DevOps

Setting Up a Comprehensive Python Build Validation Pipeline in Azure DevOps

Step-by-Step Setup

Step 1: Create the Azure Pipeline YAML File

Step 2: Clone the PR Repository

Step 3: Install Dependencies

Step 4: Run Code Quality Checks

Step 5: Run Tests and Generate Coverage Report

Step 6: Check Code Coverage Threshold

Step 7: Publish Coverage Report

Setting Up Local Development Environment

Prerequisites

Setting Up Your Environment

Running Checks Locally

Setting Up Pre-commit Hooks

Configuring isort

Step 1: Create a Personal Access Token (PAT)

Step 2: Add the PAT as a Pipeline Variable

Step 3: Update Your Pipeline YAML

Step 4: Create a Branch Policy

Step 5: Configure Branch Protection

Step 6: Update Repository Settings

Step 7: Educate Your Team

Conclusion

Creating Azure Web App CI/CD with Terraform and Azure DevOps

Step-by-Step Guide: Creating Azure Web App CI/CD with Terraform and Azure DevOps

Read my blogs:

Connect with Me:

Streamlining Synapse CI/CD & Dedicated SQL pool with Azure DevOps

Streamlining Synapse CI/CD & Dedicated SQL pool with Azure DevOps: Best Practices and Implementation Tips

What is Synapse Analytics?

what is the need for CI-CD for synapse analytics?

What are the different components of Synapse Analytics?

why it is hard to implement ci cd for synapse analytics?

Let’s understand the CI -CD Approach:

Continuous Integration :

let’s look at the high-level workflow of the things we have done so far!

CI-CD for dedicated SQL pool

First, let’s setup the repository for the SQL pool

IMPLEMENTING THE RELEASE PIPELINE TO DEPLOY THE ABOVE-GENERATED BUILD ARTIFACTS

Read my blogs:

Connect with Me:

A comprehensive list of Azure Best Practices

Azure Best Practices

Security:

Enable Azure AD Conditional Access and enforce MFA

Disable RDP/SSH from the Internet

Management:

Operations:

Cost Optimization:

Services:

➢ Database

➢ AppService:

Conclusion:

Credits:

Read my blogs:

Connect with Me:

Mastering Private Repositories in Enterprise with GitHub

Mastering Private Repositories in Enterprise with GitHub: A Comprehensive Guide

Getting Started with GitHub and Git

Securing Your Connection with SSH

Additional Tips and Techniques

Leverage GitHub’s Issue Tracker

2. Use Branching Strategically

3. Take Advantage of GitHub Actions

4. Protect Sensitive Information with .gitignore

5. Stay Informed with Webhooks

Read my blogs:

Connect with Me:

Exploring Azure Storage Services

Exploring Azure Storage Services 🌐🗄️

Reach at : https://heylink.me/kunaldas

Table of Contents

Introduction to Azure Storage

Azure Blob Storage

What is Blob Storage? 🤔

Key Features 🌟

Use Cases 🛠️

Azure File Storage