DEV Community: Andrea Grillo

Avoid name conflicts in Terraform workspaces

Andrea Grillo — Tue, 22 Aug 2023 12:11:28 +0000

Terraform workspaces are a feature of some backends to associate many terraform states to the same configuration. It is convenient to have a workspace where you can try changes to infrastructure. When the changes have been tested, you can switch to the production workspace and apply the changes.

Since the configuration files are the same for all workspaces, there may be conflicts among the names of the cloud resources.

For instance, you want to create a AWS security group named allow_http. You switch to the lab workspace and apply the configuration. Now you have a security group named “allow_http” in your AWS account.

You switch to the prod workspace and apply the terraform configuration again. However, Terraform complains that the security group allow_http already exists!

When you use terraform workspaces, you should pay attention to clashes among the names of cloud resources.

At the time of writing this post, the latest version of Terraform is 1.5.5.

Create a terraform resource with a fixed name

You have a Terraform configuration that creates a security group:

resource "aws_security_group" "allow_http" {
  name        = "allow_http"
  description = "Allow HTTP inbound traffic"

  ingress {
    description = "HTTP"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Create the resources in the workspace “dev”

First you create the terraform workspace dev and then you apply the configuration:

$ terraform workspace new dev
$ terraform apply

Copy the id of the security_group somewhere because it will be needed later:

$ terraform state show aws_security_group.allow_http
...
id = "sg-0a89544d88354a70b"
...

Create the resources in the workspace “prod”

Then you want to create the resources in the new terraform workspace prod:

$ terraform workspace new prod

If you make a terraform plan, terraform wants to create the security group again:

$ terraform plan
...
Plan: 1 to add, 0 to change, 0 to destroy.

If you try to apply the changes, it fails because the resource already exists:

$ terraform apply
...
Error: creating Security Group (allow_http): InvalidGroup.Duplicate: The security group 'allow_http' already exists for VPC 'vpc-0688682e4898523a4'
...

Solution 1: Import resources already existing

First solution is to import the existing resource in this workspace as well:

$ terraform import aws_security_group.allow_http sg-0a89544d88354a70b

However, in larger terraform configurations may be complex to import all resources that cannot be duplicated in other workspaces.

Solution 2: Use terraform.workspace in resource names

The second approach is to use the terraform.workspace variable in resource names to avoid duplicate errors:

resource "aws_security_group" "allow_http" {
  name        = "allow_http_${terraform.workspace}"
  description = "Allow HTTP inbound traffic"

  ingress {
    description = "HTTP"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Conclusions

Terraform workspaces are a convenient feature to manage many terraform states related to the same configuration. However, you should use the terraform.workspace variable to avoid conflicts with the resources created by other workspaces.

Discover more posts about Terraform!

Change Kubernetes service type from LoadBalancer to NodePort and viceversa

Andrea Grillo — Fri, 11 Aug 2023 13:55:18 +0000

Changing the Service type from NodePort to LoadBalancer and viceversa is useful depending on the cluster you are working on. If you are doing experiments in a local development cluster, like Kind, you don't have a real load balancer in front of your cluster and you need to expose your services externally using a NodePort service. On contrary, if you are working on a kubernetes cluster with a proper load balancer, you can expose your services through a LoadBalancer service type that uses the load balancer domain name and port.

Convert to a LoadBalancer service

It is very likely to have a load balancer in front of your kubernetes services when you are working in a production cluster or on a kubernetes cluster hosted by a cloud provider. The load balancer is useful to expose your services using a single domain name, the load balancer domain name.

The command to convert a service type to a LoadBalancer service type is:

kubectl patch svc <service-name> -n <namespace> \
  -p '{"spec": {"type": "LoadBalancer"}}'

Convert to a NodePort service

If you are working on a development kubernetes cluster you probably don't have a load balancer to forward the incoming requests to the services. To connect to your services you have to expose them using a Node Port service type that open the same port on each node of the cluster. So to connect to the service in a multi-node cluster, you type the name of a node and the exposed port. Compared to a load balancer, you should know the node names before connecting to the service.

The command to convert a service type to a NodePort service type is:

kubectl patch svc <service-name> -n <namespace> \
  -p '{"spec": {"type": "NodePort"}}'

kubectl port-forward to expose a service locally

kubectl tool contains the port-forward command to expose a service locally on the port we choose. It means you can connect to the service using localhost:port.

The command to expose the service locally is:

kubectl port-forward svc/<service-name> <local-port>:<service-port>

For instance, if you have the service frontend on port 80, to expose it locally on port 8080:

kubectl port-forward svc/frontend 8080:80

The kubectl port-forward is a process running on the foreground. So you need to open a new terminal if you want to issue new commands.

The Argo CD Getting Started guide shows different methods to access the Argo CD Api Server: using a LoadBalancer service type or a exposing the service locally using kubectl port-forward.

I personally used the combination of a NodePort service and kubectl port-forward command when working with Istio Gateways. Istio creates a LoadbBalancer service when you create a Gateway. Since I was working in a Kind cluster, I first changed the service type and then I exposed it locally.

Conclusion

Depending if you a have load balancer or not, you may need to change service type to connect to them.

Discover more posts about Kubernetes!

Don't waste your food again with AiFame

Andrea Grillo — Tue, 08 Mar 2022 21:13:14 +0000

Overview of My Submission

How many times did you forget to eat you food before expiring? Not only it is a waste of money, but also a waste of food that you could have eaten, used to prepare a dinner with your friends or gifted to someone.

We built a web application called AiFame made for the Microsoft Azure Trial Hackathon on DEV 2022. Don't waste your food again!

AiFame allows you to:

Take a picture of your shopping and computer vision algorithms will detect foods automatically
Manage your food inventory like modifying quantity and expiration date.

Try it live here!
Try to manage your inventory or scan foods from a test image.

Submission Category:

We chose to participate in the Computing Captains category because the app was deployed using Azure App Service. The container image is saved in the Azure Container Registry.

Link to Code on GitHub

andregri / AiFame

AiFame is Flask application made for the Microsoft Azure Trial Hackathon on DEV 2022

AiFame

AiFame is web application made for the Microsoft Azure Trial Hackathon on DEV 2022. Don't waste your food again!

AiFame allows you to:

Take a picture of your shopping and computer vision algorithms will analyze it for you to detect food automatically
Manage your food inventory like quantity and expiration date

Try it live here!

output.mp4

Our tech stack

The backend is built with Python Flask and containerized with Docker.
Docker images are pushed to Azure Container Registry and deployed with Azure App Service
User data are stored in Azure SQL Database
Images are uploaded to Azure Storage Account and analyzed with Azure Computer Vision

Authors

View on GitHub

Azure Services we used

The app deployed with App Service interacts with a SQL database to store user data and a storage blob to store the images uploaded by users.

Whenever an image is pushed to the blob, Azure Computer Vision analyzes it to detect objects bounding box. Then, for each bounding box we used again Computer Vision to obtain the food name of the object.

The CICD pipeline uses both GitHub Actions and Azure App Service.
Every push or pull request on development branch triggers the workflow that builds and tests the application:

# This is a basic workflow to help you get started with Actions

name: ci

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the main branch
  push:
    branches: [ main, dev ]
  pull_request:
    branches: [ main, dev ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

      - name: Setup Python
        uses: actions/setup-python@v2.3.1
        with:
          # Version range or exact version of a Python version to use, using SemVer's version range syntax.
          python-version: 3.9
          # The target architecture (x86, x64) of the Python interpreter.
          architecture: x64

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi

Every push on the main branch, the Docker image is built and pushed automatically to the Azure Container Registry:

on: [push]
name: cd

jobs:
    build-and-deploy:
        runs-on: ubuntu-latest
        steps:
        # checkout the repo
        - name: 'Checkout GitHub Action'
          uses: actions/checkout@main

        - name: 'Login via Azure CLI'
          uses: azure/login@v1
          with:
            creds: ${{ secrets.AZURE_CREDENTIALS }}

        - name: 'Build and push image'
          uses: azure/docker-login@v1
          with:
            login-server: ${{ secrets.REGISTRY_LOGIN_SERVER }}
            username: ${{ secrets.REGISTRY_USERNAME }}
            password: ${{ secrets.REGISTRY_PASSWORD }}
        - run: |
            docker build . -t ${{ secrets.REGISTRY_LOGIN_SERVER }}/aifame --file Dockerfile.prod
            docker push ${{ secrets.REGISTRY_LOGIN_SERVER }}/aifame

In addition to that, it is necessary to enable the "Continuous Deployment" option on App Service as show in the image below:

Below there is the list of all resources we used:

Authors

andregri
CaptainMich
Dadigno

Explore how countries produce energy for MongoDB Atlas Hackathon

Andrea Grillo — Sat, 01 Jan 2022 17:03:51 +0000

Overview of My Submission

The climate change is an important topic we should all care to protect our planet. So, I wanted to create a simple web application to explore how countries produce energy, if they rely on renewable energy or fossil fuels.

To visit my web application click here! Similarly to the TV show Our Planet, I wanted to call my app Our Energy because it is a resource we all should be care of!

Below you can see the homepage:

You can look the amount of MegaWatts of energy produced by a country. To find the countries in the Mongo Database I used Atlas Search service. Below you can see the values for my country, Italy:

Moreover, from the homepage you can explore the top 5 countries producing energy using a particular renewable source. You can choose among solar, wind, waves and tidal, and hydro. Below there is a screenshot for the wind energy:

Data preparation

The original dataset was in .csv format. Firstly I removed many columns that were not necessary to my scope to reduce the size. Then I converted it to a JSON file which is easy to be uploaded to MongoDB Atlas.

Connecting to Atlas Cloud is as easy as connecting to any local database thank you to this official guide. To upload the dataset onto Mongo Cloud I used literally one function: insert_many(). You can explore the full code here.

Atlas Search

To search data by country I had to match the input string with the country_long field of the dataset. I used the Atlas UI to create an index with static mapping for this field. You can refer to this guide to create one.

Aggregation pipeline

The dataset consists of many power plants located in many countries. To retrieve a summary information for a certain country I exploited the power aggregation pipeline feature of MongoDB. Look at this code snippet that I use to find the 5 countries producing the biggest quantity of energy from a particular fuel (oil, gas, wind, solar, etc..., renewable energy sources are considered fuels as well):

result = collection.aggregate([
      {
         '$match': {
            'primary_fuel': fuel
         }
      },
      {
         # Accumulate capacity by fuel type
         '$group':
         {
            '_id': '$country_long',
            'totCapacity': {
               '$sum': {'$toDecimal': '$capacity_mw'}
            }
         }
      },
      {
         '$sort': {'totCapacity': -1}
      },
      {
         '$limit': 5
      }
   ])

A pipeline is made of many stages that performs different operations to extract the data we want from the database. The output of a stage is the input of the next stage.

The first step find all the entries whose field 'primary_fuel' matches the string fuel:

{
    '$match': {
        'primary_fuel': fuel
    }
},

This step groups the output of the previous step by country (using the field country_long, the country name) and accumulate the energy production of a power plant. Up to now, we know the amount of wind energy produced by each country.

{
     '$group':
     {
        '_id': '$country_long',
        'totCapacity': {
           '$sum': {'$toDecimal': '$capacity_mw'}
        }
     }
},

We sort the output in descending order

{
    '$sort': {'totCapacity': -1}
},

Eventually, we limit the output to only 5 results

{
     '$limit': 5
}

And in the web app we can see the Top 5 countries producing that kind of energy. For instance, if you want to discover who are the 5 biggest producer of energy from waves and tydal are:

Conclusion

It was exciting to produce a simple web application from scratch to production using Python and MongoDB.

If you want to contribute to my project or propose some improvements, visit the GitHub repository.

Submission Category:

Choose Your Own Adventure

GitHub repository

andregri / OurEnergy

Climate Care is Flask application made for the MongoDB Atlas Hackathon 2021

🌍 OurEnergy

OurEnergy is a web application to explore some meaningful data about the Global Power Plant Dataset. You can explore either how each country is producing energy or little charts comparing the top 5 countries that are distinguishing for producing green energy 🌱

Live Demo

🔧 Tech Stack

Python Flask
MongoDB Atlas
Heroku

🔌 Run Locally

Preparation of the dataset on MongoDB

Environment variables:

export MONGODB_URI='<your connection string to MongoDB'
export GPPDB_PATH='path/to/dataset/csvfile' # Do not write the extension '.csv'

For example my GPPDB_PATH env variable for the file ./data/global_power_plant_database.csv is:

export GPPDB_PATH='./data/global_power_plant_database'

Finally you can run two scripts to convert the dataset to JSON and to upload it to MongoDB Atlas:

python db_preparation/convert.py
python db_preparation/upload.py --cloud

Run Flask app

FLASK_ENV=development FLASK_DEBUG=true FLASK_APP=app flask run

Deploy to production

Create a .env file with the following variables:

export MONGODB_URI='<your

…

View on GitHub

Resources

Global Power Plant Database