DEV Community: Aman Singh

Migrating Projects Between Google Cloud Organizations

Aman Singh — Mon, 10 Feb 2025 18:46:05 +0000

This blog outlines the process of migrating a project from one Google Cloud organization to another, focusing on necessary updates to organizational policies, project migration, VPC and subnet updates, and post-migration test.

Organizational Policies for Export and Import

To allow the export and import of projects between organizations, certain organizational policies need to be configured

Source Organization (Exporting Project)

You need to configure the policy constraints/resourcemanger.allowExportDestinations to allow the export of projects to the destination organization.

Target Organization (Import Project)

The target organization must allow imports using the constraints/resourcemanager.allowedImportSources policy.

Steps to Set Organization policies

Enable Exporting in the Source Organization - Run the following command to configure the source organization resourcemanager.allowImportSources to allow export to the target organization

gcloud resource-manager org-policies allow --organization "SOURCE_ORG_ID" \
resourcemanager.allowImportSources "under:organizations/TARGET_ORG_ID"

Enable Importing in the Target Organization

Similarly, configure the target organization resourcemanager.allowedImportSources to allow imports from the source organization.

glcoud resource-manager org-policies allow --organization "TARGET_ORG_ID" \
resourcemanager.allowedImportSources "under:organizations/SOURCE_ORG_ID"

Analyze resource migration blockers

Before migrating a project, it’s crucial to analyze potential blockers that could prevent a smooth migration. You can use following commmand to analyze resource migration blockers.

glcoud asset analyze-move --project=PROJECT_ID \
--destination-folder=DESTINATION_FOLDER_ID

Link Billing Account to the Project
Make sure the project to be migrated is linked to the correct billing account. Use the following command.

gcloud billing project link PROJECT_ID --billing-account=BILLING_ACCOUNT_ID

Migrate Project to New Organization

Once the necessary policies are set, you can migrate the project using the following gcloud command:

gcloud beta projects move PROJECT_ID --organization=TARGET_ORG_ID

Run Test
Once the migration and configurationm changes are completed, its important to run tests to verify that the migration was successful, the project is functioning properly in the new organization, and that the required networking configurations (VPC, subnets) are correct.

Conduct smoke test to vaildate that the project’s services and APIs are running smoothly in the target organizaiton.
Ensure that all dependencies and network connections are intact.

Summary of Steps

Update Organization policies: Enable export and import permisions between source and target organizations.
Link Billing Account: Ensure the project is linked to the correct billing account.
Migrate Project: Use gcloud beta projects move to move the project to the new organization.
Run Test: Verify that everything works properly in the new organization.
Analyze migration blockers: Use gcloud asset analyze-move to identify andy potenctional migration blockers.

AWS Cert Manager integration with Prometheus with Domain Name

Aman Singh — Tue, 18 Jun 2024 20:29:34 +0000

Problem
When using CloudWatch metrics for ACM (AWS Certificate Manager), there is a limitation in that only the ARN (Amazon Resource Name) of ACM certificates is displayed. This makes it difficult to easily identify which domain is expiring, as ARNs are not human-friendly and are hard to interpret at a glance. Instead, having the domain name displayed would be more user-friendly and would make it easier to manage and monitor certificate expirations.

Solution
One effective solution to this problem is to integrate Prometheus for monitoring ACM certificates. Prometheus allows for more customizable and readable metrics. Here is how you can set up Prometheus to monitor ACM certificates by domain name:

Install Prometheus: First, install Prometheus on your monitoring server or use a managed service like Amazon Managed Service for Prometheus.
Set Up Exporter: Use a custom exporter or an existing one that can fetch ACM certificates details, including domain names. The exporter will query AWS ACM and transform the ARN-based metrics into domain-based metrics.
Prerequisites
Before we dive into the integration process, ensure you have the following:

An AWS account with access to AWS Certificate Manager.
A running Prometheus instance.
Basic understanding of AWS IAM roles and permissions.
Step 1: Setting Up AWS IAM Permissions
To allow Prometheus to access ACM, you’ll need to set up appropriate IAM permissions.

Create an IAM Policy:
Navigate to the IAM console in AWS.

Click on “Policies” and then “Create policy”.

Add the following JSON to allow read-only access to ACM:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "acm:ListCertificates",
        "acm:DescribeCertificate"
      ],
      "Resource": "*"
    }
  ]
}

Create an IAM Role:

Go to the IAM console and click on “Roles” > “Create role”.

Select the “EC2” service, assuming Prometheus is running on an EC2 instance. Attach the policy you created in the previous step and assign role to an instance running Prometheus.

Step 2: Create a python script to fetch ACM details from AWS.

import boto3
import http.server
import socketserver
import json
import time
import logging

PORT = 9102

# Set up logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

class ACMExporter(http.server.SimpleHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/metrics':
            self.send_response(200)
            self.send_header('Content-type', 'text/plain')
            self.end_headers()
            try:
                metrics = self.generate_metrics()
                self.wfile.write(metrics.encode())
            except Exception as e:
                logging.error(f"Error generating metrics: {e}")
                self.send_response(500)
                self.end_headers()
        else:
            self.send_response(404)
            self.end_headers()

    def generate_metrics(self):
        acm_client = boto3.client('acm', region_name='us-west-1')  # Change the region if necessary
        try:
            response = acm_client.list_certificates()
        except Exception as e:
            logging.error(f"Error listing certificates: {e}")
            raise

        certificates = response.get('CertificateSummaryList', [])

        metrics = []
        for cert in certificates:
            try:
                expiration_time = int(cert['NotAfter'].timestamp())
                cert_arn = cert['CertificateArn']
                domain_name = self.get_certificate_domain(acm_client, cert_arn)
                metrics.append(f'acm_cert_expiration_timestamp{{domain_name="{domain_name}"}} {expiration_time}')
            except Exception as e:
                logging.error(f"Error processing certificate {cert['CertificateArn']}: {e}")

        return '\n'.join(metrics)

    def get_certificate_domain(self, acm_client, certificate_arn):
        try:
            response = acm_client.describe_certificate(CertificateArn=certificate_arn)
            certificate_detail = response['Certificate']
            domain_name = certificate_detail['DomainName']
            return domain_name
        except Exception as e:
            logging.error(f"Error describing certificate {certificate_arn}: {e}")
            raise

def run(server_class=socketserver.TCPServer, handler_class=ACMExporter):
    server_address = ('', PORT)
    httpd = server_class(server_address, handler_class)
    logging.info(f'Starting ACM exporter on port {PORT}...')
    httpd.serve_forever()

if __name__ == "__main__":
    run()

Now, run the script using nohupcommand. Before running command install all the required libraries using pip3 .

nohup python3 cert-manager-metric.py &

Once above script is running using PORT: 9102 . Use curl command to check if this script is getting metric or not.

curl http://localhost:9102

Step 3: Configure Prometheus to read metrics
Edit the Prometheus configuration file and following lines to

scrape_configs

  - job_name: 'acm-exporter'
    static_configs:
      - targets: ['localhost:9102']

Once you are done with above configuration, restart Prometheus service

systemctl restart prometheus
And you are done. Now go to the Prometheus and check the metrics.

GCP Landing zone for FI

Aman Singh — Fri, 09 Sep 2022 21:24:46 +0000

HI Everyone,

With the continuous efforts and lots of teamwork. We are able to achieve the end product. From past 9 months I was working on a project to create landing zone for Google Cloud Platform which should be Complient to fit for financial institution.
Today, I will not write much but requesting you all to Kindly visit below GitHub link and try it.

GitHub Link

Scan Terraform with OPA

Aman Singh — Thu, 02 Jun 2022 13:42:45 +0000

Recently, I started working on compliance projects, and it became necessity to discover any tool which can show the written IAC Terraform code is compliant to certain standards such as CIS, SOC2 etc. And There I got introduced to regula, and my journey started in the world of compliance.

In this blog will take about with is Open agent policy ? What is Rego? And how they can be used with regula to generate report.

What is OPA?
The Open Policy Agent (OPA, pronounced “oh-pa”) is an open source, general-purpose policy engine that unifies policy enforcement across the stack.OPA provides a high-level declarative language that lets you specify policy as code and simple APIs to offload policy decision-making from your software. You can use OPA to enforce policies in microservices, Kubernetes, CI/CD pipelines, API gateways, and more.

To understand this better, we need to understand rego.

Introduction to Rego
Rego was inspired by Datalog, which is a well understood, decades old query language. Rego extends Datalog to support structured document models such as JSON.

Rego queries are assertions on data stored in OPA. These queries can be used to define policies that enumerate instances of data that violate the expected state of the system.

Why use rego

1. Easy to read and write.
2. Powerful support for referencing nested document
3. declarative

Regula
Regula is a tool that evaluates infrastructure as code files for potential AWS, Azure, Google Cloud, and Kubernetes security and compliance violations prior to deployment.

Regula supports the following file types:

CloudFormation JSON/YAML templates
Terraform HCL code
Terraform JSON plans
Kubernetes YAML manifests
Azure Resource Manager (ARM) JSON templates (in preview)
Install regula

Download the Regula archive for your platform from the Releases page.
Extract the downloaded archive.
Move the extracted regula binary to somewhere in your PATH:
Also, one can useHomebrew
Mac:

#brew install regula

Linux:

#sudo mv regula /usr/local/bin
OR
#brew install regula

Windows cmd:

md C:\regula\bin 
move regula.exe C:\regula\bin 
setx PATH "%PATH%;C:\regula\bin"

Now, let’s jump into the main scenario of this blog. In today’s world, we are using IAC to write and deploy our IAC to cloud environment. But, for most of the cases we need to check whether our code is compliant and has all parameters secured before deploying over cloud, here comes OPA to help.

Let write some terraform script. Here, we are creating AWS ec2 instance with certain details.

resource "aws_instance" "instance" {
  ami                         = "ami-0ddb956ac6be95761"
  instance_type               = "t2.small"
  key_name                    = "key-pair-name"
  vpc_security_group_ids      = "sg-name
  subnet_id                   = "subnet-xxxxxxx"
  associate_public_ip_address = true

  root_block_device {
    volume_size           = 50
    delete_on_termination = true
  }

  tags = {
    Name = "demo-server"
  }
}

Let, compare above terraform code with OPA rule, to write a rule we need to define a rule with file extension rego

package rules.tf_aws_ec2_instance_no_public_ip

__rego__metadoc__ := {
   "id": "POLICY_ID01",
   "title": "Ensure instance have no public IP associated",
   "description": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
   "custom": {"severity": "Low"},
}

resource_type := "aws_instance"

default allow = false

allow {
   input.associate_public_ip_address == false
}

let’s go through the above rule,

package :- Packages group the rules defined in one or more modules into a particular namespace. Because rules are name-spaced, they can be safely shared across projects.
rego__metadoc :- It enhances rule or policy reporting, with details like id , title description custom severity etc. refer.
resource_type :- the resource you are looking at in terraform.
Above rule will check if associate_public_ip_address is set to false is so, Then it will all the scan to pass. If not, it will prompt an error like below.

Run regula command against the main.tf file.

#regula run main.tf
POLICY_ID01: Ensure instance have no public IP associated [Low]

       in main.tf:1:1

If required, you can get this output in JSON format as well.

#regula run main.tf -f json
"rule_id": "POLICY_ID01",
      "rule_message": "",
      "rule_name": "tf_aws_ec2_instance_no_public_ip",
      "rule_raw_result": false,
      "rule_result": "FAIL",
      "rule_severity": "Low",
      "rule_summary": "Ensure instance have no public IP associated",
      "source_location": [
        {
          "path": "main.tf",
          "line": 1,
          "column": 1
        }
      ]
    },

You can also run regula with tf_plan file. For that, we need to run plan and exact plan output file.

#terraform plan -out="plan.tfplan"
#terraform show -json plan.tfplan > plan.json

once you have plan.json a file in JSON format, you can compare it with regula. The output will be the same.

#regula run plan.json

Conclusion

It is very important to use IAC to make your deployment faster and robust. But, at the same time it’s also import to check whether it's compliant with standards or not and OPA is a good solution to help you get compliant & and one can mold it as per their use case.

Reference

OPA — https://www.openpolicyagent.org/docs/latest/
regula — https://regula.dev/