DEV Community: Binoy

Create the Admin Terraform Module and Terraform Admin User

Binoy — Sun, 20 Jul 2025 17:34:59 +0000

Purpose
This document outlines the architectural decision to create a dedicated Terraform admin module and an admin user, which are responsible for provisioning administrative resources, roles, and users. These components are foundational and will support other Terraform modules across different functional areas.
The key objective is to enable modular, independently deployable infrastructure components that avoid cross-environment impact and align with best practices in security, maintainability, and compliance.

Scope and Context
The Terraform admin module will provision privileged users and roles that are categorized to support the following functional areas:

Shared Resources: Networking components such as VPCs, subnets, domains, firewall rules, etc.
DevOps Resources: Tools and services needed for development pipelines and deployment operations.
Monitoring Resources: Centralized logging, alerting, and observability for management, development, and production environments.
Application Environments:
- Development
- Production

By segregating users and roles by domain and environment, we reduce the risk of cross-impact between critical systems. For example, changes in the development environment must not affect production.

Security and Compliance Objectives
This approach supports the following best practices and compliance requirements:

Least Privilege Principle
1. Users and roles are scoped with minimal necessary access, and can be reused across modules responsible for different operational domains.
Maintainability, Testability, and Deployability
1. Each module can be developed, tested, and deployed in isolation.
Resource Clean-up
1. Modularization simplifies identifying and removing obsolete or unused resources.
Documentation of Administrative Tasks
1. Automation replaces manual configuration, ensuring consistency and clarity.
Auditability and Monitoring
- All administrative activities are captured via cloud audit logs.
- Role-based access control enforces accountability.
- Environment separation aids incident analysis and root cause tracing.
- Enable the log anomaly detection for log review
Single Responsibility and Bounded Context
- Each module, user, and role has a clearly defined scope and responsibility.

ISO & Regulatory Compliance Alignment
This design supports alignment with ISO/IEC 27001 and 27002 standards, as well as industry benchmarks such as CIS. Specifically:

ISO/IEC 27001:
- ✔️ Segregation of duties (A.6.1.2)
- ✔️ Access control policy (A.9.1)
- ✔️ Secure user access management (A.9.2)
- ✔️ Monitoring and logging (A.12.4, A.12.7)
ISO/IEC 27002:
- ✔️ Emphasis on least privilege, auditability, accountability
- ✔️ Secure administration of systems and services

Audit capability must cover who, when, where, and which changes—which is addressed by enabling cloud-native audit logging (e.g., CloudTrail, Activity Logs) and alerting on sensitive operations.

Prerequisites

Root User Setup
Root user should only be used to provision the initial admin-terraform-deployer user and role:
- User: admin-terraform-deployer
- Role: admin-terraform-deployer-role
Root Usage Mitigation
- Avoid using the root account in ongoing operations, per CIS security benchmarks.

Admin Module Responsibilities

Provision the admin-terraform-deployer user/role.
Create and manage the KMS key (admin-terraform-kms-key) for encryption and rotation.
Provision the Terraform state GCP bucket bucket with:
- Versioning enabled (retain last 5 versions)
- Retention and deletion policies
- At-rest encryption using admin-terraform-kms-key
- Separate buckets or folders for dev, prod, and management states
Store secrets in a Secrets Manager with encryption using the KMS key
Enable Cloud Audit Logging for all admin operations
Define a retention policy for logs (e.g., 365 days)
Configure alerting and monitoring for:
- Changes to GCP bucket state files
- Modifications to the KMS key or key rotations
- Changes to IAM roles and user privileges

Mitigation Strategies
To address risks and challenges:

Maintain clear documentation of:
- Terraform modules
- User and role definitions
- Execution flows and state handling
Enforce Terraform state security:
- Use encrypted remote backends
- Implement state locking and restricted access policies
Implement robust monitoring and alerting
Schedule regular security audits to ensure compliance with least privilege principles

Data Compliance (PII and PHI) Network Architecture

Binoy — Mon, 14 Jul 2025 18:25:07 +0000

This architecture is designed to maintain PII (Personally Identifiable Information) data compliance, adhering to strict security and regulatory requirements. It leverages a multi-VPC, shared networking approach with granular subnet isolation and robust security controls.

1. VPC Structure and Isolation
The architecture employs a logical separation of environments through dedicated Virtual Private Clouds (VPCs):

Development VPC: Hosts development and QA environments.
Management (DevOps & Monitoring) VPC: Centralized VPC for DevOps tooling, monitoring, and administrative tasks across all environments.
Staging VPC: (Implicitly similar to Production VPC configuration) A pre-production environment for rigorous testing before deploying changes to production.
Production VPC: Dedicated for live production environments handling critical applications and PII data.

This multi-VPC strategy ensures strong isolation, preventing unauthorized access and impact between different stages of the software development lifecycle.
2. Shared Networking Approach
The design adopts a shared networking approach, enabling the deployment of multiple projects within the same network infrastructure. This promotes reusability and simplifies network management while maintaining segregation through strict access controls at the project and subnet levels.
3. Application Deployment (Development & Production VPCs)
Applications deployed within the Development and Production VPCs adhere to a highly segmented subnet strategy with strict Network Access Control Lists (NACLs) to enforce an "air gap" and minimize the attack surface. This granular isolation ensures data compliance and auditability.
Subnet Isolation:

Public Subnet: Hosts internet-facing resources.
Frontend Subnet: Runs all frontend-related services.
Microservices Subnet: Dedicated for application-related microservices.
Controller Subnet: Manages Kubernetes master nodes, VM discovery services, and other cluster control resources.
DB Subnet: Contains database resources.
PII Service Subnet: Specifically for services handling PII data.
PII DB Subnet: Stores PII databases.
Tools & Monitoring Subnet: For application-specific tools and monitoring resources.
External Service Subnet: For external API services requiring internet access. All other private subnets are strictly isolated without direct internet access.

This stringent network segmentation ensures logical separation and single responsibility for each subnet, simplifying monitoring and auditing.
PII Data Compliance Controls:

Strict Access Controls: PII resources have stringent roles and permissions, controlling which services and ports can connect. For example, PII services have access only with the Microservices Subnet.
Authentication and Authorization: Strictly monitored via ID tokens, enabling detailed auditing of "who, where, and when" access, critical for data compliance and regularity.
Service Mesh Architecture: Implemented for centralized end-to-end encryption (transit-level encryption) and comprehensive monitoring of requests and responses.
PII Data Classification: PII data is classified, and access is understood (e.g., email and phone numbers decrypted only when sending emails or SMS).
Encryption at Rest and Column-Level Encryption: PII service and DB implement both column-level and encryption at rest.
PII DB Visibility: PII DB provides visibility into data access, recording which columns are accessed by which microservices during queries.

4. Management VPC (DevOps & Monitoring)
The Management VPC centralizes resources for monitoring all environments and managing DevOps operations, including software and application deployment to development and production.
Subnet Isolation:

Public Subnet: Hosts internet-facing management resources.
DevOps Subnet: Runs DevOps tools (e.g., Jenkins, Packer, Puppet).
Artifact Subnet: Manages artifact repositories (e.g., JFrog, Nexus, Harbor).
Monitoring Subnet: Hosts monitoring services (e.g., Prometheus, Thanos, Grafana, Elastic Service).
Controller Subnet: Manages Kubernetes master nodes and other control resources for the management cluster.
DB Subnet: Contains database resources for management tools.

High-Level Responsibilities of Management VPC:

Golden Image Creation: Generates base images for development and production VMs.
Vulnerability Checks: Performs vulnerability scanning of software and libraries before deployment.
SAST and DAST Execution: Conducts Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST).
Synthetic Testing: Executes synthetic transactions to monitor application availability and performance.
Artifact Maintenance: Manages central artifact repositories.
Resource Monitoring: Monitors resources and services across all environments.
Resource Provisioning: Provisions resources in development and production.
Source Code Management: Manages source code repositories.
Code Pipeline: Manages CI/CD pipelines for software, patching, application, container, and configuration deployments.
Temporary Agents/VMs/Containers: Creates temporary resources for building applications, golden images, and vulnerability checks.
Software Management Tools: Runs tools like SaltStack, Puppet, and Ansible for configuration management.
Security Patching & Versioning: Manages security patching and versioning of golden images.
Centralized Artifact Management: Manages artifacts for central repositories (e.g., Maven, Pip).

5. Common Networking Security and Monitoring
This architecture incorporates common security and monitoring controls across all environments:

SSH Tunnel: For secure access to private VMs.
Common Firewall Rules: Consistent firewall policies across environments.
Flow Logs: To monitor network traffic and identify anomalies.
Log Monitoring: Centralized logging for comprehensive auditing and troubleshooting.
Alerting: Based on monitoring logs for proactive incident response.
Web Application Firewall (WAF): To protect internet-facing applications from common web exploits.
Common IAM Roles: For consistent management of networking resources.
Shared Resources: Secure sharing of resources via buckets between environments. ## High-Level Quality Attributes and Compliance

This networking architecture is designed with the following quality attributes and compliance standards in mind:

ISO Standard Meeting: The stringent controls, documentation, and auditing capabilities inherent in this architecture contribute significantly to meeting various ISO standards, particularly ISO 27001 for Information Security Management Systems. The focus on data classification, access control, and continuous monitoring aligns with ISO 27001 principles.
Air Gap Architecture: The strict network segmentation, particularly the absence of direct internet access for most private subnets and the isolation of PII data, effectively creates an "air gap" for sensitive data. This significantly reduces the potential attack surface.
Reduced Surface Attack: Through micro-segmentation, minimal open ports, strict NACLs, and the isolation of PII data, the attack surface is drastically minimized. This limits the potential entry points for malicious actors and contains the blast radius of any successful attack.
Monitoring: Comprehensive monitoring is central to this architecture.
- Network Flow Logs: Provide visibility into network traffic patterns.
- Centralized Logging: For all application, system, and security events.
- Application Performance Monitoring (APM): Via service mesh and dedicated monitoring tools.
- Security Information and Event Management (SIEM): (Implied by log monitoring and alerting) for correlation and analysis of security events.
- Alerting: Proactive notification of security incidents and operational issues.
Data Compliance (PII and PHI) and Data Regularity (GDPR, CCPA):
- PII/PHI Segregation: Dedicated subnets for PII services and databases are a cornerstone.
- Encryption: In-transit (service mesh) and at-rest (column-level and disk encryption) for PII/PHI.
- Access Control: Granular, audited access controls to PII/PHI resources, enforced through IAM and service mesh.
- Auditability: Detailed logging of all access to PII/PHI, enabling full audit trails for compliance reporting.
- Data Classification: Understanding and classifying PII/PHI to apply appropriate controls.
- Regulatory Alignment: The architecture's emphasis on data minimization, access control, encryption, and audit trails directly supports compliance with regulations like GDPR (General Data Protection Regulation) and CCPA (California Consumer Privacy Act).
Security: Multi-layered security approach:
- Network Security: VPCs, subnets, NACLs, Firewall Rules, WAF.
- Identity and Access Management (IAM): Least privilege principle, ID token-based authentication.
- Data Encryption: In-transit and at-rest.
- Vulnerability Management: SAST, DAST, Golden Image vulnerability checks.
- Security Monitoring: Flow logs, centralized logging, alerting.
- Patch Management: Automated patching of golden images.
Auditing:
- Comprehensive Logging: All network traffic, system events, application logs, and access attempts are logged.
- Immutable Logs: Logs are designed to be tamper-proof for integrity.
- Centralized Log Management: For easy access and analysis during audits.
- Audit Trails: Detailed records of all changes, deployments, and access attempts.
Testability:
- Staging Environment: A replica of production for rigorous testing of changes before deployment.
- Isolated Development/QA Environments: Allow for independent testing without impacting production.
- Synthetic Testing: Automated checks for application health and performance.
Deployment:
- Automated CI/CD Pipelines: Managed by the Management VPC for consistent and repeatable deployments.
- Golden Images: Ensures consistent and secure base environments.
- Infrastructure as Code (IaC): (Implied by automated provisioning and configuration management tools) for repeatable and version-controlled infrastructure.
Performance:
- Distributed Architecture: Microservices and segregated databases can be scaled independently.
- Network Design: Efficient routing within VPCs and subnets.
- Monitoring: Proactive identification and resolution of performance bottlenecks.
Cost:
- Shared Networking: Can lead to cost efficiencies by centralizing certain network services.
- Resource Optimization: Monitoring and management tools help optimize resource utilization.
- Scalability: Allows for scaling resources up or down based on demand, potentially reducing unnecessary expenditure.
- Operational Efficiency: Automation of deployment and management tasks reduces operational costs.

DevOps - Software Deployment Pipeline

Binoy — Sun, 13 Jul 2025 16:55:35 +0000

DevOps - Software Deployment Pipeline

This diagram outlines a robust DevOps software deployment pipeline, seemingly designed for a GCP-K8s environment, with a strong emphasis on security, performance, and maintainability.

DevOps Software Deployment Pipeline Explanation:

Code Version [Git]: The pipeline begins with version-controlled source code, likely stored in a Git repository. This is the single source of truth for all software assets.
Code Pipeline [Jenkins]: Jenkins acts as the central orchestration engine for the entire pipeline. It triggers builds and manages the flow of the software through various stages.
Path 1: Infrastructure as Code (IaC) for Temporary VMs:
- Create Temp VM [Terraform]: Jenkins initiates the creation of temporary Virtual Machines (VMs) using Terraform. This indicates an IaC approach, ensuring consistency and repeatability in VM provisioning.
- Vulnerabilities check Softwares and Tools [VM Agent Temporary]: Once a temporary VM is up, an agent on the VM performs security checks, including vulnerability scanning of installed software and tools. This is a crucial security control early in the pipeline.
- Artifact [Sonatype Nexus Repository]: After the security checks, artifacts (potentially vetted software components or dependencies) are stored in Sonatype Nexus, an artifact repository. This ensures that only approved and scanned components are used downstream.
Path 2: VM Image Building and Configuration Management:
- Puppet & SaltStack: These tools represent configuration management. They are used to configure and provision the VMs with necessary software and settings, ensuring consistency across environments.
- Packer - Build Image: Packer is used to automate the creation of machine images (e.g., AMI for AWS, custom images for GCP). It takes the configured VMs (from Puppet/SaltStack or a temporary VM) and bakes them into reusable images.
- Create Temp VM [Terraform]: Another instance of Terraform creating temporary VMs, likely to facilitate the image building process with Packer. This temporary VM might be where Puppet/SaltStack configurations are applied before Packer creates the final image.
Create Golden Images [VM Agent Temporary]: This is a critical step where "Golden Images" are created. These are hardened, pre-configured, and tested VM images that contain the application and all its dependencies, along with security configurations. The "VM Agent Temporary" here might imply final validation or hardening steps on a temporary VM before the image is finalized. This also likely incorporates the vetted artifacts from Sonatype Nexus.
Upload Golden Image - Artifact [Sonatype Nexus Repository]: The Golden Images are then uploaded to a version-enabled Artifact. Versioning is crucial for rollbacks and maintaining a history of images.
Deploy Golden Image on Dev / Production VMs: Finally, the Golden Images are deployed onto Development, Staging, and Production VMs. The "VPCs" (Virtual Private Clouds) mentioned in the prompt are implied here, indicating separate, isolated network environments for each stage.
Code Pipeline [Jenkins]: Jenkins again orchestrates this final deployment step, ensuring that the correct Golden Image is deployed to the appropriate environment.

Quality Attributes Explanation:

1. Security (Secure Complaints where No Air Gap Architecture):

Vulnerability Scanning (Early Detection): The "Vulnerabilities check Softwares and Tools [VM Agent Temporary]" step is crucial. By performing these checks early on temporary VMs, the pipeline identifies and addresses security flaws before they are baked into Golden Images. This is vital in a "no air gap" architecture where direct internet exposure might be present.
Golden Images for Reduced Attack Surface: Golden Images are inherently more secure than building VMs from scratch for each deployment. They are pre-hardened, patched, and include only necessary components, significantly reducing the attack surface.
Artifact Repository (Sonatype Nexus): Using Nexus ensures that all external dependencies and internal artifacts are scanned and approved before use. This prevents the introduction of known vulnerabilities from third-party libraries.
Configuration Management (Puppet & SaltStack): These tools enforce desired state configurations, preventing configuration drift and ensuring security baselines are consistently applied.
IaC (Terraform): Terraform defines infrastructure as code, which can be version-controlled and peer-reviewed, reducing manual errors that often lead to security misconfigurations.
Security Patching: The architecture facilitates easy security patching. By updating the base image or specific components within the Packer/Puppet/SaltStack stages, a new Golden Image can be created and rolled out, ensuring all VMs receive the latest patches. This process is automated, reducing the time to patch.
Monitoring VPC: The mention of a "monitoring VPC" implies dedicated infrastructure for security monitoring, intrusion detection, and log analysis, providing continuous visibility into the environment's security posture.

2. Performance for Deploying Software:

Automated Pipeline: The entire pipeline is automated, significantly reducing deployment time compared to manual processes.
Golden Images for Rapid Deployment: Deploying pre-built Golden Images is much faster than provisioning and configuring VMs from scratch for each deployment.
Spot Instances for Temporary VMs: Utilizing "spot instances" for temporary VMs (as mentioned) is a cost-effective strategy for burstable workloads like image building and vulnerability scanning. While spot instances can be interrupted, the temporary nature of these VMs and the idempotent nature of the image building process make them suitable for this use case.
Parallelism (Implicit): Jenkins can orchestrate parallel execution of certain steps (e.g., multiple temporary VMs for different vulnerability checks), further speeding up the process.

3. Deployment:

Automated and Repeatable: The pipeline ensures consistent and repeatable deployments across all environments (Dev, Staging, Production).
Versioned Deployments: Golden Images are versioned, enabling easy rollbacks to previous stable versions if issues arise in a new deployment.
Multi-Environment Support: The pipeline explicitly supports deployment to Dev, Staging, and Production VPCs, allowing for a phased release strategy.

4. Maintainability:

Infrastructure as Code (Terraform): Managing infrastructure through code makes it easier to understand, modify, and maintain.
Configuration Management (Puppet & SaltStack): Centralized configuration management simplifies updating and maintaining the software and configurations on VMs.
Modular Design: The pipeline appears modular, with distinct stages for code, image building, security checks, and deployment, making it easier to troubleshoot and update specific parts without affecting the entire pipeline.
Versioned Artifacts: Storing artifacts in Nexus and Golden Images in a version-enabled storage bucket simplifies tracking and managing different versions.

5. Cost:

Spot Instances for Temporary VMs: As mentioned, using spot instances for temporary VMs significantly reduces the cost of compute resources during image building and scanning phases.
Automated Processes: Automation reduces manual effort and associated labor costs.
Optimized Resource Utilization: By using temporary VMs only when needed, the pipeline avoids continuously running expensive instances.
Golden Images for Efficient Resource Use: Golden Images reduce the need for extensive post-deployment configuration, leading to faster startup times and potentially less compute time needed for new instances.

6. Monitoring:

Monitoring VPC: The explicit mention of a "monitoring VPC" indicates a dedicated environment for collecting metrics, logs, and traces from all other VPCs and pipeline components. This allows for comprehensive operational monitoring and alerting.
VM Agent Temporary: The "VM Agent Temporary" suggests agents are deployed on temporary VMs, which can collect logs and metrics during the build and scan phases, providing insights into the pipeline's health and security.
Jenkins Dashboard: Jenkins provides a central dashboard for monitoring the status of pipeline runs, build failures, and deployment successes.

7. Testability:

Automated Security Checks: The "Vulnerabilities check Softwares and Tools" step represents automated security testing.
Golden Image Creation (Implicit Testing): The creation of Golden Images implies that these images undergo rigorous testing (functional, performance, security) before being deemed "golden."
Staging Environment: The deployment to "Dev / Production VMs" implicitly includes a staging environment (often between Dev and Production) where more extensive testing (e.g., integration, performance, user acceptance testing) can occur before deploying to production.
Reproducible Environments (IaC): Terraform ensures that environments are reproducible, making it easier to isolate and debug issues during testing.

ISO Standard

The architecture have provided demonstrates many practices and controls that are highly aligned with the requirements of these ISO standards:

Strong Security Controls (ISO 27001 & 27701):
- Vulnerability Scanning: "Vulnerabilities check Softwares and Tools" directly addresses a key control for identifying and mitigating security weaknesses.
- Golden Images: Creating hardened, pre-configured images reduces the attack surface, improves consistency, and ensures that security configurations are applied uniformly, which are all critical for ISO 27001.
- Configuration Management (Puppet, SaltStack): These tools ensure consistent and secure configurations, preventing drift and enforcing security baselines as required by ISO.
- Infrastructure as Code (Terraform): Version-controlled and auditable infrastructure definitions contribute to the "plan-do-check-act" cycle of ISO 27001 by making infrastructure changes transparent and reviewable.
- Artifact Repository (Nexus): Managing and scanning artifacts helps ensure the integrity and security of software components, preventing the introduction of vulnerable dependencies.
- Segregated VPCs and Monitoring VPC: Network segmentation and dedicated monitoring infrastructure are fundamental for isolating environments and detecting security incidents, both core to ISO 27001.
- Automated Patching: The ability to easily perform security patching via Golden Images is crucial for maintaining security posture and addressing vulnerabilities in a timely manner.
- Access Controls (Implicit): While not explicitly detailed, the pipeline implies proper access controls for Jenkins, Git, Nexus, and cloud environments, which are central to ISO 27001.

Run the K8s in the private VPC setup

Binoy — Sat, 14 Jun 2025 18:54:24 +0000

Overview

This section explains how to set up a Kubernetes (K8s) cluster on Google Cloud Platform (GCP) using Terraform. Instead of using GCP's managed Kubernetes service, we will install open-source Kubernetes manually inside virtual machines (VMs).

Objective

In software and microservice architecture design, it is essential to consider the scalability, availability, and performance of the application. I am proposing the following design considerations when selecting Kubernetes as part of the software architecture:

When the application requires high performance, availability, scalability, and security
When greater control over and maintenance of the infrastructure is needed
When the application architecture involves complex microservices
When opting for open-source solutions (optional)

When Kubernetes may not be the right choice:

If the application is small and can be supported by a cloud provider’s managed services, such as AWS ECS with Fargate (serverless technology)
If the application does not require high throughput
If the goal is to minimize IT operational costs and reduce infrastructure management responsibilities

In this setup, I considered the following points while building Kubernetes (K8s) on GCP:

Terraform: Used to easily deploy, manage, and destroy the cluster infrastructure.
HAProxy: Implemented an open-source load balancer instead of relying on GCP’s native load balancer. HAProxy provides high-performance load balancing.
Consul: Implemented VM discovery to automatically register Kubernetes worker nodes with the HAProxy load balancer.
GCP Auto Scaling and VM Health Checks: Set up an autoscaling group with TCP-based health checks to ensure the availability and reliability of virtual machines.
GCP RBAC: Leveraged GCP’s Role-Based Access Control to simplify node joining, manage Kubernetes-related files in GCP Buckets, and associate service accounts with bucket roles and virtual machines.
Minimal Permissions: As a best practice, configured minimal necessary roles for infrastructure components to enhance security.
Firewall Rules: Define and control inbound (ingress) and outbound (egress) network traffic to secure communication between resources.
Private VPC: Create a dedicated Virtual Private Cloud (VPC) to isolate and secure resources, avoiding use of the default VPC. Resources in the private VPC, such as VMs, do not require external IP addresses and are accessed via internal IPs.
IAP (Identity-Aware Proxy) TCP Forwarding: Enable secure SSH access to virtual machines running in private subnets without exposing them to the public internet.
Private Google Access: Allow resources in private subnets to access Google services (e.g., Cloud Storage) over internal IPs by enabling this option at the subnet level.
Public and Private Subnets: Segregate application components and manage network traffic more securely by deploying resources into distinct public and private subnets.
Impersonate Service Account: Set up a service account that can be impersonated, granting it the necessary permissions to create and manage resources in Google Cloud Platform (GCP).
- Advantage: This approach enhances security by allowing controlled, auditable access without needing to share long-lived credentials. It also enables role-based delegation, allowing users or services to act with limited, predefined permissions.

Network Archiecture

Infrastructure Archiecture

Note: This is not a production-ready architecture. In production environments, you should create your own VPCs, such as separate networks for Management and Production environments (for eg: following HIPAA-compliant recommended network architecture practices).

Prerequisites

We have to setup the following tools and accounts

Create GCP account [we can create free tier account]
Install Terraform in your machine [Min Terraform v1.9.3]
Install gcloud command line in your machine

Create the servie account and permission in GCP

Enable Identity and Access Management (IAM) API
1. Navigate to "APIs & Services": In the left-hand navigation menu, click on APIs & Services.
2. Go to the "Library": On the "APIs & Services" overview page, click on + ENABLE APIS AND SERVICES or select Library from the sub-menu on the left.
3. Search for the IAM API: In the search bar, type "Identity and Access Management (IAM) API".
4. Select the API: Click on the "Identity and Access Management (IAM) API" from the search results. You'll be taken to the API's overview page.
5. Enable the API: On the API overview page, you will see an Enable button if the API is not already enabled for your project. Click this button.
Select the project
Select GCP->Service Account
1. Click "Create service account"
2. Give service account name is "terraform-deployer"
3. Give service account id is "terraform-deployer"
4. Give service descripton is "Terraform deployer for creating the resources and networking"
  1. Output will be Email Id : terraform-deployer@{project}-{project_id}.iam.gserviceaccount.com
Add grand for root user
1. GCP->IAM
2. Click "Grand Access"
3. Give new principle name "root@email.com"
4. Assign Roles: Click “Select a role” dropdown.
  1. Common roles:
  2. Service Account Token Creator - Generating access tokens, Calling GCP services on behalf of the service account
  3. IAP Role
    1. IAP-secured Tunnel User - This is the essential role. It allows a user or service account to connect to resources (like Compute Engine VMs) through IAP's TCP forwarding feature.
    2. Compute OS Admin Login - (basic user access, no sudo)
    3. Compute OS Login - (admin access, with sudo)
Add grand for service account of terraform deployer of management to access shared service resources
1. GCP->IAM
2. Click "Grand Access"
3. Give new principle name "terraform-deployer@{project}-{project_id}.iam.gserviceaccount.com"
4. Assign Roles: Click “Select a role” dropdown.
  1. Common roles:
  2. Viewer – for read-only access
  3. Editor – for general write access
  4. Service Account Admin – for service account admin access
  5. Compute Network Admin – for networking admin access
  6. Compute Security Admin – for security admin access
  7. Service Account Token Creator - Generating access tokens, Calling GCP services on behalf of the service account
  8. Compute Instance Admin (v1) - for create instance

Login GCP account

gcloud auth login

Create the configuration gcloud to separate/isolate the configuration for the project

gcloud config configurations create devops-k8s
gcloud config configurations activate devops-k8s

Authenticate for a Specific Project To log in and set a GCP project:

gcloud config set project <PROJECT_ID>  --configuration=devops-k8s

Replace GCP PROJECT_ID with your actual project.

Note: Please follow the procedure if we are facing quota issues

* WARNING: Your active project does not match the quota project in your local Application Default Credentials file. This might result in unexpected quota issues.
* Update the quota project associated with ADC:

gcloud auth application-default set-quota-project <PROJECT_ID> 
gcloud config set project <PROJECT_ID> --configuration=devops-k8s
#If we are using env variable CLOUDSDK_CORE_PROJECT in .bash_profile, need to unset
unset CLOUDSDK_CORE_PROJECT
gcloud auth login --configuration=devops-k8s

Configure the `env` variables

Following environment variables need to configure.

GOOGLE_IMPERSONATE_SERVICE_ACCOUNT: This variable configure the impersonate service account
TF_VAR_gcp_project_id: This variable to configure the GCP project id to use the terraform

vi ~/.bash_profile
export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=terraform-deployer@{project}-{project_id}.iam.gserviceaccount.com
export set TF_VAR_terrafor_impersonate_service_account=terraform-deployer@{project}-{project_id}.iam.gserviceaccount.com
export set TF_VAR_gcp_project_id={your-gcp-project-id}
source ~/.bash_profile

Create the Terraform workspace

The following command is used to create a Terraform workspace. A workspace in Terraform functions similarly to managing different environments in software development, such as dev and prod. In this setup, the workspace helps differentiate resources for each environment within the same GCP project.

We are following a consistent naming convention for resource creation in the GCP account:
Naming Pattern: {project-id}-{env}-{resource-name}

Examples for the `dev` environment:

myp-dev-secure-bucket
myp-dev-k8s-master-node
myp-dev-k8s-worker-node-1

Examples for the `prod` environment:

myp-prod-secure-bucket
myp-prod-k8s-master-node
myp-prod-k8s-worker-node-1

Note: As a best practice, production workloads should be managed in a separate GCP project. This approach improves production performance, enhances security, and ensures complete isolation between development and production environments.

terraform workspace new dev

Terraform Configuration

Resource configurations can be defined in a dev.tfvars variable file. Different variable files can be maintained for different environments (e.g., dev.tfvars, prod.tfvars).
For example, project-specific values such as the project ID and project name for each environment can be configured in these files.

For example:

#--------------------- Development Project Configuration ---------------------
#Development Project configuration, this project configuration is used to maintain resources for this project. eg: project_id will be used to create the GCP resources
project_id   = "myp"
project_name = "My Project"
# --------------------- GCP Project and Regsion Configuration ---------------------
gcp_region = "us-east1"
gcp_zone   = "us-east1-b"
#--------------------- Network Configuration ---------------------
nw_network_name          = "default"
nw_subnet_public_address_range = "10.0.0.0/24"
nw_subnet_private_address_range = "10.0.1.0/24"

Setup the resources in GCP

Run the following Terraform command to create the resources in the GCP .

terraform init --upgrade
terraform plan -var-file="dev.tfvars"
terraform apply -var-file="dev.tfvars"

Setup and connect IAP TCP Forwarding

To establish an SSH tunnel using IAP TCP Forwarding, we use the following command. During the initial execution, a password will be generated for the SSH connection. This password should be securely stored, as it will be required for future SSH access to the VM through the IAP tunnel.

gcloud compute ssh <YOUR_PRIVATE_VM_NAME> \
    --zone=<YOUR_VM_ZONE> \
    --tunnel-through-iap

User-Pro:infra user$ gcloud compute ssh myp-dev-k8s-master-node --zone=us-east1-b --tunnel-through-iap
WARNING: The private SSH key file for gcloud does not exist.
WARNING: The public SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
This tool needs to create the directory [/Users/user/.ssh] before being able to generate SSH keys.
Do you want to continue (Y/n)?  Y
Generating public/private rsa key pair.
Enter passphrase for "/Users/user/.ssh/google_compute_engine" (empty for no passphrase): xxxxxxxx

Service Verification

Ensure that the virtual machines (VMs) are properly created, linked, and connected. This step helps identify any issues during VM provisioning or while installing necessary services. The following Git repository contains detailed debugging notes and documentation: (Debug Notes)[https://github.com/developerhelperhub/setup-k8s-cluster-on-gcp/tree/main/debugs]

Key Verification Steps:

Confirm VM creation and successful setup
Validate storage configuration
Check Consul installation and status
Verify the Kubernetes master node
Ensure HAProxy is properly configured and running

Install Web Applications K8s cluster

We deployed nginx-web as a sample application on the Kubernetes worker nodes and accessed it through the HAProxy load balancer. As HAProxy is the chosen load balancer in our architecture, we configured an HAProxy Ingress Controller during the master node setup. This controller efficiently routes incoming traffic to the appropriate applications running within the pods.

Install the HAProxy ingress controller into master VM in the GCP console

gcloud compute ssh myp-dev-k8s-master-node --zone=us-east1-b --tunnel-through-iap
sudo su ubuntu
cd

Ensure nodes are joined with master nodes

kubectl get nodes

#Output
NAME                           STATUS   ROLES           AGE     VERSION
myp-dev-k8s-master-node        Ready    control-plane   6m50s   v1.32.4
myp-dev-k8s-worker-node-0q3j   Ready    <none>          5m26s   v1.32.4
myp-dev-k8s-worker-node-c3js   Ready    <none>          5m28s   v1.32.4

Ensure HAProxy pods running properly in all worker nodes

kubectl get pods -n haproxy-controller -o wide

#Output
NAME                                        READY   STATUS      RESTARTS   AGE     IP                NODE                           NOMINATED NODE   READINESS GATES
haproxy-kubernetes-ingress-27lwj            1/1     Running     0          5m27s   192.168.100.1     myp-dev-k8s-worker-node-c3js   <none>           <none>
haproxy-kubernetes-ingress-crdjob-1-5rwd4   0/1     Completed   0          6m25s   192.168.100.2     myp-dev-k8s-worker-node-c3js   <none>           <none>
haproxy-kubernetes-ingress-khrxt            1/1     Running     0          4m56s   192.168.209.129   myp-dev-k8s-worker-node-0q3j   <none>           <none>

Ensure HAProxy service running properly in 30080 port

#Verify the service
kubectl get svc -n haproxy-controller haproxy-kubernetes-ingress

#Output
NAME                         TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                                                   AGE
haproxy-kubernetes-ingress   NodePort   10.97.64.228   <none>        80:30080/TCP,443:30443/TCP,443:30443/UDP,1024:30002/TCP   7m35s

Ensure Kube-proxy is Running Properly If HAProxy is running on the worker node, but the port is not open, it could be due to issues with kube-proxy, which manages the network routing for services in Kubernetes. Check if kube-proxy is running on the worker nodes:

kubectl get pods -n kube-system -o wide | grep kube-proxy

#Output
kube-proxy-9hqbv                                  1/1     Running   0          7m15s   10.0.1.4          myp-dev-k8s-worker-node-c3js   <none>           <none>
kube-proxy-cnlbl                                  1/1     Running   0          8m29s   10.0.1.3          myp-dev-k8s-master-node        <none>           <none>
kube-proxy-jrh2p                                  1/1     Running   0          7m13s   10.0.1.5          myp-dev-k8s-worker-node-0q3j   <none>           <none>

Install sample nginx server in the K8s and create ingress resource of HAProxy

Install nginx-web server on K8s and expose 80 port

kubectl create deployment nginx-web --image=nginx --port=80
kubectl get deployments.apps -o wide
kubectl expose deployment nginx-web --port=80 --target-port=80 --type=ClusterIP
kubectl get  svc

#Output
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   60m
nginx-web    ClusterIP   10.110.173.152   <none>        80/TCP    16m

To Actually Spread Pods Across Multiple Nodes

kubectl scale deployment nginx-web --replicas=2
kubectl get pods -o wide -l app=nginx-web

Output:

NAME                         READY   STATUS    RESTARTS   AGE   IP                NODE                           NOMINATED NODE   READINESS GATES
nginx-web-8684b95849-j59tv   1/1     Running   0          11s   192.168.209.130   myp-dev-k8s-worker-node-0q3j   <none>           <none>
nginx-web-8684b95849-wn224   1/1     Running   0          32s   192.168.100.3     myp-dev-k8s-worker-node-c3js   <none>           <none>

Create the ingress resource to rout the request into nginx-web:80 application

#Install VI editor to create the resource file
sudo apt install vim -y
vi nginx-ingress.yaml

#Copy following configuration insuide `nginx-ingress.yaml`
#nginx-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: default
  annotations:
    ingress.class: "haproxy"
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-web
            port:
              number: 80

Apply the ingress resource

kubectl apply -f nginx-ingress.yaml

Identify the worker node Ip and test access of app through Node Port of worker node

kubectl get nodes -o wide
curl http://<worker-node1-ip>:30080
curl http://<worker-node2-ip>:30080

We will see following output of Nginx server home page

<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Identify the haporxay external Ip address from GCP console and execute following command on browser

http://<haproxy-vm-external-ip>

Destroy the resources in GCP account

Execute following commands to destroy / clean the resources which are created on GCP

terraform destroy -var-file="dev.tfvars"

Source Github - https://github.com/developerhelperhub/setup-k8s-cluster-on-gcp

K8s cluster setup in GCP with worker nodes autoscale and its discovery

Binoy — Thu, 08 May 2025 03:46:43 +0000

Overview

Objective

When the application requires high performance, availability, scalability, and security
When greater control over and maintenance of the infrastructure is needed
When the application architecture involves complex microservices
When opting for open-source solutions (optional)

When Kubernetes may not be the right choice:

If the application is small and can be supported by a cloud provider’s managed services, such as AWS ECS with Fargate (serverless technology)
If the application does not require high throughput
If the goal is to minimize IT operational costs and reduce infrastructure management responsibilities

In this setup, I considered the following points while building Kubernetes (K8s) on GCP:

Terraform: Used to easily deploy, manage, and destroy the cluster infrastructure.
HAProxy: Implemented an open-source load balancer instead of relying on GCP’s native load balancer. HAProxy provides high-performance load balancing.
Consul: Implemented VM discovery to automatically register Kubernetes worker nodes with the HAProxy load balancer.
GCP Auto Scaling and VM Health Checks: Set up an autoscaling group with TCP-based health checks to ensure the availability and reliability of virtual machines.
GCP RBAC: Leveraged GCP’s Role-Based Access Control to simplify node joining, manage Kubernetes-related files in GCP Buckets, and associate service accounts with bucket roles and virtual machines.
Minimal Permissions: As a best practice, configured minimal necessary roles for infrastructure components to enhance security.
Firewall Rules: Configured the following rules:
- HAProxy
- Ingress rule
  - Port: 80
  - Public access of HAPRoxy endpoint
- HAProxy → Kubernetes Worker Nodes (port 30080).
- Consul rule
- Ingress and Egress
  - For all agents based on VM tag (HAProxy, K8s Worker Nodes)
  - Port: 8301, 8600, 8300 for Ingress and Egress rule
  - Private and public network
- K8s Rule
- For master node and all worker nodes based on VM tag
  - Ingress
  - Port: 10250, 30000-32767, 10255, 6443
  - Private network only
  - Egress
  - Port: 443, 6443
  - Private network only
- For all worker nodes connecting to HProxy based on VM tag
  - Ingress
  - Port: 30080
  - Public network only
  - Egress
  - Port: 30080
  - Private network only
- Health Check API of GCP based on k8s nodes tag
  - Ingress
  - Port: 10250
  - GCP Health check netowrk
Public and Private Subnets: Separated application workloads and network traffic by isolating resources into public and private subnets.

Note: This is not a production-ready architecture. In production environments, the default network should not be used. Instead, you should create your own VPCs, such as separate networks for Management, Development, and Production environments (for eg: following HIPAA-compliant recommended network architecture practices).

Prerequisites

We have to setup the following tools and accounts

Create GCP account [we can create free tier account]
Install Terraform in your machine [Min Terraform v1.9.3]
Install gcloud command line in your machine

Login GCP account

gcloud auth login

(Optional) Re-authenticate for a Specific Project
To log in and set a GCP project:

gcloud config set project PROJECT_ID

Replace GCP PROJECT_ID with your actual project.

Configure the `env` variables

Following environment variables need to configure. Replace the GCP project id with {your-gcp-project-id}.

CLOUDSDK_CORE_PROJECT: This variable configure the GCP project id to use the gcloud cli command
TF_VAR_gcp_project_id: This variable to configure the GCP project id to use the terraform

export CLOUDSDK_CORE_PROJECT={your-gcp-project-id}
export set TF_VAR_gcp_project_id={your-gcp-project-id}

Terraform Structure

Following the terraform module maintain to deploy the resources in the GCP account

Create the Terraform workspace

We are following a consistent naming convention for resource creation in the GCP account:
Naming Pattern: {project-id}-{env}-{resource-name}

Examples for the `dev` environment:

myp-dev-secure-bucket
myp-dev-k8s-master-node
myp-dev-k8s-worker-node-1

Examples for the `prod` environment:

myp-prod-secure-bucket
myp-prod-k8s-master-node
myp-prod-k8s-worker-node-1

terraform workspace new dev

Terraform Configuration

For example:

#--------------------- Development Project Configuration ---------------------
#Development Project configuration, this project configuration is used to maintain resources for this project. eg: project_id will be used to create the GCP resources
project_id   = "myp"
project_name = "My Project"
# --------------------- GCP Project and Regsion Configuration ---------------------
gcp_region = "us-east1"
gcp_zone   = "us-east1-b"
#--------------------- Network Configuration ---------------------
nw_network_name          = "default"
nw_subnet_public_address_range = "10.0.0.0/24"
nw_subnet_private_address_range = "10.0.1.0/24"

Setup the resources in GCP

Run the following Terraform command to create the resources in the GCP .

terraform init --upgrade
terraform plan -var-file="dev.tfvars"
terraform apply -var-file="dev.tfvars"

It creates the following resources in GCP project account

Service Account:
VPC Network (default)→Subnet:
- myp-dev-private-subnet
- myp-dev-public-subnet
VPC Network (default)→Firewall:
- myp-dev-consol-incoming-from-consol-server
- myp-dev-consol-outgoing-to-consol-server
- myp-dev-haproxy-lb-allow-http
- myp-dev-k8s-incoming-from-haproxy-lb
- myp-dev-k8s-wroker-incoming-from-k8s-master
- myp-dev-k8s-wroker-egress-to-k8s-master
- myp-dev-k8s-outgoing-to-haproxy-lb-allow
- myp-dev-k8s-incoming-from-gcp-helath-service
Buckets:
- Bucket name: myp-dev-{uniqueid}-secure-bukcet
- Unique id can be configured in the dev.tfvars
- Folder name: /k8s/master-node
- Permission
- myp-k8s-master-sa@xxxxxx.gserviceaccount.com
- myp-k8s-worker-sa@xxxxxx.gserviceaccount.com
- myp-consul-sa@xxxxxx.gserviceaccount.com
- myp-lb-haproxy-sa@xxxxxx.gserviceaccount.com
VM Instances
- myp-dev-consul-server
- myp-dev-haproxy-lb-1
- myp-dev-k8s-master-node
- myp-dev-k8s-worker-node-mig
- myp-dev-k8s-worker-node-bxsp
Instance Templates
- myp-dev-k8s-worker-node-template
Instance Groups
- myp-dev-k8s-worker-node-mig
Health Checks
- myp-dev-k8s-worker-node-health-check
- All health of nodes should be 100% healthy

Install Web Applications K8s cluster

We deployed nginx-web as a sample application on the Kubernetes worker nodes and accessed it through the HAProxy load balancer.
As HAProxy is the chosen load balancer in our architecture, we configured an HAProxy Ingress Controller during the master node setup. This controller efficiently routes incoming traffic to the appropriate applications running within the pods.

Install the HAProxy ingress controller into master VM in the GCP console

sudo su ubuntu
cd

Ensure nodes are joined with master nodes

kubectl get nodes

#Output
NAME                           STATUS   ROLES           AGE     VERSION
myp-dev-k8s-master-node        Ready    control-plane   6m50s   v1.32.4
myp-dev-k8s-worker-node-0q3j   Ready    <none>          5m26s   v1.32.4
myp-dev-k8s-worker-node-c3js   Ready    <none>          5m28s   v1.32.4

Ensure Hproxy pods running properly in all worker nodes

kubectl get pods -n haproxy-controller -o wide

#Output
NAME                                        READY   STATUS      RESTARTS   AGE     IP                NODE                           NOMINATED NODE   READINESS GATES
haproxy-kubernetes-ingress-27lwj            1/1     Running     0          5m27s   192.168.100.1     myp-dev-k8s-worker-node-c3js   <none>           <none>
haproxy-kubernetes-ingress-crdjob-1-5rwd4   0/1     Completed   0          6m25s   192.168.100.2     myp-dev-k8s-worker-node-c3js   <none>           <none>
haproxy-kubernetes-ingress-khrxt            1/1     Running     0          4m56s   192.168.209.129   myp-dev-k8s-worker-node-0q3j   <none>           <none>

Ensure Hproxy service running properly in 30080 port

#Verify the service
kubectl get svc -n haproxy-controller haproxy-kubernetes-ingress

#Output
NAME                         TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                                                   AGE
haproxy-kubernetes-ingress   NodePort   10.97.64.228   <none>        80:30080/TCP,443:30443/TCP,443:30443/UDP,1024:30002/TCP   7m35s

Ensure Kube-proxy is Running Properly If HAProxy is running on the worker node, but the port is not open, it could be due to issues with kube-proxy, which manages the network routing for services in Kubernetes. Check if kube-proxy is running on the worker nodes:

kubectl get pods -n kube-system -o wide | grep kube-proxy

#Output
kube-proxy-9hqbv                                  1/1     Running   0          7m15s   10.0.1.4          myp-dev-k8s-worker-node-c3js   <none>           <none>
kube-proxy-cnlbl                                  1/1     Running   0          8m29s   10.0.1.3          myp-dev-k8s-master-node        <none>           <none>
kube-proxy-jrh2p                                  1/1     Running   0          7m13s   10.0.1.5          myp-dev-k8s-worker-node-0q3j   <none>           <none>

Install sample nginx server in the K8s and create ingress resource of HAProxy

Install nginx-web server on K8s and expose 80 port

kubectl create deployment nginx-web --image=nginx --port=80
kubectl get deployments.apps -o wide
kubectl expose deployment nginx-web --port=80 --target-port=80 --type=ClusterIP
kubectl get  svc

#Output
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   9m3s
nginx-web    ClusterIP   10.111.110.192   <none>        80/TCP    1s

To Actually Spread Pods Across Multiple Nodes

kubectl scale deployment nginx-web --replicas=2

kubectl get pods -o wide -l app=nginx-web

Output:

NAME                         READY   STATUS    RESTARTS   AGE   IP                NODE                           NOMINATED NODE   READINESS GATES
nginx-web-8684b95849-j59tv   1/1     Running   0          11s   192.168.209.130   myp-dev-k8s-worker-node-0q3j   <none>           <none>
nginx-web-8684b95849-wn224   1/1     Running   0          32s   192.168.100.3     myp-dev-k8s-worker-node-c3js   <none>           <none>

Create the ingress resource to rout the request into nginx-web:80 application

#Install VI editor to create the resource file
sudo apt install vim -y
vi nginx-ingress.yaml

#Copy following configuration insuide `nginx-ingress.yaml`
#nginx-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: default
  annotations:
    ingress.class: "haproxy"
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-web
            port:
              number: 80

Apply the ingress resource

kubectl apply -f nginx-ingress.yaml

Identify the worker node Ip and test access of app through Node Port of worker node

kubectl get nodes -o wide
curl http://<worker-node1-ip>:30080
curl http://<worker-node2-ip>:30080

We will see following output of Nginx server home page

<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Identify the haporxay external Ip address from GCP console and execute following command on browser

http://<haproxy-vm-external-ip>

Destroy the resources in GCP account

Execute following commands to destroy / clean the resources which are created on GCP

terraform destroy -var-file="dev.tfvars"

Source Code : (setup-k8s-cluster-on-gcp)[https://github.com/developerhelperhub/setup-k8s-cluster-on-gcp]

Debug the services

Following git repo contains the all debug documents to debug the issues

(Debug Notes)[https://github.com/developerhelperhub/setup-k8s-cluster-on-gcp/debugs]
Verify VMS installation
Verify storage
Verify the consul
Verify the master node
Verify the HAProxy

Setup K8s cluster on GCP VMs

Binoy — Mon, 28 Apr 2025 17:07:13 +0000

Overview

Objective

When the application requires high performance, availability, scalability, and security
When greater control over and maintenance of the infrastructure is needed
When the application architecture involves complex microservices
When opting for open-source solutions (optional)

When Kubernetes may not be the right choice:

If the application is small and can be supported by a cloud provider’s managed services, such as AWS ECS with Fargate (serverless technology)
If the application does not require high throughput
If the goal is to minimize IT operational costs and reduce infrastructure management responsibilities

In this setup, I considered the following points while building Kubernetes (K8s) on GCP:

Terraform: Used to easily deploy, manage, and destroy the cluster infrastructure.
HAProxy: Implemented an open-source load balancer instead of relying on GCP’s native load balancer. HAProxy provides high-performance load balancing.
GCP RBAC: Leveraged GCP’s Role-Based Access Control to simplify node joining, manage Kubernetes-related files in GCP Buckets, and associate service accounts with bucket roles and virtual machines.
Minimal Permissions: As a best practice, configured minimal necessary roles for infrastructure components to enhance security.
Firewall Rules: Configured the following rules: Public → HAProxy (port 80) and HAProxy → Kubernetes Worker Nodes (port 30080).
Public and Private Subnets: Separated application workloads and network traffic by isolating resources into public and private subnets.

Prerequisites

We have to setup the following tools and accounts

Create GCP account [we can create free tier account]
Install Terraform in your machine [Min Terraform v1.9.3]
Install gcloud command line in your machine

Login GCP account

gcloud auth login

(Optional) Re-authenticate for a Specific Project
To log in and set a GCP project:

gcloud config set project PROJECT_ID

Replace GCP PROJECT_ID with your actual project.

Configure the `env` variables

Following environment variables need to configure. Replace the GCP project id with {your-gcp-project-id}.

CLOUDSDK_CORE_PROJECT: This variable configure the GCP project id to use the gcloud cli command
TF_VAR_gcp_project_id: This variable to configure the GCP project id to use the terraform

export CLOUDSDK_CORE_PROJECT={your-gcp-project-id}
export set TF_VAR_gcp_project_id={your-gcp-project-id}

Terraform Structure

Following the terraform module maintain to deploy the resources in the GCP account

Create the Terraform workspace

We are following a consistent naming convention for resource creation in the GCP account:
Naming Pattern: {project-id}-{env}-{resource-name}

Examples for the `dev` environment:

myp-dev-secure-bucket
myp-dev-k8s-master-node
myp-dev-k8s-worker-node-1

Examples for the `prod` environment:

myp-prod-secure-bucket
myp-prod-k8s-master-node
myp-prod-k8s-worker-node-1

terraform workspace new dev

Terraform Configuration

For example:

#--------------------- Development Project Configuration ---------------------
#Development Project configuration, this project configuration is used to maintain resources for this project. eg: project_id will be used to create the GCP resources
project_id   = "myp"
project_name = "My Project"
# --------------------- GCP Project and Regsion Configuration ---------------------
gcp_region = "us-east1"
gcp_zone   = "us-east1-b"
#--------------------- Network Configuration ---------------------
nw_network_name          = "default"
nw_subnet_public_address_range = "10.0.0.0/24"
nw_subnet_private_address_range = "10.0.1.0/24"

Setup the resources in GCP

Run the following Terraform command to create the resources in the GCP .

terraform init --upgrade
terraform plan -var-file="dev.tfvars"
terraform apply -var-file="dev.tfvars"

It creates the following resources in GCP project account

Service Account:
- k8s-master-node-sa@xxxxxx.gserviceaccount.com
- k8s-worker-node-sa@xxxxxx.gserviceaccount.com
VPC Network (default)→Subnet:
- myp-dev-private-subnet
- myp-dev-public-subnet
VPC Network (default)→Firewall:
- myp-dev-haproxy-lb-allow-http
- myp-dev-k8s-incoming-from-haproxy-lb
- myp-dev-k8s-wroker-incoming-from-k8s-master
- myp-dev-k8s-wroker-egress-to-k8s-master
- myp-dev-k8s-outgoing-to-haproxy-lb-allow
Buckets:
- Bucket name: myp-dev-{uniqueid}-secure-bukcet
  - Unique id can be configured in the dev.tfvars
- Folder name: /k8s/master-node
- Permission
  - k8s-master-node-sa@xxxxxx.gserviceaccount.com
  - k8s-worker-node-sa@xxxxxx.gserviceaccount.com
VM Instances
- myp-dev-haproxy-lb-1
- myp-dev-k8s-master-node
- myp-dev-k8s-worker-node-1

Verify the services are installed in the K8s VM and HAProxy VM

We can verify whether the services have been installed successfully. The following steps can help monitor and validate the status of the services.

SSH into Master VM in the GCP console

Verify all initial script execute properly while start the VM

#For Google Cloud VM Instances:
#When using the metadata_startup_script in a GCP VM, the startup script output is logged to:
tail -500f /var/log/syslog

cat /var/log/syslog | grep "Intalled the tools!"
cat /var/log/syslog | grep "Container configure default configuration!"
cat /var/log/syslog | grep "Enabiling IP forwarding"
cat /var/log/syslog | grep "Configured the paramters of VM for K8s!"
cat /var/log/syslog | grep "Kubernetes has initialized successfully!"
cat /var/log/syslog | grep "Uploaded token into bucket!"

SSH into Worker VM in the GCP console

Verify all initial script execute properly while start the VM

tail -500f /var/log/syslog
cat /var/log/syslog | grep "Joined the master node successfully!"

SSH into HAProxy VM in the GCP console

Verify all initial script execute properly while start the VM

    tail -500f /var/log/syslog
    cat /var/log/syslog | grep "Restart HAPorxy!"

    ##Output:
    startup-script: Restart HAPorxy!

Verify HAProxy configuration and checking internal IP configure of k8s backend service

cat /etc/haproxy/haproxy.cfg

#Following information will be added in the configuration, Output:
frontend http_front
        bind *:80
        mode http
        default_backend haproxy_ingress_backend
backend haproxy_ingress_backend
    mode http
    balance roundrobin
    server k8s-node1 <worker-node-internip>:30080 check maxconn 32

Verify the tcp connection 30080 port

sudo apt install telnet -y
telnet <worker-node-internip> 30080

Verify the Bucket

Check the bucket storage to verify whether the file /k8s/master-node/join_node.sh has been uploaded.
This shell script is used to join worker nodes to the Kubernetes cluster. It will be downloaded and executed on each worker node during the setup process.

Verify the nodes join

We can verify the worker nodes are joined with master node. Following step helps monitor the services
SSH into Master VM in the GCP console

Verify the nodes are joined in the K8s cluster. Execute following command

#Login into ubuntu user
sudo su ubuntu
##Check the nodes
kubectl get nodes

Following Output:

NAME                            STATUS   ROLES           AGE   VERSION
myp-dev-k8s-master-node     Ready    control-plane   26m   v1.32.3
myp-dev-k8s-worker-node-1   Ready    <none>          25m   v1.32.3

Verify the pods are installed eg: network plugin (Calico in this case)

##Check the pods
kubectl get pods --all-namespaces

##Following Output:
NAMESPACE     NAME                                                  READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-7498b9bb4c-44pf8              1/1     Running   0          29m
kube-system   calico-node-b4bnh                                     1/1     Running   0          28m
kube-system   calico-node-bhpv8                                     1/1     Running   0          29m
kube-system   coredns-668d6bf9bc-8tdsb                              1/1     Running   0          29m
kube-system   coredns-668d6bf9bc-j6q4p                              1/1     Running   0          29m
kube-system   etcd-myp-default-k8s-master-node                      1/1     Running   0          29m
kube-system   kube-apiserver-myp-default-k8s-master-node            1/1     Running   0          29m
kube-system   kube-controller-manager-myp-default-k8s-master-node   1/1     Running   0          29m
kube-system   kube-proxy-4qgkc                                      1/1     Running   0          28m
kube-system   kube-proxy-zvlsb                                      1/1     Running   0          29m
kube-system   kube-scheduler-myp-default-k8s-master-node            1/1     Running   0          29m

Install Web Applications K8s cluster

We are deploying nginx-web as a sample application on the worker nodes and connecting to it through the HAProxy load balancer.
Since we are using HAProxy as the load balancer in our architecture, we need to configure an HAProxy Ingress Controller to properly route incoming requests to the applications running inside the pods.

Install the HAProxy ingress controller into master VM in the GCP console

Install helm

#Login to ubuntu user
sudo su ubuntu

#Install helm
curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
sudo apt-get install apt-transport-https --yes
echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
# Install
sudo apt update
sudo apt install helm

#Check the version
helm version

#Output:
version.BuildInfo{Version:"v3.17.2", GitCommit:"cc0bbbd6d6276b83880042c1ecb34087e84d41eb", GitTreeState:"clean", GoVersion:"go1.23.7"}

Add HAProxy repo

helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm repo update

Install the HAProxy ingress controller and expose the http and https Node Port to connect the HAProxy Load Balancer

helm install haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
    --create-namespace \
    --namespace haproxy-controller \
    --set controller.service.type=NodePort \
    --set controller.service.nodePorts.http=30080 \
    --set controller.service.nodePorts.https=30443 \
    --set controller.service.nodePorts.stat=30002 \
    --set controller.service.nodePorts.prometheus=30003

#Verify the service
kubectl get svc -n haproxy-controller haproxy-kubernetes-ingress
#Output
haproxy-kubernetes-ingress   NodePort   10.109.135.187   <none>        80:30080/TCP,443:30443/TCP,443:30443/UDP,1024:30002/TCP   103s

Install sample nginx server in the K8s and create ingress resource of HAProxy

Install nginx-web server on K8s and expose 80 port

kubectl create deployment nginx-web --image=nginx --port=80
kubectl get deployments.apps -o wide
kubectl expose deployment nginx-web --port=80 --target-port=80 --type=ClusterIP
kubectl get  svc

#Output
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   60m
nginx-web    ClusterIP   10.110.173.152   <none>        80/TCP    16m

Create the ingress resource to rout the request into nginx-web:80 application

#Install VI editor to create the resource file
sudo apt install vim -y
vi nginx-ingress.yaml

#Copy following configuration insuide `nginx-ingress.yaml`
#nginx-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: nginx-ingress
    namespace: default
    annotations:
    ingress.class: "haproxy"
spec:
    rules:
    - http:
        paths:
        - path: /
        pathType: Prefix
        backend:
            service:
            name: nginx-web
            port:
                number: 80

Apply the ingress resource

kubectl apply -f nginx-ingress.yaml

Identify the worker node Ip and test access of app through Node Port of worker node

kubectl get nodes -o wide
curl http://<worker-node-ip>:30080

We will see following output of Nginx server home page

<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Identify the haporxay external Ip address from GCP console and execute following command on browser

http://<haproxy-vm-external-ip>

Destroy the resources in GCP account

Execute following commands to destroy / clean the resources which are created on GCP

terraform destroy -var-file="dev.tfvars"

Git Repo : (setup-k8s-cluster-on-gcp)[https://github.com/developerhelperhub/setup-k8s-cluster-on-gcp]

Spring Boot 3.4.1 Keycloak introspect and method level security

Binoy — Tue, 21 Jan 2025 18:26:42 +0000

Spring Boot 3.4.1 Keycloak introspect and method level security

This section outlines the process of implementing the Keycloak introspection flow in Spring Boot version 3.4.1 and demonstrates how to enable method-level security based on roles configured in Keycloak.

Integration

Spring Boot 3.4.1
Java 22
Keycloak Integration and RBAC implementation
Spring Oauth2 Resource Server and Method Level Security
MongoDB NoSQL DB and Auditing

Installing Dependencies Services

The following services are required dependencies before starting the Spring Boot application.

Keycloak Installation in Docker

Execute the following commands to start the Keycloak service in a Docker container. The admin username and password are set to "admin" and "admin," respectively.

docker run -d --name klight-authentication-server -p 8080:8080 \
  -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin \
  quay.io/keycloak/keycloak:26.0 start-dev

MongoDB installation in Docker

Run the following commands to start the MongoDB service in a Docker container. The root username and password are both set to klight-api-gateway. After installing MongoDB, you need to create a database named "klight-api-gateway" to enable connection with the Spring Boot application, as this database is referenced in the application-dev.yaml file.

docker run --name klight-api-gateway-mongo -p 27017:27017 -e MONGO_INITDB_ROOT_USERNAME=klight-api-gateway -e MONGO_INITDB_ROOT_PASSWORD=klight-api-gateway -d amd64/mongo:8.0.3

Spring Security Dependencies

The following dependencies must be configured to enable the introspection flow. The admin service is responsible for validating tokens with the Keycloak service. A token must be generated from Keycloak and included in the Authorization header of the API request.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
</dependency>

Keycloak configuration in YAML file

We have to configure the following attributes in the YAML file. This configuration uses in the SecurityConig.java class to configure the introspection flow.

keycloak:
  introspection-uri: http://127.0.0.1:8080/realms/klight-api-gateway/protocol/openid-connect/token/introspect
  client-id: klight-api-gateway-admin-connect
  client-secret: HDYMxWu2G2VWn5u59MJKwECIBd6VTO7E

Security Configuration

The following attributes must be configured in the YAML file. This configuration is utilized in the SecurityConfig.java class to set up the introspection flow.

@Configuration
@EnableWebSecurity
@EnableMethodSecurity(prePostEnabled = true, securedEnabled = true, jsr250Enabled = true)
public class SecurityConfig {

    @Value("${keycloak.introspection-uri}")
    private String introspectionUrl;

    @Value("${keycloak.client-id}")
    private String clientId;

    @Value("${keycloak.client-secret}")
    private String clientSecret;

    @Autowired
    private ExceptionAccessDeniedHandler accessDeniedHandler;

    @Autowired
    private ExcetionAuthenticationEntryPoint entryPoint;

    @Bean
    public SecurityFilterChain securityFilterChain(HttpSecurity httpSecurity, OpaqueTokenAuthenticationConverter converter) throws Exception {


        httpSecurity
                .csrf(AbstractHttpConfigurer::disable)
                .authorizeHttpRequests(auth->
                        auth.anyRequest().authenticated())
                .oauth2ResourceServer(auth->
                        auth.opaqueToken(op->
                                op.introspectionUri(this.introspectionUrl)
                                        .introspectionClientCredentials(
                                                this.clientId,
                                                this.clientSecret)
                                        .authenticationConverter(converter)
                                )

                                .authenticationEntryPoint(entryPoint.handler())
                                .accessDeniedHandler(accessDeniedHandler.handler())
                                )
                .exceptionHandling(ex ->
                        ex.authenticationEntryPoint(entryPoint.handler())
                                .accessDeniedHandler(accessDeniedHandler.handler()))
        ;

        return httpSecurity.build();
    }

}

The key configurations in the above setup are as follows:

Enable method-level security in the Spring Boot application using @EnableMethodSecurity(prePostEnabled = true, securedEnabled = true, jsr250Enabled = true).
Implement the OpaqueTokenAuthenticationConverter converter to process realm_access and resource_access from the token and create a list of GrantedAuthority based on the roles configured in the Keycloak service.
Disable csrf protection for the REST API service.
Authenticate all incoming requests using authorizeHttpRequests.
Configure the introspection flow with oauth2ResourceServer.
Handle authentication exceptions using authenticationEntryPoint.
Set up an access denied handler with accessDeniedHandler.

Method Level Configuration

We are securing the functionality of the ApiService.java methods using the following configuration. The @PreAuthorize("hasAuthority('ROLE_ADMIN')") annotation should be added to service class methods to validate role permissions. We should mention ROLE_ADMIN role name in the annotation which we have created in the Keycloak service.

@PreAuthorize("hasAuthority('ROLE_ADMIN')")
public void create(ApiServiceRequest request) {
    log.info("Service creating");

    ApiServiceCreate service = createFactory.getObject();

    map(service, request);

    service.create();

}

Keycloak Configuration

Following configurations need to setup in the Keycloak service. We have to login into Keycloak http://localhost:8080 in the browser with admin user and password.

Create the Realm `klight-api-gateway`

Keycloak supports multi-tenancy, enabling us to create application-specific configurations through realm setup. For the Klight API Gateway, we maintain a dedicated realm. This realm, we are using to maintain the clients, user, group and roles.

Click on the "Keycloak master" realm in the left corner.
Click the "Create realm" button.
Enter the realm name as "klight-api-gateway."
Enable the realm by toggling the "Enabled" option.
Click the "Create" button.

Note: Ensure that the "klight-api-gateway" realm is selected before configuring anything, as the default realm is set to "master."

Create the Client `klight-api-gateway-admin-connect`

Following setup to create new client. This client is support OpenID connection where admin service can connect the Keycloak server through this client.

Click "Clients" in the left sidebar menu.
Click the "Create Client" button.
Configure the following "General Settings":
- Set the client type to "OpenID Connect"
- Enter the Client ID: klight-api-gateway-admin-connect.
- Enter the Client Name: Klight API Gateway Admin Connect
- Click the "Next" button.
Configure the following "Capability Configuration":
- Enable "Client Authentication"
- Select only the "Direct access grants" flow. This flow allows generating tokens using the "username" and "password" method, which is ideal for trusted clients such as mobile apps or backend services.
- Click the "Next" button.
Configure the "Login Settings":
- No additional configuration is needed here for now.
Click the "Save" button.
After the client is created, select the "Credentials" tab.
- Set the "Client Authenticator" to "Client ID and Secret"
- Click "Generate" to create a new "Client Secret."
Select the "Roles" tab
- Click the "Create Role" button
- Enter the role name ROLE_ADMIN, this is the role name is configuring in the method level
- Click "Save" button

Note: This client id and client secret are configured in the application-dev.yaml file

We have to create the ROLE_ADMIN in the client Roles tab

Create the User

In this flow, we need to create a new user to authenticate the server.

Click "Users" in the left sidebar menu.
Click the "Create new user" button.
Enable the "Email Verified" option.
Enter the following "General" information:
- Username: "api-admin"
- Email: "my-user@test.com"
- First Name: "my"
- Last Name: "user"
Click the "Create" button.
Select the user `api-admin' from the user list to configure password and role
Click the "Credentials" tab.
- Click the "Set password" button.
- Enter "test" for both the "Password" and "Password Confirmation" fields.
- Set "Temporary" to Off.
- Click the "Save" button.
- Click the "Save password" button.
Click the "Role Mapping" tab
- Click the "Assign Role" button
- Search the ROLE_ADMIN in the search box
- Select the check box of ROLE_ADMIN
- Click on the Assign button

Testing Endpoint

Once configuration done, we can test the API to test the authentication and authorization

Get the access token

Run the following command to get the access token from Keycloak
shell curl --location 'http://127.0.0.1:8080/realms/klight-api-gateway/protocol/openid-connect/token' \ --header 'Content-Type: application/x-www-form-urlencoded' \ --data-urlencode 'grant_type=password' \ --data-urlencode 'client_id=klight-api-gateway-admin-connect' \ --data-urlencode 'client_secret=HDYMxWu2G2VWn5u59MJKwECIBd6VTO7E' \ --data-urlencode 'scope=openid' \ --data-urlencode 'username=api-admin' \ --data-urlencode 'password=test'

Test endpoint of admin service

Run the following command to test the endpoint to create the service
shell curl --location 'http://localhost:8082/admin/services' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiI...' \ --header 'Cookie: JSESSIONID=7B6F7EB454EF0392D6EB21658A922900' \ --data '{ "name": "Test Service", "host": "localhost", "path": "test-service", "protocol": "https" }'

Other Coding Pattern

This source contains following coding pattern implemented as well

Single Responsible of business logic for service ApiService.java, this help to maintain the readability, maintainability, reuse this business this class in different functionality, for example
- ApiServiceCreate.java for creating the service
- ApiServiceUpdate.java for creating the service
@EnableMongoAuditing enabled auditing MongoDbConfig.java
- CreatedDate, LastModifiedDate, CreatedBy, LastModifiedBy
Added the logs with help Slf4j for debugging purpose
Maintain the unique error code for exception, implemented global exception handling ExceptionControllerAdvice.java

Source Code in Github 'klight-api-gateway-admin'

I am developing a project that provides APIs for creating services for the microservice-based Klight API gateway, which I built using the OpenResty web application. This API gateway delivers high performance compared to a Spring Boot API gateway because the OpenResty framework is built on top of the NGINX server and uses Lua, a language commonly used in game programming.

You might wonder why Kong API Gateway wasn't used. The reason is that the open-source version of Kong does not support OpenID Connect and pay to integrate with OpenID Connect. When building high-performance applications in a microservice architecture, it is crucial to choose an API gateway optimized for performance.

References

[Boost]

Binoy — Wed, 01 Jan 2025 08:28:37 +0000

High Availability and Scalability deployment Microservices on Kubernetes cluster

Binoy ・ Sep 20 '24

#kubernetes #terraform #architecture

[Boost]

Binoy — Sun, 29 Dec 2024 05:38:27 +0000

High Availability and Scalability deployment Microservices on Kubernetes cluster

Binoy ・ Sep 20 '24

#kubernetes #terraform #architecture

[Boost]

Binoy — Sat, 28 Dec 2024 12:06:32 +0000

Setup efficient CICD Pipeline Jenkins to build binary and push docker image - Kubernetes cluster

Binoy ・ Sep 5 '24

#springboot #terraform #kubernetes #jenkins

Terraform - Kong API Gateway deployment in Kubernetes

Binoy — Sat, 28 Sep 2024 17:47:48 +0000

This section helps to basic understand how can we install the Kong API Gateway in the Kubernetes Cluster with help of Terraform

Objective

In a microservice architecture, one of the key components is the API Gateway, which acts as a reverse proxy. Instead of clients communicating directly with multiple services, all requests are sent to a single server (the API Gateway), which routes them to the appropriate microservices based on routing policies configured in the reverse proxy server. One might wonder if a reverse proxy server like Nginx could replace an API Gateway. However, an API Gateway offers more advanced capabilities compared to a standard Nginx server. Below are some key differences:

Plugins for authentication (JWT, OAuth2, key authentication).
Rate limiting and request throttling.
Logging and monitoring (integration with Prometheus, Grafana).
Service discovery and health checks.
Supports running in various environments, including Kubernetes and Docker.
Load balancing with round-robin, least-connections, etc.
Integration with third-party services like Datadog, Zipkin, and Cassandra.
Manage the version API
Manage the deployment strategy like “green/blue”, “canary” deployments
Comes with built-in API management capabilities and is easier to set up for managing APIs. You can use its admin API to configure routes, services, and plugins dynamically.
Built with scaling in mind, it easily integrates with tools like PostgreSQL and Cassandra for storing API configuration and scaling horizontally across many nodes. ## Setup local environment to build DevOps resources

I use docker containers to set up work environments for multiple applications(Setup Environment). This approach ensures fully isolated and maintainable environments for application development, allowing us to easily start and terminate these environments. Below is the Docker command to create the environment.

docker run -it --name test-microservices-module-envornment-box -v ~/.kube/config:/work/.kube/config -e KUBECONFIG=/work/.kube/config -v ${HOME}:/root/ -v ${PWD}:/work -w /work --net host developerhelperhub/kub-terr-work-env-box

The container contains Docker, Kubectl, Helm, Terraform, Kind, Git

Setup Kong on Kubernetes Cluster

I have created the terraform mdoules, which are available in the GitHub repository. You can download and set up Kong on a Kubernetes cluster, which runs locally in a Docker container.

Clone the repository onto your local Linux machine to get started.

git clone https://github.com/developerhelperhub/kuberentes-help.git
cd kuberentes-help/terraform/sections/00008/terraform

main.tf terraform script

module "microservices" {
    source = "git::https://github.com/developerhelperhub/microservices-terraform-module.git//microservices?ref=v1.2.0"
    kind_cluster_name = var.kind_cluster_name
    kind_http_port    = 80
    kind_https_port   = 443
    kubernetes_namespace = "microservices"
    kong_enable            = true
    kong_admin_domain_name = var.kong_admin_domain_name
    kong_proxy_domain_name = var.kong_proxy_domain_name
    kong_db_user           = "mykong"
    kong_db_name           = "mykongdb"
    kong_db_password       = "MyPassword2222@"
    kong_db_admin_password = "MyPassword2222@"
    kong_persistence_size  = "5Gi"
}

variables.tf terraform script

#This is variable arguments while running the terraform scripts
variable "kind_cluster_name" {
    type = string
    description = "Kind cluster name"
}
variable "kong_admin_domain_name" {
    type = string
    description = "Kong admin api domain name"
    default = "admin.kong.myapp.com"
}
variable "kong_proxy_domain_name" {
    type = string
    description = "Kong proxy domain name"
    default = "api.gateway.mes.app.com"
}

These Terraform scripts install and configure resources in the cluster:

Create the Kubernetes cluster in docker container locally, the cluster name will be “microservices-development-cluster-control-plane”
Install the ingress controller and exposes ports 80 and 443 to allow access to services from outside the cluster.
Create a namespace called "microservices"
Install Kong in the "microservices" namespace using a Helm chart.
Kong postgres username and password default “mykong” and “MyPassword2222@”
Setup the Kong admin API to create the service, route etc.
Setup the Kong Ingress resource to connect the Ingress controller with the Kong Admin API and Kong Proxy services.
Disabled monitoring Keycloak, Grafana and Prometheus

Cluster create terraform script under kind folder

terraform init
terraform plan  -var="kind_cluster_name=microservices-development-cluster"
terraform apply  -var="kind_cluster_name=microservices-development-cluster"

Following command verify the services

kubectl cluster-info #verify cluster info
kubectl get nodes -o wide #verify node

kubectl get namespace #verify the microservices namespace
kubectl get -n microservices pod #verify server is running
kubectl get -n microservices svc #verify service

As per my experience, this Kong services take time to start and ready to use it. Make sure all services should be ready status before open the service. Eg:

kubectl -n microservices get pod --watch
NAME                              READY   STATUS      RESTARTS   AGE
kong-kong-6bdd9944d-cp79n         2/2     Running     0          9m39s
kong-kong-init-migrations-fptth   0/1     Completed   0          9m39s
kong-postgresql-0                 1/1     Running     0          9m51s

Following command use to login into postgres, the password will be prompt executed the command.

kubectl -n microservices exec -it pod/kong-postgresql-0 -- psql -U mykong -d mykongdb

Add our domain to the bottom of the /etc/hosts file on your local machine. This configuration should not be inside our working Linux box “test-microservices-module-envornment-box”; it should be applied to your personal machine's /etc/hosts file. (you will need administrator access):

127.0.0.1       api.gateway.mes.app.com
127.0.0.1       admin.kong.myapp.com

Verify the admin api service, this api response contains the kong API gateway details which is configured in the cluster.

curl --location 'http://admin.kong.myapp.com/'

Run the microservices in Kubernetes cluster

I have created two spring boot applications which are available in the docker hub repository for testing the routing the request to multiple microservices in the cluster. The microservice services will be running 8080 port

“developerhelperhub/mes-item-service:1.0.0.1-SNAPSHOT”
“developerhelperhub/mes-order-service:1.0.0.1-SNAPSHOT”

Deploy the item service in the cluster

kubectl -n microservices -f microservices/item-service/kube-deployment.yaml apply
kubectl -n microservices -f microservices/item-service/kube-service.yaml apply

Deploy the order service in the cluster

kubectl -n microservices -f microservices/order-service/kube-deployment.yaml apply
kubectl -n microservices -f microservices/order-service/kube-service.yaml apply

Following command verify the pods of microservices

NAME                                 READY   STATUS      RESTARTS   AGE
mes-item-service-544f54f9fd-2jkbs    1/1     Running     0          2m46s
mes-item-service-544f54f9fd-bm85t    1/1     Running     0          2m46s
mes-item-service-544f54f9fd-cddrh    1/1     Running     0          2m46s
mes-item-service-544f54f9fd-zdnjd    1/1     Running     0          2m46s
mes-order-service-5f7cf68dd9-79z48   1/1     Running     0          2m39s
mes-order-service-5f7cf68dd9-qx2dv   1/1     Running     0          2m39s
mes-order-service-5f7cf68dd9-w28nt   1/1     Running     0          2m39s

Following command verify the services are running in the correct port

kubectl -n microservices get svc
NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
kong-kong-admin                ClusterIP   10.96.215.110   <none>        8001/TCP                        24m
kong-kong-manager              NodePort    10.96.185.130   <none>        8002:32038/TCP,8445:31091/TCP   24m
kong-kong-proxy                ClusterIP   10.96.125.115   <none>        80/TCP,443/TCP                  24m
kong-kong-validation-webhook   ClusterIP   10.96.180.81    <none>        443/TCP                         24m
kong-postgresql                ClusterIP   10.96.129.217   <none>        5432/TCP                        24m
kong-postgresql-hl             ClusterIP   None            <none>        5432/TCP                        24m
mes-item-service               ClusterIP   10.96.241.39    <none>        8080/TCP                        3m32s
mes-order-service              ClusterIP   10.96.252.164   <none>        8080/TCP                        3m25s

Create Service and Route path in Kong API

Kong Admin API provides endpoints to create the services and its route path of the services.

Create the service of item-service

curl --location 'http://admin.kong.myapp.com/services/' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'name=mes-item-service' \
--data-urlencode 'url=http://mes-item-service.microservices.svc.cluster.local:8080'

Create the route path of item-service. We are routing the all request into “item-service” if the path contains “/item-service”

curl --location 'http://admin.kong.myapp.com/routes/' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'service.id=<service_id>' \
--data-urlencode 'paths%5B%5D=/item-service'

Note : Replace the <service_id> with actual service id of item-service.

Create the service of order-service

curl --location 'http://admin.kong.myapp.com/services/' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'name=mes-order-service' \
--data-urlencode 'url=http://mes-order-service.microservices.svc.cluster.local:8080'

Create the route path of item-service. We are routing the all request into “order-service” if the path contains “/item-service”

curl --location 'http://admin.kong.myapp.com/routes/' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'service.id=<service_id>' \
--data-urlencode 'paths%5B%5D=/order-service'

Note : Replace the <service_id> with actual service id of item-service.

Testing the endpoints of Item service

#get the list of items
curl --location 'http://api.gateway.mes.app.com/item-service/items'
#get the service info
curl --location 'http://api.gateway.mes.app.com/item-service/actuator/info'
#get the helth info
curl --location 'http://api.gateway.mes.app.com/item-service/actuator/health'

Testing the endpoints of Item service

#get the list of items
curl --location 'http://api.gateway.mes.app.com/order-service/items'
#get the service info
curl --location 'http://api.gateway.mes.app.com/order-service/actuator/info'
#get the helth info
curl --location 'http://api.gateway.mes.app.com/order-service/actuator/health'

Note : Postman collections and environment files are available in the repo

Reference

Terraform - Keycloak install on Kubernetes cluster

Binoy — Wed, 25 Sep 2024 18:13:56 +0000

This section helps to basic understand how can we install the Keycloak in the Kubernetes Cluster with help of Terraform

Setup local environment to build DevOps resources

docker run -it --name test-microservices-module-envornment-box -v ~/.kube/config:/work/.kube/config -e KUBECONFIG=/work/.kube/config -v ${HOME}:/root/ -v ${PWD}:/work -w /work --net host developerhelperhub/kub-terr-work-env-box

The container contains Docker, Kubectl, Helm, Terraform, Kind, Git

Setup Keycloak on Kubernetes Cluster

I have created the terraform mdoules, which are available in the GitHub repository. You can download and set up Keycloak on a Kubernetes cluster, which runs locally in a Docker container.

Clone the repository onto your local Linux machine to get started.

git clone https://github.com/developerhelperhub/kuberentes-help.git
cd kuberentes-help/terraform/sections/00007/terraform

main.tf terraform script

module "microservices" {
    source = "git::https://github.com/developerhelperhub/microservices-terraform-module.git//microservices?ref=v1.1.0"
    kind_cluster_name = var.kind_cluster_name
    kind_http_port    = 80
    kind_https_port   = 443
    kubernetes_namespace = "microservices"
    keycloak_enable      = true
    keycloak_domain_name = var.keycloak_domain_name
    keycloak_admin_user     = "admin"
    keycloak_admin_password = "MyPassword2222@"
    keycloak_resources_requests_cpu    = "500m"
    keycloak_resources_requests_memory = "1024Mi"
    keycloak_resources_limit_cpu       = "500m"
    keycloak_resources_limit_memory    = "1024Mi"
    keycloak_db_password               = "MyPassword2222@"
    keycloak_db_admin_password         = "MyPassword2222@"
    keycloak_autoscaling_min_replicas  = 1
    keycloak_autoscaling_max_replicas  = 1
    keycloak_persistence_size          = "8Gi"
}

variables.tf terraform script

#This is variable arguments while running the terraform scripts
variable "kind_cluster_name" {
    type = string
    description = "Kind cluster name"
}
variable "keycloak_domain_name" {
    type = string
    description = "Keycloak domain name"
    default = "keycloak.myapp.com"
}

These Terraform scripts install and configure resources in the cluster:

Create the Kubernetes cluster in docker container locally, the cluster name will be “microservices-development-cluster-control-plane”
Install the ingress controller and exposes ports 80 and 443 to allow access to services from outside the cluster.
Create a namespace called "microservices"
Install Keycloak in the "microservices" namespace using a Helm chart.
Keycloak username and password default “admin” and “MyPassword2222@”
Set up the Keycloak Ingress resource to connect the Ingress controller with the Keycloak service.
Configure the Keycloak container to run on port 80 and expose it to port 80 through the Ingress controller.
Disabled monitoring Grafana and Prometheus

Cluster create terraform script under kind folder

terraform init
terraform plan  -var="kind_cluster_name=microservices-development-cluster"
terraform apply  -var="kind_cluster_name=microservices-development-cluster"

Following command verify the Jenkins Service

kubectl cluster-info #verify cluster info
kubectl get nodes -o wide #verify node

kubectl get namespace #verify the microservices namespace
kubectl get -n microservices pod #verify keycloak server is running
kubectl get -n microservices svc #verify keycloak service

As per my experience, this keycloak service take time to start and ready service. Make sure all services should be ready status before open the service. Eg:

kubectl -n microservices get pod --watch
NAME                    READY   STATUS    RESTARTS   AGE
keycloak-0              1/1     Running   0          6m11s
keycloak-postgresql-0   1/1     Running   0          7m6s

Following command use to login into postgres, the password will be prompt executed the command.

kubectl -n microservices exec -it pod/keycloak-postgresql-0 -- psql -U keycloak -d keycloakdb

Note: The Terraform state file should be kept secure and encrypted (using encryption at rest) because it contains sensitive information, such as usernames, passwords, and Kubernetes cluster details etc.
Add our domain to the bottom of the /etc/hosts file on your local machine. This configuration should not be inside our working Linux box “test-microservices-module-envornment-box”; it should be applied to your personal machine's /etc/hosts file. (you will need administrator access):

127.0.0.1       keycloak.myapp.com

Keycloak Username is "admin" and password "MyPassword2222@", URl "http://keycloak.myapp.com"

Reference

Helm Keyloak Install Example

DEV Community: Binoy

Create the Admin Terraform Module and Terraform Admin User

Data Compliance (PII and PHI) Network Architecture

DevOps - Software Deployment Pipeline

DevOps - Software Deployment Pipeline

DevOps Software Deployment Pipeline Explanation:

Quality Attributes Explanation:

ISO Standard

Run the K8s in the private VPC setup

Overview

Objective

Network Archiecture

Infrastructure Archiecture

Prerequisites

Create the servie account and permission in GCP

Login GCP account

Configure the env variables

Create the Terraform workspace

Examples for the dev environment:

Examples for the prod environment:

Terraform Configuration

Setup the resources in GCP

Setup and connect IAP TCP Forwarding

Service Verification

Install Web Applications K8s cluster

Install the HAProxy ingress controller into master VM in the GCP console

Install sample nginx server in the K8s and create ingress resource of HAProxy

Destroy the resources in GCP account

K8s cluster setup in GCP with worker nodes autoscale and its discovery

Overview

Objective

Prerequisites

Login GCP account

Configure the env variables

Terraform Structure

Create the Terraform workspace

Examples for the dev environment:

Examples for the prod environment:

Terraform Configuration

Setup the resources in GCP

Install Web Applications K8s cluster

Install the HAProxy ingress controller into master VM in the GCP console

Install sample nginx server in the K8s and create ingress resource of HAProxy

Destroy the resources in GCP account

Debug the services

Setup K8s cluster on GCP VMs

Overview

Objective

Prerequisites

Login GCP account

Configure the env variables

Terraform Structure

Create the Terraform workspace

Examples for the dev environment:

Examples for the prod environment:

Terraform Configuration

Setup the resources in GCP

Verify the services are installed in the K8s VM and HAProxy VM

SSH into Master VM in the GCP console

SSH into Worker VM in the GCP console

SSH into HAProxy VM in the GCP console

Verify the Bucket

Verify the nodes join

Install Web Applications K8s cluster

Install the HAProxy ingress controller into master VM in the GCP console

Install sample nginx server in the K8s and create ingress resource of HAProxy

Destroy the resources in GCP account

Spring Boot 3.4.1 Keycloak introspect and method level security

Spring Boot 3.4.1 Keycloak introspect and method level security

Integration

Installing Dependencies Services

Keycloak Installation in Docker

MongoDB installation in Docker

Spring Security Dependencies

Keycloak configuration in YAML file

Security Configuration

Method Level Configuration

Keycloak Configuration

Create the Realm klight-api-gateway

Create the Client klight-api-gateway-admin-connect

Configure the `env` variables

Examples for the `dev` environment:

Examples for the `prod` environment:

Configure the `env` variables

Examples for the `dev` environment:

Examples for the `prod` environment:

Configure the `env` variables

Examples for the `dev` environment:

Examples for the `prod` environment:

Create the Realm `klight-api-gateway`

Create the Client `klight-api-gateway-admin-connect`