Build Production-Ready GCP Infrastructure from Scratch Part 02

Build Production-Ready GCP Infrastructure from Scratch: A Complete Console Guide

A 4-Part Series for Complete Beginners

---

Table of Contents

• Part 1: Foundation - Project Setup, VPC & Networking
• Part 2: Security Services - Secrets, Bastion & IAM ← You are here
• Part 3: Database & Compute Resources
• Part 4: Observability & Load Balancer

---

Part 2: Security Services - Secrets, Bastion & IAM

Overview

In this part, you'll build the security layer for your infrastructure. We'll create secrets for sensitive data, set up a bastion host for secure SSH access, configure IAP (Identity-Aware Proxy) for zero-trust access, and establish IAM permissions.

What you'll build:

• Secret Manager secrets for database credentials and API keys
• Bastion host with IAP tunneling
• OS Login for IAM-managed SSH authentication
• IAM roles for service account and user access

Estimated time: 30-45 minutes

Estimated cost: ~$11/month (Bastion VM only)

Cumulative cost: ~$53/month

---

Prerequisites

Before continuing, ensure you've completed Part 1:

• [ ] VPC dev-network exists
• [ ] 5 subnets created (including public-subnet)
• [ ] Cloud NAT gateway is running
• [ ] 4 service accounts created (including bastion-dev-sa)
• [ ] Firewall rules created

If you missed Part 1: Start with Part 1: Foundation →

---

Step 1: Create Secret Manager Secrets

What is Secret Manager?

Secret Manager is Google Cloud's secure storage for sensitive data:

• Database credentials
• API keys
• Certificates
• Tokens

Secrets are encrypted at rest and accessed via IAM permissions.

Why Not Hardcode Secrets?

Security Risk: Hardcoding secrets in:

• Configuration files (committed to Git)
• Startup scripts (visible to anyone with VM access)
• Environment variables (visible in process lists)

Best Practice: Store secrets in Secret Manager and fetch at runtime.

Cost: Secret Manager is free for active secrets. Storage costs ~$0.03/GB after the free tier.

Secret 1: Database Credentials

Navigation Path

1. Navigate to Security → Secret Manager
2. Click "Create Secret"

Secret Configuration

Basic Settings:

Field	Value	Notes
Name	db-credentials-dev	Environment-suffixed
Secret value	(See JSON below)	Click "Create or upload"

Secret Value (JSON format):

{
  "username": "app_admin",
  "password": "REPLACE_WITH_SECURE_PASSWORD",
  "host": "10.100.0.2",
  "database": "appdb",
  "port": "5432"
}

Note: The host IP will be updated after we create Cloud SQL in Part 3. For now, use the placeholder.

Password Generation:

# Generate a secure password
openssl rand -base64 32

Replication:

Field	Value	Notes
Replication	Automatic	Replicates across regions

Rotation:

Field	Value	Notes
Rotation period	Leave blank	No auto-rotation for now

Version Access:

Field	Value	Notes
Version access	Enable secret version	Access immediately

Click "Create secret".

Secret 2: API Key

Create API Key Secret

1. In Secret Manager, click "Create Secret"

Field	Value
Name	api-key-dev
Secret value	your-api-key-here

Replace with actual value: Use your actual API key or generate a placeholder for testing.

Replication: Automatic (same as above)

Click "Create secret".

Verify Secrets Created

You should see 2 secrets in Secret Manager:

Secret Name	Created	Versions
db-credentials-dev	(timestamp)	1
api-key-dev	(timestamp)	1

---

Step 2: Grant Secret Access to Service Accounts

What is IAM Access to Secrets?

Service accounts need permission to access secrets. Without this, VMs cannot fetch secrets at runtime.

Grant Access to Backend SA

1. Click on db-credentials-dev secret
2. Click the "Permissions" tab
3. Click "Grant access"

Add Principal:

Field	Value
New principals	backend-dev-sa@PROJECT_ID.iam.gserviceaccount.com

Select Role:

Role	Purpose
Secret Manager Secret Accessor	Can read secret values

Click "Save".

Grant Access to Cache and Observability SAs

Repeat for:

1. api-key-dev secret → Grant access to backend-dev-sa
2. db-credentials-dev secret → Grant access to cache-dev-sa (for PgBouncer)
3. api-key-dev secret → Grant access to observability-dev-sa (if needed)

Verify IAM Bindings

For db-credentials-dev, you should see:

Principal	Role
backend-dev-sa@PROJECT_ID	Secret Manager Secret Accessor
cache-dev-sa@PROJECT_ID	Secret Manager Secret Accessor

---

Step 3: Enable IAP (Identity-Aware Proxy)

What is IAP?

IAP provides zero-trust access to your VMs without:

• Public IPs
• VPN connections
• SSH keys in metadata

How it works:

1. User authenticates with Google Cloud
2. IAP establishes a secure tunnel
3. Traffic flows through Google's network (not public internet)

Why IAP is more secure: SSH traffic never traverses the public internet. Access is controlled by IAM, not SSH keys.

Navigation Path

1. Navigate to Security → Identity-Aware Proxy
2. Click "Go to Identity-Aware Proxy"
3. Click "Tick the checkbox" to enable IAP

Configure IAP for SSH

1. Click "Configure SSH and TCP Resources"
2. Click "Enable IAP"

SSH and TCP forwarding: ✓ Enable

Note: IAP for TCP forwarding is required for SSH tunneling.

Verify IAP Enabled

You should see:

• Status: Enabled
• Resources: (None yet - bastion will be added)

---

Step 4: Create IAP Firewall Rule

What is the IAP Firewall Rule?

This rule allows IAP's infrastructure to connect to your VM. IAP uses IP range 35.235.240.0/20 for tunneling.

Note: If using the Terraform bastion module, this rule is created automatically. For console setup, we create it manually.

Navigation Path

1. Navigate to VPC networks → Firewall
2. Click "Create firewall rule"

Firewall Configuration

Field	Value	Notes
Name	dev-firewall-allow-iap	Descriptive
Network	dev-network	Our VPC
Priority	1000	Standard allow priority
Direction	Ingress	Inbound traffic
Action	Allow	Allow traffic
Target service accounts	bastion-dev-sa	Only bastion host

Source filter:

Field	Value
Source IPv4 ranges	35.235.240.0/20

Why this range: This is Google's IAP forwarding range. Without this rule, IAP cannot tunnel to your VM.

Protocols and ports:

Field	Value
TCP	✓ Checked
Ports	22

Logging:

Field	Value
Log config	On

Click "Create".

---

Step 5: Create Bastion Host VM

What is a Bastion Host?

A bastion host is a secure entry point to your private network. It's the only VM with:

• Public IP (optional)
• SSH access enabled
• Access to private subnets

Security Model:

• Users SSH to bastion via IAP
• From bastion, SSH to backend VMs
• Backend VMs have no public IP

Cost Alert: Bastion VM costs ~$11/month (e2-small). You can stop it when not in use to save costs.

Navigation Path

1. Navigate to Compute Engine → VM instances
2. Click "Create instance"

Basic Settings

Field	Value	Notes
Name	dev-bastion	Environment-prefixed
Region	europe-west1	Belgium
Zone	europe-west1-b	Zone b

Machine Configuration

Field	Value	Notes
Machine type	e2-small	2 vCPU, 2GB RAM
CPU platform	Intel/AMD	Default (Automatic)

Cost: e2-small ≈ $11/month. Sufficient for bastion (just forwards SSH).

Boot Disk

Click "Change" to configure:

Field	Value	Notes
OS	Ubuntu	Popular Linux distribution
Version	Ubuntu 22.04 LTS Minimal	Long-term support
Disk type	pd-balanced	Balance cost/performance
Size	20 GB	Sufficient for bastion

Click "Select".

Network Interface

Network settings:

Field	Value	Notes
Network	dev-network	Our VPC
Subnetwork	public-subnet	Must be public subnet
Network interface type	IPv4	IPv4 only

External IPv4 address:

Field	Value	Notes
Network Service Tiers	Premium	Default (recommended)
External IPv4 address	Ephemeral	Do NOT create static IP

Why ephemeral IP: Bastion uses IAP for access, so public IP is not directly accessed. Save money by using ephemeral IP.

Network Tags

Add network tag:

Field	Value
Network tags	bastion-host

Note: Network tags are used for the IP whitelist backup firewall rule (IAP is primary).

Identity and API Access

Service account:

Field	Value	Notes
Service account	bastion-dev-sa	Our bastion SA

Access scopes:

Field	Value
Access scopes	Allow full access to all Cloud APIs

Why full access: Bastion needs to access Secret Manager (for credentials) and other services. In production, restrict to specific scopes.

Security - OS Login

Expand "Security" section:

Field	Value	Notes
Enable OS Login	✓ Enable	Critical for IAM-based SSH

What is OS Login: OS Login allows you to manage SSH access via IAM. No SSH key distribution needed. Users log in with their Google account.

Advanced Options - Metadata

No custom metadata needed for bastion (OS Login handles authentication).

Create the VM

Review all settings and click "Create".

Wait 2-3 minutes for VM creation. You should see:

Instance 'dev-bastion' is RUNNING

---

Step 6: Grant IAP Access to Users

What IAM Roles Are Needed?

Users need two roles to SSH via IAP:

1. IAP-secured Tunnel User - Permission to use IAP tunnel
2. Compute OS Login - Permission to log in to VMs

Note: These roles are granted at the project level, not per VM.

Grant IAP Tunnel Access

1. Navigate to IAM & Admin → IAM
2. Click "Grant access"

Add principal:

Field	Value
New principals	[e-mail]

Select role:

Role	Path
IAP-secured Tunnel User	Identity-Aware Proxy → IAP-secured Tunnel User

Click "Save".

Grant OS Login Access

Click "Grant access" again:

Add principal: Same email as above

Select role:

Role	Path
Compute OS Login	Compute Engine → Compute OS Login

Click "Save".

Verify IAM Bindings

You should see your user with these roles:

Principal	Role
[e-mail]	IAP-secured Tunnel User
[e-mail]	Compute OS Login

---

Step 7: Add SSH Key for OS Login (Optional)

What is OS Login SSH Key?

OS Login can use IAM-managed SSH keys. These are:

• Stored in Google Cloud
• Automatically rotated
• Managed via IAM

Alternative: You can add your own SSH key to VM metadata.

Best Practice: Use OS Login with IAM for production. No SSH key management overhead.

Add SSH Key to Project (Optional Backup)

If OS Login fails, you can use SSH keys:

1. Navigate to Compute Engine → Metadata
2. Click "SSH Keys" tab
3. Click "Edit"
4. Click "Add item"

Generate SSH key locally:

ssh-keygen -t rsa -f ~/.ssh/gcp_bastion -C [e-mail]

Copy public key:

cat ~/.ssh/gcp_bastion.pub

Paste the public key into the SSH Keys field:

Field	Value
SSH key	ssh-rsa AAAA... [e-mail]

Click "Save".

Security Note: Metadata SSH keys are less secure than OS Login. Use only as backup.

---

Part 2 Verification Checklist

Before moving to Part 3, verify:

• [ ] 2 secrets created (db-credentials-dev, api-key-dev)
• [ ] Backend, cache, and observability SAs have Secret Accessor role
• [ ] IAP is enabled for TCP forwarding
• [ ] IAP firewall rule created (allows 35.235.240.0/20)
• [ ] Bastion VM is RUNNING
• [ ] Bastion has bastion-dev-sa attached
• [ ] OS Login is enabled on bastion
• [ ] Your user has IAP-secured Tunnel User role
• [ ] Your user has Compute OS Login role

---

Test SSH via IAP

Verify IAP Access

Test SSH connection to bastion:

gcloud compute ssh dev-bastion \
  --project=PROJECT_ID \
  --zone=europe-west1-b \
  --tunnel-through-iap

Expected output:

Welcome to Ubuntu 22.04 LTS
your-email@dev-bastion:~$

Troubleshooting: If SSH fails:

1. Verify IAP is enabled
2. Check IAP firewall rule exists
3. Verify your user has IAP-secured Tunnel User role
4. Check OS Login is enabled on bastion

---

Cost Summary - Part 2

Component	Monthly Cost	Notes
Bastion Host (e2-small)	~$11	24/7 operation
Secret Manager	Free	Within free tier
IAP	Free	No additional cost
Total Part 2	~$11	~$11/month

Cumulative Cost (Part 1 + 2):

Component	Monthly Cost
Part 1 (VPC, NAT, etc.)	~$42
Part 2 (Bastion)	~$11
Total	~$53/month

Cost Optimization: Stop bastion VM when not in use to save ~$11/month. Start it only when needed for SSH access.

---

Troubleshooting - Part 2

Issue: Cannot Access Secret

Symptom: "Permission denied" when accessing secret

Solution:

1. Check service account has Secret Manager Secret Accessor role
2. Verify secret name matches exactly (case-sensitive)
3. Check secret has at least 1 version

# List secret versions
gcloud secrets versions list db-credentials-dev

# Access secret
gcloud secrets versions access latest --secret=db-credentials-dev

Issue: IAP Connection Fails

Symptom: "IAP does not have permission" error

Solution:

1. Verify IAP is enabled
2. Check your user has IAP-secured Tunnel User role
3. Verify IAP firewall rule exists (allows 35.235.240.0/20)

# Check IAP status
gcloud compute ssh dev-bastion --tunnel-through-iap --dry-run

Issue: OS Login Not Working

Symptom: "OS Login is not enabled" error

Solution:

1. Verify OS Login is enabled on bastion VM
2. Check your user has Compute OS Login role
3. Ensure no conflicting SSH keys in metadata

# Enable OS Login on existing VM
gcloud compute instances add-metadata dev-bastion \
  --metadata enable-oslogin=TRUE

Issue: Bastion Cannot Access Secrets

Symptom: Secret access denied from bastion

Solution:

1. Verify bastion-dev-sa has Secret Manager Secret Accessor role
2. Check service account is attached to VM
3. Ensure VM has access scopes for Secret Manager

---

What's Next - Part 3?

In Part 3: Database & Compute, you'll build:

• Private Service Connection for Cloud SQL
• Cloud SQL PostgreSQL instance with HA
• Managed Instance Group (MIG) for backend VMs
• Cache VM with Redis and PgBouncer

Continue to Part 3: Database & Compute →

---

References

• Secret Manager Documentation
• IAP Documentation
• OS Login Documentation
• Bastion Host Best Practices

---

Security Layer Complete! Your secrets are securely stored, bastion host is ready for secure SSH access, and IAM permissions are configured. Next, we'll add the data and compute layers (Cloud SQL, MIG, Cache).