DEV Community: cypher682

Event-Driven AWS Security Automation: Real-Time Detection and Remediation with Terraform and Pulumi

cypher682 — Wed, 29 Jul 2026 20:08:31 +0000

cloudguard-pro is an event-driven AWS security platform that detects and automatically remediates cloud misconfigurations. Seven Lambda functions, a shared data contract, dual IaC provisioning, and a unified findings store backed by DynamoDB Streams and SNS.

This writeup covers the architecture, the IaC split, and the implementation decisions that shaped the system.

Architecture

CloudTrail → EventBridge (default bus)
               ↓ forward rule
           EventBridge (cloudguard-pro-bus)
               ↓ rules per event type
    ┌──────────┬──────────┬──────────┐
    ▼          ▼          ▼          ▼
event-     sg-        s3-        iam-
ingestor  remediator remediator remediator
    │          │          │          │
    └──────────┴──────────┴──────────┘
               ↓ all write to
           DynamoDB (findings table)
               ↓ Streams
           finding-notifier → SNS → email

CloudWatch Schedule (6h)  → cis-scanner        → DynamoDB
CloudWatch Schedule (1d)  → security-hub-sync  → DynamoDB
AWS Config (continuous)   → managed rules      → Security Hub

CloudTrail delivers management events to the default bus only. A forwarding rule relays relevant EC2, S3, and IAM events to a custom bus — cloudguard-pro-bus — which keeps cloudguard's routing isolated from anything else in the account. Remediators trigger independently and in parallel on specific event patterns. Everything writes to a single DynamoDB findings table.

Detection and Remediation Coverage

Auto-remediated:

Condition	Lambda	Action
SG ingress: 0.0.0.0/0 on port 22 or 3389	sg-remediator	Revoke the offending rule
S3 public access block removed	s3-remediator	Re-enable all four block flags
IAM user: console access, no MFA	iam-remediator	Delete login profile

Flagged, human review:

Condition	Lambda	Severity
Root account API activity	event-ingestor	CRITICAL — no auto-remediation
S3 bucket policy or ACL change	event-ingestor	HIGH — intent ambiguous

Scheduled — CIS AWS Foundations Benchmark v1.4 (every 6 hours):

CIS Control	Check
1.4	Root account has no active access keys
1.5	MFA enabled for root account
1.10	MFA for all IAM users with console access
3.1	CloudTrail enabled and logging
3.3	CloudTrail log file validation enabled
4.1	No unrestricted SSH (port 22)
4.2	No unrestricted RDP (port 3389)

Continuous — AWS Config managed rules:

CIS Control	Rule
1.5	ROOT_ACCOUNT_MFA_ENABLED
1.10	IAM_USER_MFA_ENABLED
2.1.1	S3_BUCKET_LEVEL_PUBLIC_ACCESS_PROHIBITED
2.2.1	ENCRYPTED_VOLUMES
3.1	CLOUD_TRAIL_ENABLED
4.1	INCOMING_SSH_DISABLED

Security Hub aggregates Config findings alongside GuardDuty and Inspector. The security-hub-sync Lambda pulls active Security Hub findings into DynamoDB daily, unifying them with CloudTrail-sourced and CIS scanner findings into a single view.

Remediation scope is conservative by design. Only explicit on/off controls are auto-restored. Anything requiring interpretation of intent is flagged and left for human review.

Dual IaC: Terraform and Pulumi

Terraform owns the static infrastructure layer. Pulumi owns the compute layer.

Terraform: EventBridge bus and rules, DynamoDB table and GSIs, SNS topic, AWS Config recorder and managed rules, Security Hub, S3 state backend.

Pulumi: 7 Lambda functions, 7 IAM execution roles, Lambda Layer, EventBridge invoke permissions, DynamoDB Streams event source mapping.

The split is motivated by what each tool handles cleanly. Terraform's declarative model is well-suited to long-lived, low-churn infrastructure. Pulumi's imperative Python is better suited to compute provisioning that requires logic at deploy time — walking source directories, building zips, conditionally attaching layers:

# pulumi/__main__.py
def _make_lambda(name: str, role: aws.iam.Role, env_vars: dict, timeout: int = 30):
    zip_path = _zip_lambda(name)
    fn_name = f"cloudguard-{name.replace('_', '-')}"

    return aws.lambda_.Function(
        fn_name,
        name=fn_name,
        runtime="python3.12",
        code=pulumi.FileArchive(zip_path),
        handler="handler.handler",
        role=role.arn,
        layers=[shared_layer.arn],
        timeout=timeout,
        environment=aws.lambda_.FunctionEnvironmentArgs(variables={
            "FINDINGS_TABLE_NAME": findings_table_name,
            "SNS_TOPIC_ARN": sns_topic_arn,
            "AWS_ACCOUNT_ID": aws_account,
            **env_vars,
        }),
    )

Deploy sequence is three-pass: Terraform creates the EventBridge rules with null targets → Pulumi creates Lambdas and exports ARNs → Terraform re-applies to wire Lambda ARNs as rule targets. Pulumi reads Terraform outputs via pulumi config set. The cross-tool dependency is explicit, documented, and handled without any custom glue.

The full rationale, tradeoffs, and honest assessment of when you'd choose one over the other is in docs/iac-comparison.md in the repo.

The Finding Data Contract

Every Lambda in the system — ingestor, remediators, CIS scanner, Security Hub sync — writes the same Finding dataclass:

@dataclass
class Finding:
    service: str
    finding_type: str
    severity: Severity          # CRITICAL, HIGH, MEDIUM, LOW, INFO
    source: FindingSource       # CLOUDTRAIL, CONFIG_RULE, CIS_SCANNER, SECURITY_HUB
    resource_id: str
    description: str
    region: str
    account_id: str

    finding_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    created_at: str = field(default_factory=_utc_now_iso)
    status: FindingStatus = FindingStatus.DETECTED
    remediation_action: str | None = None
    remediated_at: str | None = None
    raw_event_summary: dict | None = None
    cis_check_id: str | None = None

finding-notifier reads this same model from DynamoDB Streams before publishing SNS alerts. No raw dict juggling across function boundaries. Schema drift surfaces as a dataclass validation error in tests, not a silent write to DynamoDB with a missing field.

DynamoDB Schema

Table: cloudguard-findings
Billing: PAY_PER_REQUEST
Streams: Enabled, NEW_AND_OLD_IMAGES — triggers finding-notifier on INSERT

Key	Type	Purpose
`finding_id` (PK)	String UUID	Unique per finding
`created_at` (SK)	ISO8601	Time ordering within partition
GSI: `severity-index`	PK: severity	Query all CRITICAL findings
GSI: `service-index`	PK: service	Query all EC2 or IAM findings

Testing

All 70 tests run against moto-mocked AWS services. No live account required:

pip install -r requirements-dev.txt
python -m pytest tests/ -v
# 70 passed — 85.39% coverage

Moto mocks DynamoDB, EC2, S3, IAM, SNS, CloudTrail, Config, and Security Hub. Each handler is tested in isolation with production-representative CloudTrail event payloads. CI runs all six jobs on every push: pytest, Bandit, Checkov, Ruff, and two guarded deploy jobs that only run on manual dispatch.

Sprint Results — July 27, 2026

Open SSH rule (port 22, 0.0.0.0/0) added to a security group — auto-revoked within 60 seconds. IpPermissions: [] confirmed via CLI. CRITICAL SNS alert received.
IAM user created with console access and no MFA — login profile deleted within 30 seconds. NoSuchEntity confirmed via CLI. CRITICAL SNS alert received.
CIS scanner invoked manually — CIS_4_1_UNRESTRICTED_PORT_22 (HIGH/DETECTED) and CIS_3_1_PASS (INFO/ACKNOWLEDGED) written to DynamoDB.
7 distinct SNS alerts received across all severity levels.
Pulumi: 37 resources created in 30 seconds. pulumi destroy cleaned up cleanly.
Total sprint cost: ~$3–4.

Repo

github.com/cypher682/cloudguard-pro

Building the Full LGTM Observability Stack from Scratch: Metrics, Logs, Traces, and Alerting

cypher682 — Wed, 29 Jul 2026 14:35:15 +0000

observability-platform is a full LGTM (Loki, Grafana, Tempo, Prometheus) observability stack built from scratch — containerized with Docker Compose, deployable to Kubernetes, with a custom Python Prometheus exporter, 7 Grafana dashboards provisioned as code, Alertmanager routing, and an OpenTelemetry Collector pipeline.

The application never talks to a monitoring backend directly. It emits telemetry once, to one collector, which owns the routing decisions from there. Everything downstream — storage, correlation, alerting, dashboards — is built and operated as part of this system, not bolted on afterward.

Specifically: the app exports traces to the OTel Collector. Promtail ships its container logs to Loki independently via the Docker socket. Prometheus scrapes metrics directly from exporters (node-exporter, cAdvisor, the custom business exporter). The collector is configured to also handle OTLP metrics and logs, but this stack intentionally mixes push (traces) and scrape (metrics) patterns rather than routing every signal through one pipe.

Below: the architecture, the module boundaries, and the specific tradeoffs behind each one.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                      Observability Stack                        │
│                                                                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────┐   │
│  │Prometheus │  │  Loki    │  │  Tempo   │  │ Alertmanager │   │
│  │(metrics)  │  │ (logs)   │  │(traces)  │  │  (routing)   │   │
│  └─────┬────┘  └────┬─────┘  └────┬─────┘  └──────┬───────┘   │
│        │             │             │                │            │
│  ┌─────┴─────────────┴─────────────┴────────────────┴───────┐   │
│  │                    Grafana (dashboards)                   │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌─────────────┐  ┌──────────────┐  ┌──────────────────────┐   │
│  │  OTel        │  │  Promtail    │  │  custom-exporter     │   │
│  │  Collector   │  │  (log ship)  │  │  (Python, /metrics)  │   │
│  └──────┬──────┘  └──────┬───────┘  └──────────┬───────────┘   │
│         │                │                      │                │
│  ┌──────┴────────────────┴──────────────────────┴───────────┐   │
│  │              Target App (FastAPI) + Load Generator        │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Data flows:

Metrics: Node Exporter (host), cAdvisor (containers), Blackbox Exporter (endpoints), and custom-exporter (business logic) are scraped by Prometheus.
Logs: Promtail auto-discovers containers via Docker socket and ships logs to Loki.
Traces: A FastAPI target app instrumented with the OTel SDK pushes spans to the OTel Collector, which routes to Tempo.
Alerting: Prometheus evaluates alert rules, passes firing events to Alertmanager, which routes to a webhook receiver.

The OpenTelemetry Collector Pipeline

The design decision worth explaining: traces don't go straight from the app to Tempo. They go to the OTel Collector first (port 4317, gRPC), which then forwards to Tempo. Logs and metrics take separate, more direct paths — Promtail ships container logs to Loki, and Prometheus scrapes metrics from exporters. The collector's config also defines metrics and logs pipelines (so it could handle all three signals), but in this stack it's actively used for traces.

That's a deliberate mix, not an oversight: traces benefit from a broker in front of them (batching, resource enrichment, and an easy point to add sampling later), while metrics and logs already have well-established scrape/push patterns in this ecosystem that don't need one.

# otel-collector-config.yaml (simplified)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"
  otlp/tempo:
    endpoint: "tempo:4317"
    tls:
      insecure: true
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      exporters: [loki]

Trace-to-Log Correlation

The real power of the LGTM stack is not having metrics, logs, and traces — it is being able to jump between them.

The target FastAPI app enables OpenTelemetry log correlation, then emits normal application logs:

LoggingInstrumentor().instrument(set_logging_format=True)
logger.info("work request completed")

In Grafana's Loki datasource config, a derivedFields block maps that trace_id to Tempo:

derivedFields:
  - datasourceUid: tempo
    matcherRegex: "trace_id=(\\w+)"
    name: TraceID
    url: "$${__value.raw}"

Click any log line in Grafana. The right panel loads the exact trace waterfall. No searching by timestamp, no copying IDs. The correlation is built into the data path.

Custom Python Exporter

Off-the-shelf exporters cover databases and runtimes. But understanding the Prometheus data model means being able to build one from scratch.

The custom exporter is a FastAPI app running prometheus_client on port 9200. A background simulation thread updates metrics at configurable intervals:

# Metric types demonstrated:
active_users_total    = Gauge("active_users_total", "Simulated active database users")
jobs_queued_total     = Counter("jobs_queued_total", "Total jobs queued")
jobs_processed_total  = Counter("jobs_processed_total", "Total jobs processed", ["status"])
webhook_duration      = Histogram("webhook_delivery_duration_seconds", "Webhook delivery latency")
webhook_success       = Gauge("webhook_delivery_success_ratio", "Webhook success rate")
queue_depth           = Gauge("job_queue_depth", "Current job queue depth")

It demonstrates all three core metric types (gauge, counter, histogram), a label cardinality decision on jobs_processed_total (split by status, not by job ID — cardinality stays bounded), and the plain scrape protocol. Prometheus hits /metrics on port 9200 exactly like it would hit any production exporter.

Dashboards as Code

Seven Grafana dashboards are provisioned as JSON files in the dashboards/ directory. On first boot, Grafana loads them automatically — zero manual configuration.

Dashboard	What It Shows
Infrastructure Overview	CPU, memory, disk, network per host
Container Metrics	Per-container resource usage from cAdvisor
Application SLIs	Request rate, error rate, latency (p50/p95/p99)
Custom Business Metrics	From the custom Python exporter
Loki Log Explorer	Pre-built LogQL queries
Tempo Traces	Trace search + waterfall view
Alertmanager	Active alerts, silence status

The provisioning model: Grafana's datasources/ and dashboards/ config are mounted as volumes locally. The Kubernetes path enables the Grafana Helm chart's dashboard sidecar (sidecar.dashboards.enabled: true in the Helm values), which watches for ConfigMaps labeled grafana_dashboard and loads them automatically — the dashboard ConfigMaps themselves aren't checked into this repo, the sidecar mechanism is. The entire local observability UI is reproducible from a docker compose up.

Alertmanager: Noise Suppression

Alert fatigue is a real operational problem. Alertmanager implements it with inhibition rules:

inhibit_rules:
  - source_matchers: [severity="critical"]
    target_matchers: [severity="warning"]
    equal: ["component", "instance"]

When a host has critical CPU usage, warning-level application latency alerts on the same instance are suppressed. The critical alert is the signal. The warning is noise while the critical is active.

Seven alert rules are defined:

Rule	Condition	Severity
HostHighCPU	CPU > 85% for 5m	critical
HostDiskFull	Disk > 90%	critical
ContainerOOMKilled	OOM kill event	critical
AppHighErrorRate	Error rate > 5% for 2m	critical
AppHighLatency	p99 > 500ms for 3m	warning
BlackboxEndpointDown	Endpoint unreachable for 1m	critical
CustomExporterJobQueueHigh	Queue depth > 100	warning

Kubernetes Transition

The stack is designed to move from Docker Compose to Kubernetes without architectural changes. In this repo, custom app components are represented with raw manifests, while Prometheus/Grafana/Alertmanager, Loki/Promtail, and Tempo are configured through Helm values:

Component	K8s / Helm Mapping	Why
Prometheus	kube-prometheus-stack Helm values with persistent storage	Needs durable time-series data
Grafana	kube-prometheus-stack Helm values with dashboard and datasource provisioning	Reproducible UI configuration
Alertmanager	kube-prometheus-stack Helm values with webhook routing	Alert grouping and routing
Loki	loki-stack Helm values with persistence	Durable log storage
Promtail	loki-stack Helm values, chart-managed DaemonSet	One per node, ships container logs
Tempo	tempo Helm values with persistence	Trace backend configured for the cluster
Postgres	Raw StatefulSet manifest	Local backing store for the target app
Target App	Raw Deployment manifest	Instrumented application workload
Custom Exporter	Raw Deployment manifest	Stateless business metrics exporter
Prometheus RBAC	Raw ServiceAccount, ClusterRole, ClusterRoleBinding	Documents required discovery permissions

Key lesson: Stateful services on Kubernetes require careful liveness/readiness probe tuning. Under resource pressure, slow initialization triggers cascading probe timeouts and restart loops. Conservative startup delays and resource bounds prevent this.

The full Kubernetes manifests are in kubernetes/manifests/. Helm values for kube-prometheus-stack are in kubernetes/helm-values/.

Repo

cypher682/observability-platform

Modular AWS Infrastructure With Terraform: 2-Tier VPC, CIS-Hardened EC2, OIDC CI/CD, and a $6 Sprint Model

cypher682 — Wed, 29 Jul 2026 07:03:59 +0000

infra-core is a production-grade AWS infrastructure platform built with Terraform 1.7, Ansible, and GitHub Actions. It provisions a complete multi-environment AWS stack — 2-tier VPC, EC2 Auto Scaling Group, RDS PostgreSQL, ALB, SSM-based secrets, CloudWatch observability, and a fully automated CI/CD pipeline using OIDC federation.

This article documents the system as built: architecture, module design, security decisions, configuration management, and the cost model that makes it viable for portfolio and proof-of-concept use.

Repo: cypher682/infra-core

Architecture

Internet
  │ HTTP :80
  ▼
ALB (public subnets, 2 AZs)
  │
  ▼
EC2 ASG (private subnets, 2 AZs)
  │ :5432
  ▼
RDS PostgreSQL (private subnets)

SSM Session Manager ──► EC2            (no SSH port, no bastion)
SSM Parameter Store ──► app secrets    (pulled at runtime)
S3 ────────────────────► VPC flow logs + app assets
CloudWatch ────────────► metrics, log groups, alarms, SNS, dashboard
GitHub Actions OIDC ──► Terraform CI/CD (no static credentials)

The VPC enforces a hard 2-tier separation. Public subnets hold the ALB and NAT Gateway. Private subnets hold the EC2 ASG and RDS. No EC2 instance is directly reachable from the internet. RDS is reachable only from the application security group (sg-app) on port 5432 — not from the ALB, not from a bastion, not from the internet.

Module Design

The infrastructure is composed of eight Terraform modules, each owning a single concern:

modules/
  vpc/        — VPC, subnets, SGs, NAT GW, VPC flow logs
  iam/        — GitHub OIDC role, EC2 instance profile
  s3/         — app assets + flow logs buckets
  ssm/        — Parameter Store secret namespace
  rds/        — PostgreSQL, subnet group, parameter group
  alb/        — ALB, target group, HTTP listener
  ec2-asg/    — launch template, ASG, CPU scaling policies
  cloudwatch/ — alarms, log groups, SNS topic, dashboard

Each module receives only what it needs. ec2-asg takes sg_app_id, target_group_arn, and ec2_instance_profile_name — not the full VPC object. cloudwatch takes alb_arn_suffix and target_group_arn_suffix as direct outputs from the ALB module, not derived strings.

One specific pattern worth noting — the ALB module exposes arn_suffix directly from the AWS provider resource, rather than leaving consumers to parse it from the full ARN:

# modules/alb/outputs.tf
output "alb_arn_suffix" {
  value = aws_lb.main.arn_suffix
}

output "target_group_arn_suffix" {
  value = aws_lb_target_group.app.arn_suffix
}

The AWS provider exposes this natively. Using split(":", arn)[5] is fragile — if the ARN format changes, CloudWatch alarms silently bind to a wrong dimension string. Using the native attribute avoids that entirely.

Security: No Bastion Host

The conventional EC2 access pattern uses an SSH jump box in a public subnet. It requires an open port 22, a key pair, and an additional instance to maintain and patch.

AWS Systems Manager Session Manager replaces all of it:

No port 22 in any security group
No key pairs distributed or stored
No bastion instance
Access gated by IAM policy, not network rules
Session activity logged to CloudWatch automatically

The EC2 IAM role is granted AmazonSSMManagedInstanceCore. The SSM agent ships pre-installed on Amazon Linux 2023. No additional setup.

# Interactive shell on a private EC2 instance — no SSH, no key pair
aws ssm start-session --target <instance-id>

This is also the mechanism Ansible uses. The dynamic inventory plugin queries EC2 by tag and connects via the SSM connection plugin — no static IP addresses, no host files that drift after an ASG refresh.

Security: OIDC for GitHub Actions

Storing AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in GitHub Secrets is the default pattern. These are long-lived credentials with no automatic expiry, and they carry real risk if leaked.

The alternative is OIDC federation. GitHub Actions generates a short-lived JWT per job run. AWS validates it against the registered OIDC provider and issues temporary session credentials. No long-lived keys exist.

# modules/iam/main.tf

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}

data "aws_iam_policy_document" "github_actions_assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:${var.github_org}/${var.github_repo}:*"]
    }
  }
}

The StringLike condition scopes the role to a single repository. No other repo in the org can assume it.

Secrets: SSM Parameter Store

All credentials are stored as SecureString parameters in SSM Parameter Store, encrypted with the AWS managed KMS key at no additional cost. The EC2 instance profile grants read access scoped to the project namespace only:

"arn:aws:ssm:${var.aws_region}:${var.aws_account_id}:parameter/${var.project}/*"

The application reads secrets at runtime via the AWS SDK. Credentials never appear in Terraform state, .env files committed to source control, or CI/CD logs.

SSM Parameter Store standard parameters cost $0. AWS Secrets Manager costs $0.40/secret/month. For this use case, SSM is the correct choice.

Configuration Management: Ansible

Terraform provisions the infrastructure. Ansible handles instance configuration. Six roles, run in sequence:

common/      — system update, NTP, non-root app user, directory layout
hardening/   — CIS AL2023 baseline: fail2ban, auditd, root SSH disabled, sysctl hardening
docker/      — Docker install + daemon hardening (no-new-privileges, live-restore)
nginx/       — reverse proxy configuration, security headers
monitoring/  — CloudWatch agent fetch-config from SSM + start
app-deploy/  — SSM secret fetch, docker pull from ECR, container run

The CloudWatch agent configuration is stored as a JSON parameter in SSM, managed by Terraform:

# modules/cloudwatch/main.tf
resource "aws_ssm_parameter" "cloudwatch_agent_config" {
  name  = "/AmazonCloudWatch-linux-${var.project}-${var.environment}"
  type  = "String"
  value = jsonencode({ ... })
}

The monitoring Ansible role runs amazon-cloudwatch-agent-ctl --fetch-config ssm:/<param-name>. Configuration is owned by Terraform, consumed by Ansible, applied to the instance. Clean handoff between layers.

The hardening role applies the CIS Amazon Linux 2023 baseline:

Root SSH login disabled
Password authentication disabled — key-based only
fail2ban with SSH brute-force protection
auditd rules for sensitive file monitoring (/etc/passwd, /etc/sudoers, /etc/ssh/sshd_config)
Sysctl hardening: IP forwarding off, source routing disabled, ICMP redirects disabled, SYN cookies enabled

CI/CD Pipeline

Two GitHub Actions workflows, both authenticated via OIDC:

tf-plan.yml — triggered on every pull request to main:

Configure AWS credentials (OIDC — no static keys)
terraform init
terraform validate
Checkov IaC security scan — pipeline blocks on failure
terraform plan with scoped variable injection
Plan output posted as a PR comment

tf-apply.yml — triggered on merge to main:

Configure AWS credentials (OIDC)
terraform apply -auto-approve
Retrieve ALB DNS name from Terraform output
Wait 60 seconds for EC2 instances to register with the target group
HTTP health check against /health — 5 retries, 15-second interval
Exit 0 on HTTP 200, pipeline failure on any other outcome

The smoke test is a real HTTP check against the provisioned ALB. If the EC2 instances fail to register within the health check grace period, the pipeline fails and the merge is flagged.

Cost Model

Continuous operation of this stack would cost approximately $75–100/month. The sprint model eliminates idle cost entirely: provision, collect evidence, destroy.

Resource	Hourly Rate	2-Day Total
NAT Gateway	$0.045/hr	~$2.20
ALB	$0.008/hr	~$0.40
EC2 t3.micro x1	$0.0104/hr	~$0.50
RDS db.t3.micro	$0.017/hr	~$0.82
S3 + CloudWatch	minimal	~$0.10
Total		~$4–6

A single NAT Gateway serves both AZs in dev and staging. Multi-AZ NAT Gateway would double the NAT cost (~$2.20 → ~$4.40) with no reliability benefit in non-production environments — NAT Gateway itself is the dominant cost driver.

Multi-Environment

Dev and staging are identical in structure, differentiated by CIDR blocks to allow simultaneous deployment without VPC overlap:

dev: 10.0.0.0/16 — public 10.0.1-2.0/24, private 10.0.10-11.0/24
staging: 10.1.0.0/16 — public 10.1.1-2.0/24, private 10.1.10-11.0/24

Each environment has its own Terraform state, its own SSM parameter namespace (/infra-core/dev/*, /infra-core/staging/*), and its own CloudWatch log groups and alarms.

What Comes Next

The following are documented in code but not yet enabled — intentional trade-offs for sprint cost control:

S3 remote backend with DynamoDB lock — commented-out block in environments/*/main.tf, ready to uncomment for team use
HTTPS listener — full listener block documented in modules/alb/main.tf; add ACM cert ARN and flip the listener
Multi-AZ RDS — multi_az = true in modules/rds/main.tf; disabled for cost
Kubernetes — this architecture maps directly: EC2 ASG → Deployment, ALB → Ingress Controller, RDS → managed database, SSM Parameter Store → Secrets Store CSI Driver

Repo: cypher682/infra-core

Building an Async Job Platform with BullMQ, Socket.io, and Webhooks

cypher682 — Mon, 29 Jun 2026 19:27:10 +0000

NodeFlow is an asynchronous job orchestration and webhook delivery API built with Node.js, TypeScript, BullMQ, Socket.io, Prisma, PostgreSQL, Redis, Docker, and GitHub Actions.

The goal was to build a multi-process distributed backend service — not a CRUD API with a background task bolted on. A system that forces real decisions about process separation, inter-process communication, failure handling, and delivery guarantees.

This is a writeup of what was built, what decisions were made, and why.

Architecture

NodeFlow runs as two separate processes that share state through Redis and PostgreSQL:

Client
  │
  ▼
Express API (port 4000)
  │── POST /v1/jobs      → validates, writes Job to DB, enqueues to BullMQ
  │── GET  /v1/jobs/:id  → reads job state from PostgreSQL
  │── Socket.io server   → pushes real-time status events to connected clients
  │── Bull Board         → /admin/queues visual dashboard
  │── Swagger UI         → /docs
  │
  └──► Redis (BullMQ queues: job-processing, webhook-dispatch, file-processing)
         │
         ▼
    BullMQ Worker Process
         │── picks up job from queue
         │── updates DB state (RUNNING → SUCCEEDED / FAILED)
         │── emits job:status event via Redis pub/sub → forwarded to Socket.io clients
         │── dispatches webhook delivery
         │── writes structured logs to PostgreSQL
         ▼
    PostgreSQL — durable state: jobs, logs, webhooks, deliveries, files, API keys

The API and Worker do not communicate directly. No shared memory, no imports of each other. Redis is the message bus. PostgreSQL is the source of truth.

This is the same shape as a Kubernetes deployment: two separate Deployments, one managed Redis, one managed database.

The Job Lifecycle

A client POSTs a job:

// POST /v1/jobs
{
  "type": "file.metadata.extract",
  "payload": { "fileId": "abc-123" },
  "priority": 5,
  "maxAttempts": 3
}

The API:

Validates the request with Zod.
Creates a Job record in PostgreSQL with status QUEUED.
Enqueues a BullMQ job to the job-processing queue.
Returns 202 Accepted immediately with the job ID.

The Worker:

Picks up the job from Redis.
Updates DB to RUNNING, emits job:status event.
Dispatches the handler for the registered job type.
On success: updates DB to SUCCEEDED, writes structured logs, emits another job:status, dispatches webhook.
On failure: if attempts < maxAttempts, BullMQ reschedules with exponential backoff. If exhausted: marks FAILED, emits final job:status.

The client never polls. It connects via WebSocket, subscribes to the job room, and receives state transitions as they happen.

Real-Time Updates: Cross-Process Socket.io

The hardest design question: how does the Worker push WebSocket events to clients connected to the API?

The WebSocket connection is held by the API process. The Worker is a completely separate Node.js process with no access to the API's in-memory Socket.io server. Connecting them directly would create tight coupling that breaks horizontal scaling.

The solution is @socket.io/redis-adapter on the API side and @socket.io/redis-emitter on the Worker side:

// API process — src/socket/index.ts
// Socket.io uses Redis as the pub/sub backbone
const pubClient = createRedisClient();
const subClient = pubClient.duplicate();
io.adapter(createAdapter(pubClient, subClient));

// Worker process — src/socket/emitter.ts
// Emits directly into Redis. No Socket.io server needed.
const emitter = new Emitter(createRedisClient());

export const emitJobUpdate = (jobId: string, status: string, result?: unknown) => {
  emitter.to(`job:${jobId}`).emit("job:status", { jobId, status, result });
};

The Worker writes to a Redis channel. The API subscribes to that channel and forwards to the correct WebSocket room. Neither process has a reference to the other. This is also how Socket.io works with multiple pods — the Redis adapter makes all instances appear as one.

Webhooks: HMAC Signatures, Exponential Backoff, Circuit Breaking

Webhook delivery is harder than it looks. The target might be down, slow, or returning errors. Naive delivery with no retry strategy loses events. Unlimited retries with no backoff hammers a broken endpoint.

NodeFlow uses three layers:

HMAC-SHA256 Signatures

Every payload is signed with the webhook's secret key:

const signature = createHmac("sha256", webhook.secret)
  .update(JSON.stringify(payload))
  .digest("hex");

// Sent as:
// X-Nodeflow-Signature: sha256=<hex>
// X-Nodeflow-Event: job.completed
// X-Nodeflow-Delivery-Id: <uuid>

The receiver verifies the signature to confirm the payload came from NodeFlow and was not tampered with.

Exponential Backoff

Failed deliveries are retried by BullMQ with increasing delays:

// 2s → 4s → 8s → 16s → ... up to 5 attempts
const nextRetryAt = new Date(Date.now() + Math.pow(2, attempts) * 1000);

After 5 failures the delivery is marked FAILED permanently.

Redis Circuit Breaker

Even with backoff, if an endpoint is completely dead you are still firing requests into the void:

// Redis key: circuit_breaker:failures:<url>
// Threshold: 5 consecutive failures → circuit opens for 5 minutes

if (await circuitBreaker.isCircuitOpen(webhook.url)) {
  // Skip HTTP attempt entirely. Mark delivery FAILED immediately.
  return;
}

After 5 consecutive failures to a URL, no further deliveries are attempted for 5 minutes. On circuit reset, the next attempt goes through. Success clears the failure counter.

This is the same pattern used in production service meshes — implemented at the application layer in Redis without a sidecar.

Idempotency: Safe Request Replay

Networks fail. Clients retry. Without idempotency, a retry creates a duplicate job, a duplicate file upload, a duplicate notification.

The fix is an Idempotency-Key header:

POST /v1/jobs
Idempotency-Key: my-unique-request-id-001

The middleware:

Hashes the key with the userId.
Checks PostgreSQL for a stored response for (key, userId).
If found: returns the stored response body and status — the handler never runs.
If not found: runs the handler, then caches the response asynchronously.

// Intercept res.json() to capture and cache the response
res.json = (body: unknown): Response => {
  const result = originalJson(body);
  if (res.statusCode >= 200 && res.statusCode < 500) {
    prisma.idempotencyKey
      .create({ data: { key, userId, responseStatus: res.statusCode, responseBody: body } })
      .catch(logger.error);
  }
  return result;
};

Same key, same response, zero side effects on replay. Stripe uses this exact pattern for payment safety.

File Upload Pipeline

Upload → Store → Return 202 → Process in background.

POST /v1/files (multipart/form-data)
  │── Multer buffers file in memory
  │── StorageProvider.upload() writes file to storage
  │── Prisma creates File record (status: UPLOADED)
  │── BullMQ enqueues file-processing job
  └── Returns 202 Accepted

file-processing worker:
  │── Sets status → PROCESSING
  │── Extracts metadata (mime type, dimensions, page count)
  │── Sets status → READY
  └── Dispatches file.processed webhook

Storage is abstracted behind an interface:

export interface StorageProvider {
  upload(key: string, buffer: Buffer, mimeType: string): Promise<string>;
  download(key: string): Promise<Buffer>;
  delete(key: string): Promise<void>;
}

LocalStorageProvider writes to disk today. Switching to S3: set STORAGE_PROVIDER=s3, implement S3StorageProvider. Nothing else changes.

Cross-Cutting: Auth, Rate Limiting, Versioning

API Key Authentication
Keys are stored only as SHA-256 hashes — never plaintext. On each request:

const keyHash = createHash("sha256").update(key).digest("hex");
const apiKey = await prisma.apiKey.findUnique({ where: { keyHash } });

Redis Rate Limiting
Sliding window per user/IP. Each request increments a Redis counter with a 60-second TTL. Above 100 req/min:

HTTP 429 Too Many Requests
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
Retry-After: <seconds>

API Versioning
Every response includes X-API-Version: v1 and X-Deprecated: false. Set by middleware — no individual route handler thinks about it.

Testing and CI

31 tests across 7 suites. Prisma and Redis are mocked, so tests run without live infrastructure:

PASS tests/workers.test.ts       — handler registry, job state machine
PASS tests/webhooks.test.ts      — webhook CRUD routes
PASS tests/cross-cutting.test.ts — auth, rate limiting, idempotency, versioning
PASS tests/files.test.ts         — upload, download, list, delete
PASS tests/socket.test.ts        — WebSocket server lifecycle
PASS tests/jobs.test.ts          — job CRUD and cancellation
PASS tests/health.test.ts        — liveness probe

Test Suites: 7 passed
Tests:       31 passed

CI pipeline on every push to main:

GitHub Actions
  ├── ESLint
  ├── TypeScript compile (tsc)
  ├── Jest tests (with Postgres + Redis service containers)
  ├── Docker build — API image
  ├── Docker build — Worker image
  └── Trivy vulnerability scan (both images)

Repo

cypher682/nodeflow-api

Building an IAM Service with FastAPI: Refresh Token Families, TOTP MFA, and RBAC

cypher682 — Mon, 29 Jun 2026 17:50:19 +0000

AuthCore is a production-style Identity and Access Management API built with FastAPI, PostgreSQL, Redis, Celery, Alembic, Docker, GitHub Actions, and Trivy.

The goal was not another "JWT login" tutorial project. The goal was to build a security-sensitive backend service with enough real behavior to force decisions about token lifecycle, failure handling, audit discipline, and operational evidence.

This is a writeup of the implementation — what was built, what decisions were made, and where the edge cases live.

Architecture

Client
  │
  ▼
FastAPI app (Uvicorn)
  │── PostgreSQL  — users, sessions, token families, MFA configs, RBAC, audit logs
  │── Redis       — rate limit counters, failed-login counters, lockout state
  │── Celery      — background task foundation (email delivery)
  ▼
GitHub Actions CI
  ├── Ruff lint
  ├── Black format check
  ├── Docker Compose tests (with Postgres + Redis services)
  ├── Coverage gate (80%)
  ├── Docker image build
  └── Trivy critical vulnerability scan

Auth routes: POST /api/v1/auth/register, /auth/login, /auth/refresh
MFA routes: POST /api/v1/auth/mfa/setup, /mfa/verify, /mfa/disable, /mfa/challenge/verify
Session routes: GET /api/v1/users/me/sessions, DELETE /api/v1/users/me/sessions/{session_id}
Admin routes: RBAC management, audit log queries — superuser-only

The Auth Flow

Registration:

Normalize email.
Enforce password policy (min 12 chars, lowercase, uppercase, number, symbol, rejects common weak passwords and email-username substrings).
If breach checking is enabled, check password against HaveIBeenPwned using k-anonymity (first 5 SHA-1 hash characters only — raw password never leaves the system).
Hash with bcrypt.
Create user.
Create initial session.
Issue access token + refresh token.
Write audit event.

Login:

Check Redis lockout state. If locked: 423 Locked.
Verify password. On failure: increment Redis counter, write failure audit event, return 401.
On success: clear failure counter.
If MFA is disabled: issue tokens immediately.
If MFA is enabled: return a short-lived MFA challenge token instead. Full tokens only after TOTP verification.

Refresh Token Rotation and Reuse Detection

Access tokens are short-lived. Refresh tokens live longer, which means they need stricter handling.

AuthCore tracks refresh tokens by token family. Each family has one valid token at a time — the latest issued one. The previous tokens in the family are invalidated on rotation.

The rotation flow:

Client presents a refresh token.
AuthCore verifies the token signature.
AuthCore checks that the presented token hash matches the latest hash stored for that family.
If it matches: issue a new access token and refresh token, store new hash.
If it does not match — an older refresh token was reused:
- Revoke the entire family.
- Write an audit event.
- Return 401.

The implementation detail that matters here: failure audit events that raise HTTPException are explicitly committed before raising. This matters because the FastAPI exception flow would otherwise roll back the transaction, silently losing the audit record of suspicious behavior.

# Commit the audit event before raising — do not let the rollback erase evidence
await session.commit()
raise HTTPException(status_code=401, detail="Refresh token reuse detected")

MFA: Separating Password Verification from Full Authentication

When MFA is enabled, a successful password verification is not enough to issue tokens.

Instead, login returns a short-lived MFA challenge token — not a bearer token. The client must then call:

POST /api/v1/auth/mfa/challenge/verify
Body: { "challenge_token": "...", "totp_code": "123456" }

The challenge token is signed with a short TTL and cannot be used to call any other endpoint. Only after valid TOTP verification does AuthCore issue access and refresh tokens.

This separates "password is correct" from "authentication is complete." Users who have enabled MFA cannot bypass it by replaying a login request.

The MFA setup flow:

POST /mfa/setup — creates a pending MFA config, returns the TOTP provisioning URI (for QR code display). Secret is masked in evidence screenshots.
POST /mfa/verify — validates the first TOTP code, marks MFA enabled.
POST /mfa/disable — requires a valid current TOTP code before disabling.

Sessions

AuthCore creates a session record on every register and login. Each session stores:

IP address
User agent string
Device fingerprint (derived from IP + user agent)
last_active timestamp
expires_at
Concurrent session limit with pruning of oldest sessions

Users can list their active sessions and revoke any session by ID. Session revocation writes an audit event — giving users visibility and control over active access, which is a baseline requirement for any serious IAM system.

RBAC: Roles, Permissions, and Superuser Bypass

The RBAC model:

User → UserRole → Role → RolePermission → Permission

The permission check is a FastAPI dependency injected on protected routes:

@router.get("/rbac/permission-check")
async def check_permission(
    user: User = Depends(require_permission("admin:manage")),
):
    ...

Superusers bypass named permission checks entirely. Normal users need explicit permission assignment through roles. All admin mutations write audit events.

Rate Limiting and Brute-Force Lockout

Two separate controls:

Route-level rate limiting (SlowAPI): configurable per-route limits. Exceeded: 429 Too Many Requests.

Account and IP lockout (Redis): each failed login increments a Redis counter keyed by email and IP. After the configured threshold: 423 Locked. Clears automatically after TTL.

The choice to use Redis for lockout counters (not PostgreSQL) is deliberate: these are fast-changing, ephemeral values that do not need durability.

Evidence flow: 401 on bad password → multiple 401 → 423 on lockout → 429 if the route rate limit is hit.

Password Policy and Breach Checking

Local policy: minimum 12 characters, lowercase + uppercase + digit + symbol required, rejects common weak passwords and email-username substrings.

Optional HaveIBeenPwned check (disabled by default): uses k-anonymity — only the first 5 characters of the SHA-1 hash are sent to the API. Matching suffixes are compared locally. The raw password never leaves the system.

Email Verification and Password Reset

Both flows use the same security pattern: the raw token is never stored.

Generate a random token.
Store only the SHA-256 hash in PostgreSQL, with an expiry.
Pass the raw token to the Celery email task (foundation implemented — external SMTP delivery not yet wired).
Client presents the raw token. AuthCore hashes it and compares.

For password reset: all existing refresh token families are revoked after reset, preventing an attacker with a captured refresh token from continuing to use it after a password change.

Implementation Notes Worth Calling Out

Direct bcrypt instead of Passlib: Passlib's backend probe had compatibility issues with bcrypt under Python 3.13. The implementation uses bcrypt directly — behavior is identical.

SQLAlchemy metadata column conflict: metadata is a reserved attribute on SQLAlchemy's declarative base. Fix: map the column to a different Python-side name while keeping the database column name unchanged.

event_metadata: Mapped[dict] = mapped_column("metadata", JSON, nullable=True)

Testing and CI

18 tests, 81% coverage. Docker-first — tests run inside containers with a real Postgres and Redis:

tests/test_security.py       — bcrypt, JWT, TOTP helpers
tests/test_auth.py           — register, login, refresh, reuse detection
tests/test_mfa.py            — setup, verify, disable
tests/test_sessions.py       — session listing and revocation
tests/test_rbac_admin.py     — role/permission creation and assignment
tests/test_audit.py          — audit log queries
tests/test_lockout.py        — brute-force lockout behavior
tests/test_email_flows.py    — verification and reset token flows

CI pipeline: Ruff lint → Black format check → Docker Compose tests with Alembic migrations → coverage gate (80%) → Docker image build (multi-stage, non-root runtime user) → Trivy critical vulnerability scan with SARIF upload.

Repo

https://github.com/cypher682/authcore-service

Evidence (screenshots of every flow): https://github.com/cypher682/authcore-service/tree/main/docs/evidence

Designing a Production-Grade Blue-Green ECS Platform on AWS with Terraform

cypher682 — Tue, 24 Feb 2026 15:02:49 +0000

Most AWS tutorials stop at "it works."

I wanted to build something closer to what a real engineering team would operate: network isolation, IAM least privilege, blue-green deployments, secrets management, and clean teardown—all defined as code.

This article walks through the architecture, design decisions, tradeoffs, and the 8 real issues I encountered along the way.

GitHub Repository: ecs-production-platform

Total Cost: $0.12 for complete validation (4-hour session)

What Was Built
Architecture Overview
Security Design
Blue-Green Deployment Mechanics
What Broke (8 Issues)
Production Tradeoffs
Cost Analysis

What Was Built

A production-aligned ECS Fargate platform running a Flask API backed by PostgreSQL:

Networking

Custom VPC (10.0.0.0/16) across 2 Availability Zones
Public subnets for ALB and ECS tasks
Private subnets for RDS (no internet route)

Compute

ECS Fargate services (no EC2 instance management)
Application Load Balancer with HTTPS (ACM certificate, TLS 1.3)
Blue-green target groups for zero-downtime deployments

Data

RDS PostgreSQL 15.12 (single-AZ for free tier)
Private subnet only, no public endpoint

Security

IAM role separation (execution vs task)
SSM Parameter Store for secrets (KMS encrypted)
Security group layering (internet → ALB → ECS → RDS)

Infrastructure as Code

100% Terraform (modular design)
Remote state in S3 with DynamoDB locking
Reusable modules (networking, IAM, ALB, ECS, RDS)

36 AWS resources deployed, tested, and cleanly destroyed.

Architecture Overview

┌─────────────────────────────────────────────┐
│              Internet                        │
└──────────────────┬──────────────────────────┘
                   │
         ┌─────────▼─────────┐
         │    Route 53 DNS    │
         └─────────┬──────────┘
                   │
         ┌─────────▼──────────┐
         │  ACM Certificate    │
         │    (TLS 1.3)        │
         └─────────┬───────────┘
                   │
    ┌──────────────▼──────────────┐
    │  Application Load Balancer  │
    │   (Public Subnets)          │
    └──────┬──────────────┬───────┘
           │              │
    ┌──────▼─────┐ ┌─────▼──────┐
    │ Blue TG    │ │ Green TG   │
    │ (Weight:   │ │ (Weight:   │
    │  100%)     │ │   0%)      │
    └──────┬─────┘ └─────┬──────┘
           │              │
    ┌──────▼──────────────▼──────┐
    │   ECS Fargate Services      │
    │   (Public Subnets)          │
    │   • 2 tasks (blue)          │
    │   • 0 tasks (green standby) │
    └──────────────┬──────────────┘
                   │
         ┌─────────▼──────────┐
         │  RDS PostgreSQL     │
         │  (Private Subnet)   │
         │  • No public IP     │
         │  • Port 5432 only   │
         └─────────────────────┘

Key Design Choices:

ECS in Public Subnets: Cost optimization—saves $33/month on NAT Gateway
Single-AZ RDS: Free tier constraint—production would use Multi-AZ
Security Groups: Each layer enforces isolation for the next

Terraform Module Structure

Instead of one monolithic configuration, concerns are separated into focused modules:

terraform/
├── modules/
│   ├── networking/     # VPC, subnets, security groups, DB subnet group
│   ├── iam/            # Task execution role, task role, policies
│   ├── alb/            # Load balancer, target groups, listeners
│   ├── ecs/            # Cluster, services, task definitions
│   ├── rds/            # PostgreSQL instance, parameter group
│   └── cicd/           # GitHub Actions IAM role (design only)
└── environments/
    └── prod/
        ├── main.tf         # Module composition
        ├── variables.tf    # Input variables
        ├── outputs.tf      # Stack outputs
        ├── backend.tf      # S3 + DynamoDB state
        └── versions.tf     # Provider versions

Module Communication:

Explicit inputs and outputs only
No cross-module resource references
No hidden dependencies

Example Module Call:

module "ecs" {
  source = "../../modules/ecs"

  project_name                = var.project_name
  container_image             = var.container_image
  ecs_task_execution_role_arn = module.iam.ecs_task_execution_role_arn
  ecs_task_role_arn           = module.iam.ecs_task_role_arn
  public_subnet_ids           = module.networking.public_subnet_ids
  ecs_security_group_id       = module.networking.ecs_tasks_security_group_id
  target_group_arn            = module.alb.blue_target_group_arn

  # Database connection
  db_host                     = module.rds.db_address
  db_password_ssm_param       = "/ecs-prod/db/password"
}

This keeps the architecture composable and prevents circular dependency hell.

Security Design

1. Network Isolation (Security Groups)

Traffic flows in one direction only:

Internet (0.0.0.0/0)
    ↓ Port 443/80
┌───────────────────┐
│  ALB SG           │
│  sg-0373513dd...  │
└────────┬──────────┘
         ↓ Port 8000 (from ALB SG only)
┌────────────────────┐
│  ECS Tasks SG      │
│  sg-09a3082e31...  │
└────────┬───────────┘
         ↓ Port 5432 (from ECS SG only)
┌────────────────────┐
│  RDS SG            │
│  sg-07a5aae1f9...  │
└────────────────────┘

Security Group Rules:

ALB Security Group:

ingress {
  description = "HTTPS from internet"
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
}

egress {
  description     = "To ECS tasks only"
  from_port       = 8000
  to_port         = 8000
  protocol        = "tcp"
  security_groups = [aws_security_group.ecs_tasks.id]
}

ECS Tasks Security Group:

ingress {
  description     = "From ALB only"
  from_port       = 8000
  to_port         = 8000
  protocol        = "tcp"
  security_groups = [aws_security_group.alb.id]
}

egress {
  description     = "To RDS only"
  from_port       = 5432
  to_port         = 5432
  protocol        = "tcp"
  security_groups = [aws_security_group.rds.id]
}

RDS Security Group:

ingress {
  description     = "PostgreSQL from ECS only"
  from_port       = 5432
  to_port         = 5432
  protocol        = "tcp"
  security_groups = [aws_security_group.ecs_tasks.id]
}

# No egress rules - database doesn't need outbound

Result:

No direct internet access to ECS tasks
No public database endpoint
No bypass paths

2. IAM Role Separation

Two distinct roles prevent privilege escalation:

Task Execution Role (infrastructure operations):

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability",
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
      "ssm:GetParameters"
    ],
    "Resource": "*"
  }]
}

Task Role (application runtime):

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "ssm:GetParameter",
    "Resource": "arn:aws:ssm:us-east-1:*:parameter/ecs-prod/*"
  }]
}

Why This Matters:

If a container is compromised, the attacker inherits only the task role—not the execution role. They can't:

Pull arbitrary images from ECR
Write to CloudWatch logs outside their stream
Access SSM parameters outside /ecs-prod/*

3. Secrets Management Flow

# 1. Generate password
PASSWORD=$(openssl rand -base64 32 | tr -d "=+/" | cut -c1-25)

# 2. Store in SSM (KMS encrypted)
aws ssm put-parameter \
  --name "/ecs-prod/db/password" \
  --value "$PASSWORD" \
  --type "SecureString"

# 3. Reference in task definition
{
  "name": "DB_PASSWORD_SSM_PARAM",
  "value": "/ecs-prod/db/password"
}

# 4. Application fetches at runtime
import boto3
ssm = boto3.client('ssm')
password = ssm.get_parameter(
    Name=os.environ['DB_PASSWORD_SSM_PARAM'],
    WithDecryption=True
)['Parameter']['Value']

Never in:

Source control
Docker image
Environment variables (plaintext)
Terraform state (marked sensitive)

Blue-Green Deployment Mechanics

The ALB HTTPS listener can route traffic to two separate target groups with configurable weights.

Initial State

Blue Target Group:  ████████████████████ 100% (2 healthy tasks)
Green Target Group: -------------------- 0%  (0 tasks)

Deployment Process

Step 1: Build New Version

# Update app version in app.py
# Build Docker image
docker build -t 758620460011.dkr.ecr.us-east-1.amazonaws.com/ecs-prod/flask-app:v2 .

# Push to ECR
docker push 758620460011.dkr.ecr.us-east-1.amazonaws.com/ecs-prod/flask-app:v2

Step 2: Deploy to Green

# Scale up green service
aws ecs update-service \
  --cluster ecs-prod-cluster \
  --service ecs-prod-service-green \
  --desired-count 2

# Wait for health checks (90 seconds)
aws ecs wait services-stable \
  --cluster ecs-prod-cluster \
  --services ecs-prod-service-green

Step 3: Switch Traffic (< 1 Second)

# Get listener and target group ARNs
LISTENER_ARN=$(terraform output -raw alb_listener_arn)
GREEN_TG=$(terraform output -raw green_target_group_arn)

# Instant traffic switch
aws elbv2 modify-listener \
  --listener-arn $LISTENER_ARN \
  --default-actions Type=forward,TargetGroupArn=$GREEN_TG

Step 4: Validate and Cleanup

# Monitor green deployment
curl https://app.cipherpol.xyz/health
# {"version":"2.0.0","deployment":"green","status":"healthy"}

# After 15 minutes of monitoring, scale down blue
aws ecs update-service \
  --cluster ecs-prod-cluster \
  --service ecs-prod-service \
  --desired-count 0

Why This Works

No DNS propagation delays — Traffic switches at ALB layer

No container restarts — Only listener weight changes

Instant rollback — Reverse the listener modification

No downtime — ALB handles connection draining

Rollback Command (Same as Deploy):

# Get blue target group
BLUE_TG=$(terraform output -raw blue_target_group_arn)

# Instant rollback
aws elbv2 modify-listener \
  --listener-arn $LISTENER_ARN \
  --default-actions Type=forward,TargetGroupArn=$BLUE_TG

Data Persistence Validation

Both blue and green services connect to the same RDS instance. I validated this explicitly:

# 1. Create items while blue is active
curl -X POST https://app.cipherpol.xyz/items \
  -H "Content-Type: application/json" \
  -d '{"name":"Test Item 1"}'

curl -X POST https://app.cipherpol.xyz/items \
  -H "Content-Type: application/json" \
  -d '{"name":"Test Item 2"}'

# 2. Switch traffic to green
aws elbv2 modify-listener --listener-arn $LISTENER_ARN --default-actions Type=forward,TargetGroupArn=$GREEN_TG

# 3. Verify data persists
curl https://app.cipherpol.xyz/items | jq
{
  "count": 2,
  "items": [
    {"id": 2, "name": "Test Item 2", "created_at": "2026-02-19T10:54:03"},
    {"id": 1, "name": "Test Item 1", "created_at": "2026-02-19T10:53:02"}
  ]
}

# 4. Create new item on green
curl -X POST https://app.cipherpol.xyz/items \
  -H "Content-Type: application/json" \
  -d '{"name":"Created on Green v2.0"}'

# 5. Rollback to blue
aws elbv2 modify-listener --listener-arn $LISTENER_ARN --default-actions Type=forward,TargetGroupArn=$BLUE_TG

# 6. Confirm all items still present
curl https://app.cipherpol.xyz/items | jq '.count'
# 3

Result:

5 items persisted across 3 deployment cycles

The deployment layer is stateless

The database is the single source of truth

What Broke (And What I Learned)

Issue 1: RDS Connection Delay After "Available" Status

Problem: ECS tasks failed health checks immediately after terraform apply completed RDS creation.

Symptoms:

ecs-prod-service: unhealthy targets: 2/2
Task stopped reason: Task failed container health checks

Root Cause:

RDS reports available status when the instance is running, but doesn't accept connections for another 60-90 seconds while:

Background processes initialize
Performance schema loads
Cache warms up

Fix:

Health check grace period (60s) absorbed the delay:

resource "aws_ecs_service" "app" {
  health_check_grace_period_seconds = 60
  # Tasks retry connection until RDS is ready
}

Lesson: AWS resource statuses don't always mean "ready for traffic." Plan for initialization time.

Issue 2: Docker HEALTHCHECK Missing Dependency

Problem: 100+ ECS task restarts. ALB target health showed healthy, but ECS kept replacing tasks.

Symptoms:

# ALB target group
Target health: healthy (2/2)

# ECS service events
Unhealthy container: flask-app
Task stopped, starting replacement

Root Cause:

# Original (broken)
HEALTHCHECK CMD curl -f http://localhost:8000/health || exit 1

But curl wasn't in python:3.11-slim base image.

Why ALB Passed but Container Failed:

ALB health check: HTTP request from outside the container (port 8000)
Container health check: Command executed inside the container

Fix:

# Use Python instead of curl
HEALTHCHECK CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2)"

Lesson:

ALB health checks and container health checks are independent control loops:

ALB health: Determines if task receives traffic
Container health: Determines if ECS replaces the task

ECS uses container health, not ALB health.

Issue 3: Git Bash Path Conversion on Windows

Problem: SSM parameter /ecs-prod/db/password became C:\Program Files\Git\ecs-prod\db\password

Error:

ParameterNotFound: /C:/Program Files/Git/ecs-prod/db/password

Root Cause: Git Bash on Windows auto-converts Unix-style paths to Windows paths.

Fix:

# Disable path conversion
export MSYS_NO_PATHCONV=1

# Then run AWS CLI
aws ssm get-parameter --name /ecs-prod/db/password --with-decryption

Alternative: Use Windows Command Prompt or PowerShell for AWS CLI commands.

Lesson: Git Bash is great for Unix tools, but AWS CLI needs special handling on Windows.

Issue 4: ALB Listener Syntax Constraint

Problem: After configuring blue-green with weighted routing, subsequent updates using simple syntax failed.

Error:

An error occurred (ValidationError): Cannot use both TargetGroupArn and ForwardConfig in the same action

Root Cause: Once you use ForwardConfig (weighted routing), the ALB API remembers this and requires full JSON syntax for all future updates.

Simple syntax (stopped working):

--default-actions Type=forward,TargetGroupArn=arn:aws:...

Required syntax:

--default-actions '[{
  "Type": "forward",
  "ForwardConfig": {
    "TargetGroups": [{
      "TargetGroupArn": "arn:aws:...",
      "Weight": 100
    }]
  }
}]'

Lesson: ALB API is stateful. Once you use advanced features, you can't revert to simple syntax. Document this in runbooks.

Issue 5: PostgreSQL Minor Version Retirement

Problem: Terraform apply failed with:

InvalidParameterValue: Cannot find version 15.4 for postgres

Root Cause: AWS retired PostgreSQL 15.4 in favor of 15.12 (latest patch version).

Fix:

# Before
engine_version = "15.4"

# After
engine_version = "15.12"

Better Fix (Production):

# Pin major version, allow minor updates
engine_version = "15"

Lesson: AWS manages minor version lifecycle. Pin major versions intentionally, but expect patch version changes.

Issue 6: Terraform State Lock Timeout

Problem: terraform plan hung for 5 minutes, then failed:

Error acquiring state lock: timeout waiting for lock

Root Cause: DynamoDB lock table had wrong key schema:

# Wrong
AttributeName=id,KeyType=HASH

# Correct
AttributeName=LockID,KeyType=HASH

Fix:

# Delete wrong table
aws dynamodb delete-table --table-name terraform-state-lock

# Recreate with correct schema
aws dynamodb create-table \
  --table-name terraform-state-lock \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

Lesson: Terraform's DynamoDB lock table requires exactly LockID as the partition key. Case-sensitive.

Issue 7: ECR Image Pull Failures (Intermittent)

Problem: Some task launches failed with:

CannotPullContainerError: API error: manifest unknown

Root Cause: Task execution role was missing ecr:BatchGetImage permission.

Fix: Attached AWS managed policy:

resource "aws_iam_role_policy_attachment" "ecs_task_execution" {
  role       = aws_iam_role.ecs_task_execution.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

Lesson: Custom IAM policies are error-prone. Use AWS managed policies where possible, then restrict with conditions if needed.

Issue 8: Terraform vs Manual Scaling Conflict

Problem: Terraform tried to update blue service's desired_count while I was manually scaling for testing:

Error: concurrent modification detected
Service is being modified by another operation

Fix: Added lifecycle rule to ignore runtime changes:

resource "aws_ecs_service" "app" {
  # ... other config

  lifecycle {
    ignore_changes = [desired_count]
  }
}

Lesson: When testing blue-green manually, let Terraform manage infrastructure but ignore runtime scaling changes. Use ignore_changes selectively.

What I'd Change in Production

This Project	Production Standard	Reason	Cost Impact
ECS in public subnets	Private subnets + NAT Gateway	Defense-in-depth, reduced attack surface	+$33/month
Single-AZ RDS	Multi-AZ RDS	99.95% SLA vs 99.5%, automatic failover	+$15/month
Manual ALB switch	AWS CodeDeploy blue-green	Automated rollback based on CloudWatch alarms	$0 (free)
No autoscaling	ECS Service Auto Scaling	Handle traffic spikes, reduce idle costs	Variable
SSM Parameter Store	AWS Secrets Manager	Automatic rotation, better audit	+$0.40/secret
No read replicas	RDS read replica	Offload read traffic from primary	+$15/month
7-day log retention	30-90 day retention	Compliance, longer incident investigation	+$2/month

Total Production Cost: ~$80-100/month

This Project Cost: $0.12 for validation

Cost Savings Breakdown:

NAT Gateway: $33 saved
Multi-AZ RDS: $15 saved
Secrets Manager: $0.40 saved
Read replica: $15 saved
Total: $63.40/month saved

Cost Breakdown (Actual)

4-Hour Validation Session

Service	Hourly Rate	Hours	Cost
ALB	$0.0225	3	$0.0675
ECS Fargate (2 tasks × 0.25 vCPU)	Free tier	3	$0
RDS db.t3.micro (single-AZ)	Free tier	7	$0
Route 53 Hosted Zone	$0.50/month prorated	3 days	$0.05
S3 State Bucket	<$0.01	—	$0
CloudWatch Logs	Free tier (5GB)	—	$0
Data Transfer	Free tier (100GB)	<1GB	$0
Total			$0.12

Monthly Cost If Kept Running

Service	Monthly Cost	Notes
ALB	$16.20	720 hours × $0.0225
ECS Fargate	$0	Within 400 vCPU-hour free tier
RDS db.t3.micro	$0	Within 750-hour free tier
Route 53	$0.50	Hosted zone
Storage	<$1	S3, CloudWatch
Total	~$17/month

Comparison to EKS:

EKS control plane: $72/month
ECS Fargate control plane: $0
Savings: $72/month for equivalent compute

Key Takeaways

What Worked Well

Modular Terraform — Each module had single responsibility, debugging was easier

Blue-green switching — True zero downtime, instant rollback capability

Security group layering — Network isolation without complexity

SSM secrets — No credentials in code, images, or state files

Free tier optimization — Validated production patterns for $0.12

Documentation — 8 real issues documented with root causes and fixes

What I'd Improve

Terragrunt for DRY — Multi-environment deployments without duplication

Automated testing — Pre-deployment health checks in CI/CD

CodeDeploy integration — Production should automate blue-green

Observability — CloudWatch dashboards for latency, errors, saturation

Database migrations — Flyway or Liquibase for schema versioning

Chaos engineering — Terminate random tasks to test resilience

Repository & Evidence

Full source code with detailed documentation:

GitHub: cypher682/ecs-production-platform

What's included:

Complete Terraform modules (networking, IAM, ALB, ECS, RDS)
Flask application with Dockerfile and health checks
GitHub Actions workflow (OIDC design, not tested live)
Operational runbooks (deployment failure, database connection, rollback)
Lessons learned documentation (8 issues with root cause analysis)
Cost analysis (actual vs production projection)
Evidence files (Terraform outputs, CloudWatch logs, test results)

Documentation structure:

docs/
├── ARCHITECTURE.md            # Design decisions and network diagrams
├── 01_IMPLEMENTATION.md       # Phase-by-phase build log
├── 02_LESSONS_LEARNED.md      # 8 issues with fixes
├── 03_COST_ANALYSIS.md        # Detailed cost breakdown
├── 04_SECURITY.md             # IAM policies, secrets flow
├── 05_CICD_DESIGN.md          # GitHub Actions workflow design
└── runbooks/
    ├── deployment-failure.md  # What to do when deploy fails
    ├── database-connection.md # Troubleshooting RDS connectivity
    └── rollback-procedure.md  # Step-by-step rollback guide

Questions or Feedback?

If you're building something similar or have questions about:

Blue-green deployment patterns
IAM least privilege design
AWS cost optimization strategies
Terraform module architecture
Debugging ECS task failures

Drop a comment below! I'll respond with specific examples from this build.

This project was built as a portfolio sprint to demonstrate production-ready AWS skills. The platform was deployed, validated with 5 CRUD operations, and destroyed within 4 hours—total cost $0.12. All code and documentation available on GitHub.

#AWS #Terraform #DevOps #ECS #InfrastructureAsCode #BlueGreenDeployment #CloudEngineering #Docker #PostgreSQL

Building a Production-Grade AWS Cost & Security Auditor

cypher682 — Thu, 12 Feb 2026 08:36:28 +0000

Cloud environments naturally drift. Costs creep up. Security posture degrades. Manual audits do not scale, and periodic reviews miss issues that emerge between checks.

I needed an auditing tool that could:

Identify cost waste — idle EC2 instances, unattached EBS volumes, orphaned snapshots
Detect security misconfigurations — public S3 buckets, overly permissive security groups, weak IAM hygiene
Map findings to a known framework — CIS AWS Foundations Benchmark
Operate safely — strictly read-only, no automated deletion or remediation

This article walks through the key design decisions, trade-offs, and lessons learned from building it.

Core Design Principles

1. Read-Only by Design

Decision: No auto-remediation. No destructive permissions.

Automatically deleting or modifying cloud resources is risky, especially in production. An instance that appears idle may be a disaster-recovery standby, a scheduled batch worker, or part of a failover strategy.

The tool’s role is to surface risk and waste, not to make irreversible decisions.

IAM policy scope:

{
  "Effect": "Allow",
  "Action": [
    "ec2:Describe*",
    "s3:Get*",
    "s3:List*",
    "iam:List*",
    "iam:Get*"
  ],
  "Resource": "*"
}

No Delete, Terminate, or Modify permissions. The blast radius is limited to discovery only.

This constraint shaped the entire architecture and made the tool safe to run against live accounts.

2. Defining “Idle” Using CloudWatch Metrics

Problem: “Idle” is ambiguous in cloud systems.

CPU utilization is an imperfect signal, but it is widely available and easy to reason about. I defined idle EC2 instances as those with:

Average CPU utilization < 5%
Observed over a 7-day window

Implementation:

def get_cpu_utilization(self, instance_id: str, days: int = 7) -> float:
    end_time = datetime.utcnow()
    start_time = end_time - timedelta(days=days)

    response = self.cloudwatch.get_metric_statistics(
        Namespace='AWS/EC2',
        MetricName='CPUUtilization',
        Dimensions=[{'Name': 'InstanceId', 'Value': instance_id}],
        StartTime=start_time,
        EndTime=end_time,
        Period=86400,  # 1 day
        Statistics=['Average']
    )

    if response['Datapoints']:
        return sum(dp['Average'] for dp in response['Datapoints']) / len(response['Datapoints'])
    return -1.0

Trade-offs:

Misses bursty or scheduled workloads (batch jobs, ML training)
Flags instances that are intentionally dormant

Rather than hiding these limitations, they are explicitly documented. Transparency builds trust in tooling.

3. Aligning Findings with CIS Benchmarks

Raw findings are less useful without context. Mapping issues to the CIS AWS Foundations Benchmark adds structure and credibility.

Example:

findings.append({
    'bucket_name': bucket_name,
    'issue': 'Public access enabled',
    'severity': 'HIGH',
    'cis_control': '2.1.5',
    'remediation': 'Enable S3 Block Public Access'
})

This approach:

Makes findings actionable
Aligns with how security teams think
Signals familiarity with compliance-driven environments

Results from a Real AWS Account

Running the auditor against my own AWS account produced the following:

SECURITY FINDINGS: 32
  CRITICAL: 11  (SSH/RDP open to 0.0.0.0/0)
  HIGH:      9  (IAM users without MFA, public S3 buckets)
  MEDIUM:   12  (stale access keys, permissive policies)

Even a relatively small account accumulated meaningful security drift. Continuous auditing is not optional at scale.

Handling False Positives

One flagged issue was my own AuditorToolReadOnly IAM policy using:

"Resource": "*"

At first glance, this looks overly permissive. In practice, it is required. Read-only IAM and EC2 discovery APIs (List*, Describe*) cannot be scoped to specific ARNs that are not yet known.

Key point:

Not all flagged issues are actionable
False positives should be documented, not ignored

This distinction is critical in real operational environments.

What I Would Improve for a Production Deployment

If this were moving beyond a portfolio project:

Multi-region scanning instead of single-region execution
Historical persistence using DynamoDB for trend analysis
AWS Cost Explorer integration for real billing data
Alerting via Slack or SNS for critical findings
IAM Access Analyzer integration for deeper policy analysis

The current scope balances realism with complexity without overengineering.

Key Takeaways

Read-only audits reduce risk and build trust when running against live environments
Cost and security signals are more useful when tied to metrics and known frameworks
Not every finding should be auto-remediated; judgment still matters

These principles guided the design choices throughout this project.

Try It Yourself

Repository:
https://github.com/cypher682/aws-cost-security-auditor

Run locally:

git clone https://github.com/cypher682/aws-cost-security-auditor
cd aws-cost-security-auditor
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
python src/full_audit.py --profile auditor-role

See the remediation guidance in docs/REMEDIATION_PLAYBOOK.md.

What’s Next

Planned extensions:

Multi-account support (AWS Organizations)
RDS idle detection
Lambda cost analysis

Links:

X: https://twitter.com/cypher682
LinkedIn: https://linkedin.com/in/suleiman-abdulrahman-dev

This project is part of my portfolio focused on production-grade cloud platform engineering.

Building a Production-Ready Microservices Platform with CI/CD on AWS Free Tier

cypher682 — Thu, 11 Dec 2025 14:37:20 +0000

Introduction

Building a complete microservices architecture with professional DevOps practices can be intimidating. This guide walks you through creating a production-grade system using AWS free tier, demonstrating real-world patterns that mid-level engineers can apply immediately.

We'll build a polyglot microservices platform with:

Three microservices in different languages (Node.js, Python, Go)
Complete CI/CD pipeline with automated testing and security scanning
Infrastructure as Code with Terraform
Monitoring with Prometheus and Grafana
All running on AWS free tier for under $2/month

Architecture Overview

Our platform consists of three microservices behind an API Gateway, all running on a single EC2 instance with Docker Compose. While this isn't the scalability of Kubernetes, it demonstrates core concepts without the complexity or cost.

The API Gateway handles routing and authentication, forwarding requests to the User Service (Python/FastAPI) for user operations and Product Service (Go/Gin) for product management. Both services interact with separate DynamoDB tables.

Why This Stack?

Polyglot Architecture: Using multiple languages demonstrates how microservices enable technology diversity. Each service uses the best tool for its job.

Docker Compose on EC2: Kubernetes is powerful but complex. Docker Compose provides orchestration sufficient for small-medium deployments while remaining in free tier.

DynamoDB: Pay-per-request pricing with generous free tier (25GB, 25 RCU/WCU) makes it ideal for demos that might sit idle.

GitHub Actions: Native GitHub integration means no external CI/CD service needed.

Infrastructure Setup

We use Terraform to create all AWS resources. The infrastructure includes:

VPC with public subnet
EC2 t2.micro instance
DynamoDB tables for users and products
Route53 hosted zone
IAM roles with least-privilege policies
Security groups allowing only necessary ports

The EC2 instance runs Docker and is configured via user data script to install Docker Engine during launch.

resource "aws_instance" "app" {
  ami                    = data.aws_ami.ubuntu.id
  instance_type          = "t2.micro"
  vpc_security_group_ids = [aws_security_group.ec2.id]
  iam_instance_profile   = aws_iam_instance_profile.ec2.name

  user_data = <<-EOF
              #!/bin/bash
              apt-get update
              curl -fsSL https://get.docker.com | sh
              usermod -aG docker ubuntu
              EOF
}

DynamoDB tables use on-demand billing to stay within free tier:

resource "aws_dynamodb_table" "users" {
  name         = "aws-microservices-cicd-users"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "userId"

  attribute {
    name = "userId"
    type = "S"
  }
}

Microservices Implementation

API Gateway (Node.js)

The API Gateway handles authentication via API keys stored in AWS Systems Manager Parameter Store and routes requests to appropriate services.

const authMiddleware = (req, res, next) => {
  const apiKey = req.headers['x-api-key'];
  if (!apiKey || apiKey !== API_KEY) {
    return res.status(401).json({ error: 'Unauthorized' });
  }
  next();
};

app.use('/users', authMiddleware, async (req, res) => {
  const response = await axios({
    method: req.method,
    url: `${USER_SERVICE_URL}${req.path}`,
    data: req.body
  });
  res.status(response.status).json(response.data);
});

User Service (Python)

FastAPI provides automatic API documentation and Pydantic validation:

@app.post("/", response_model=UserResponse, status_code=201)
async def create_user(user: User):
    user_id = str(uuid.uuid4())
    timestamp = datetime.utcnow().isoformat()

    table.put_item(
        Item={
            'userId': user_id,
            'email': user.email,
            'name': user.name,
            'createdAt': timestamp
        }
    )
    return item

Product Service (Go)

Go's performance makes it ideal for high-throughput services:

func createProduct(c *gin.Context) {
    var product Product
    c.ShouldBindJSON(&product)

    product.ProductID = uuid.New().String()
    product.CreatedAt = time.Now().UTC().Format(time.RFC3339)

    av, _ := dynamodbattribute.MarshalMap(product)
    db.PutItem(&dynamodb.PutItemInput{
        TableName: aws.String(tableName),
        Item: av,
    })

    c.JSON(201, product)
}

CI/CD Pipeline

The GitHub Actions pipeline has two workflows:

CI Pipeline (on pull requests):

Lint code for all three services
Run unit tests
Build Docker images
Scan containers with Trivy for vulnerabilities
Upload security findings to GitHub Security

CD Pipeline (on merge to main):

Build Docker images for all services
Push to Docker Hub with latest and SHA tags
SSH to EC2 instance
Pull new images
Restart containers with docker-compose
Run health checks
Rollback on failure

- name: Deploy to EC2
  run: |
    ssh ubuntu@${{ steps.get-ip.outputs.instance_ip }} << 'EOF'
      cd /home/ubuntu/app
      docker-compose pull
      docker-compose up -d
    EOF

Security Implementation

Container Scanning: Trivy scans images for known vulnerabilities before deployment. Critical and high severity findings block the pipeline.

Secrets Management: API keys and sensitive data live in AWS Systems Manager Parameter Store, not in environment variables or code.

SSL/TLS: Let's Encrypt provides free SSL certificates, configured via Ansible during initial setup.

IAM Policies: EC2 instance has minimal permissions, only accessing specific DynamoDB tables and Parameter Store paths.

resource "aws_iam_role_policy" "dynamodb" {
  policy = jsonencode({
    Statement = [{
      Effect = "Allow"
      Action = [
        "dynamodb:PutItem",
        "dynamodb:GetItem",
        "dynamodb:Query"
      ]
      Resource = [
        aws_dynamodb_table.users.arn,
        aws_dynamodb_table.products.arn
      ]
    }]
  })
}

Monitoring Stack

Prometheus scrapes metrics from all three services every 15 seconds. Each service exposes a /metrics endpoint with custom business metrics plus standard HTTP metrics.

Grafana visualizes the data with dashboards showing:

Request rates and latencies per service
Error rates (4xx, 5xx responses)
Container resource usage (CPU, memory)
DynamoDB operation metrics
Custom business metrics (user registrations, product creates)

The monitoring stack runs alongside the application services in Docker Compose:

prometheus:
  image: prom/prometheus:latest
  volumes:
    - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
  ports:
    - "9090:9090"

grafana:
  image: grafana/grafana:latest
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=admin
  ports:
    - "3001:3000"

Configuration Management with Ansible

Ansible handles EC2 setup tasks that are complex or stateful:

Installing Docker and Docker Compose
Configuring UFW firewall
Setting up Nginx with SSL
Obtaining Let's Encrypt certificates
Creating application directories
Installing Prometheus Node Exporter

The setup playbook runs once after Terraform creates the infrastructure:

- name: Install Docker
  apt:
    name:
      - docker-ce
      - docker-ce-cli
      - containerd.io
    state: present

- name: Obtain SSL certificate
  command: >
    certbot --nginx -d api.cipherpol.xyz
    --non-interactive --agree-tos

Cost Optimization

This architecture costs $0.50-2.00/month:

Route53: $0.50/month for hosted zone (only unavoidable cost)

EC2 t2.micro: Free tier provides 750 hours/month, enough for one instance running 24/7

DynamoDB: Free tier includes 25GB storage and 25 RCU/WCU, sufficient for development and small production workloads

Data Transfer: First 1GB/month free, typically sufficient for API traffic

S3: Used for Terraform state, negligible cost

The key is using on-demand pricing for DynamoDB (no provisioned capacity charges) and staying within EC2 free tier limits.

Lessons Learned

Keep it Simple: Docker Compose on one instance is simpler than ECS/EKS and adequate for many workloads.

Security First: Even demo projects should implement proper security. Trivy scanning and proper IAM policies cost nothing but prevent vulnerabilities.

Monitoring Matters: Prometheus and Grafana add minimal overhead but provide invaluable visibility into system behavior.

Automate Everything: Manual deployments are error-prone. GitHub Actions makes automation straightforward.

Cost-Aware Design: Understanding free tier limits enables building production-quality systems for minimal cost.

Real Challenges Faced:

Docker build failures due to deprecated npm flags and missing dependency files
IAM permission mismatches between parameter paths and policies
Port conflicts between services requiring careful docker-compose configuration
Nginx SSL configuration needing manual proxy_pass setup after certbot
Multi-stage Docker builds requiring proper user permission handling

Trade-offs and Limitations

Single Point of Failure: One EC2 instance means no high availability. For production, consider auto-scaling groups.

Limited Scalability: Docker Compose doesn't provide automatic scaling. When traffic grows, consider ECS or EKS.

Shared Resources: All services share EC2 resources. A memory leak in one service affects others.

Manual SSL Renewal: While automated via cron, this isn't as robust as AWS Certificate Manager.

Real-World Applications

This architecture pattern works well for:

Internal tools and admin dashboards
API backends for mobile apps
Prototypes and MVPs
Side projects and personal applications
Learning and portfolio projects

It's not suitable for:

High-traffic consumer applications
Systems requiring 99.99% uptime
Workloads with unpredictable traffic spikes
Applications requiring compliance certifications

Next Steps

To evolve this architecture:

Add Authentication Service: Implement OAuth2/JWT for user authentication
Implement Caching: Add Redis for frequently accessed data
Queue System: Use SQS for asynchronous processing
API Gateway Replacement: Consider AWS API Gateway for rate limiting and caching
Multi-Region: Deploy in multiple regions for disaster recovery
Kubernetes Migration: When scaling requirements justify complexity

Conclusion

Building a complete microservices platform doesn't require expensive infrastructure or complex orchestration. This project demonstrates that professional DevOps practices are accessible and affordable.

The skills developed here—Infrastructure as Code, CI/CD pipelines, container orchestration, and monitoring—translate directly to enterprise environments. The architecture patterns scale from side projects to production systems.

Most importantly, this hands-on experience with real tools and services is more valuable than theoretical knowledge. You now have a working platform to experiment with, break, fix, and improve.

The complete source code is available on GitHub, along with detailed setup instructions and troubleshooting guides. Fork it, modify it, and make it your own.

Resources

GitHub Repository: aws-microservices-cicd
AWS Free Tier: https://aws.amazon.com/free
Docker Compose Documentation: https://docs.docker.com/compose
Terraform AWS Provider: https://registry.terraform.io/providers/hashicorp/aws
Prometheus Monitoring: https://prometheus.io/docs
GitHub Actions: https://docs.github.com/actions

Complete Beginner's Guide to Blue-Green Deployment with Nginx and Real-Time Alerting

cypher682 — Tue, 09 Dec 2025 14:49:42 +0000

Introduction

Welcome to this comprehensive guide on Blue-Green Deployment - a powerful deployment strategy used by companies like Netflix, Amazon, and Facebook to achieve zero-downtime deployments. This project demonstrates how to implement a production-ready blue-green deployment system with automatic failover and real-time Slack alerting.

Repository: HNG DevOps on GitHub

What You'll Learn:

What blue-green deployment is and why it matters
How to implement automatic failover with Nginx
How to build a real-time monitoring and alerting system
How to achieve zero-downtime deployments
How to integrate Slack notifications for DevOps alerts

Prerequisites:

Basic understanding of Docker
Familiarity with command line
Basic knowledge of web servers (helpful but not required)

What is Blue-Green Deployment?

The Problem: Traditional Deployments

Imagine you're running a website. When you deploy a new version:

You stop the old version
Deploy the new version
Start the new version

Problem: During steps 1-3, your website is DOWN. Users see errors. You lose money.

The Solution: Blue-Green Deployment

Instead of having one environment, you have TWO identical environments:

BLUE (Production) - Currently serving users
GREEN (Staging) - New version waiting to go live

When you're ready to deploy:

Deploy new version to GREEN
Test GREEN thoroughly
Switch traffic from BLUE to GREEN instantly
If something goes wrong, switch back to BLUE instantly

Result: ZERO DOWNTIME

Real-World Analogy

Think of it like having two stages at a concert:

Stage 1 (Blue): Band is performing, audience is watching
Stage 2 (Green): Next band is setting up and sound-checking

When it's time to switch:

Rotate the stage 180°
Audience now sees Stage 2 (Green)
Stage 1 (Blue) becomes the setup area for the next act

If the new band has technical issues, rotate back to Stage 1 instantly!

Project Overview

This project implements a production-ready blue-green deployment system with:

Core Features

Automatic Failover
- Nginx detects when Blue instance fails
- Automatically routes all traffic to Green
- Zero failed requests to users
Real-Time Alerting
- Python watcher monitors Nginx logs
- Detects failover events instantly
- Sends alerts to Slack
- Monitors error rates
Zero-Downtime Deployment
- Deploy new version to inactive instance
- Switch traffic instantly
- Rollback in seconds if needed
Structured Logging
- Every request logged with metadata
- Pool information (blue/green)
- Release version
- Response times

Understanding the Core Concepts

1. Blue-Green vs. Load Balancing

Load Balancing:

Request 1 → Blue
Request 2 → Green
Request 3 → Blue
Request 4 → Green

Traffic is distributed between instances.

Blue-Green (This Project):

All Requests → Blue (Primary)
              Green (Backup, standby)

If Blue fails:
All Requests → Green (Backup becomes active)

Traffic goes to ONE instance at a time. The other is a hot standby.

2. Nginx Upstream Configuration

Nginx can route traffic to multiple backend servers (upstreams). This project uses:

upstream app {
    server app_blue:3000 max_fails=1 fail_timeout=5s;
    server app_green:3000 backup max_fails=1 fail_timeout=5s;
}

Key Directives:

server app_blue:3000 - Primary server
backup - Only use if primary fails
max_fails=1 - Mark as failed after 1 error
fail_timeout=5s - Try again after 5 seconds

3. Failover Mechanism

When a request fails:

Nginx tries Blue instance
Blue returns 5xx error or times out
Nginx marks Blue as failed
Nginx retries request to Green (backup)
User receives successful response from Green
All subsequent requests go to Green

Result: User never sees an error!

Architecture Overview

System Architecture Diagram

                Internet
                   |
                   v
            +--------------+
            |    Nginx     |
            |  (Port 8080) |
            +--------------+
                   |
    +--------------+--------------+
    |                             |
    v                             v
+----------+                +----------+
| App Blue | (Primary)      |App Green | (Backup)
|Port 3000 |                |Port 3000 |
+----------+                +----------+
    |                             |
    +-------------+---------------+
                  |
                  v
          +--------------+
          | Nginx Logs   |
          +--------------+
                  |
                  v
        +------------------+
        | Alert Watcher    |
        | (Python)         |
        +------------------+
                  |
                  v
          +--------------+
          |    Slack     |
          +--------------+

Component Breakdown

1. Nginx Proxy

Role: Traffic router and load balancer

Responsibilities:

Route all incoming requests
Detect backend failures
Perform automatic failover
Log all requests with metadata

2. App Blue (Primary Instance)

Role: Primary application server

Environment Variables:

APP_POOL=blue
RELEASE_ID=blue-release-1.0.0
PORT=3000

3. App Green (Backup Instance)

Role: Backup application server (hot standby)

Environment Variables:

APP_POOL=green
RELEASE_ID=green-release-1.0.0
PORT=3000

4. Alert Watcher (Python)

Role: Real-time log monitoring and alerting

Responsibilities:

Tail Nginx access logs
Parse structured log entries
Detect failover events
Monitor error rates
Send Slack alerts

Technology Stack

Component	Technology	Purpose
Reverse Proxy	Nginx (Alpine)	Traffic routing & failover
Application	Python/Flask	Demo web application
Monitoring	Python 3.11	Log watcher & alerting
Alerting	Slack Webhooks	Real-time notifications
Containerization	Docker	Package all services
Orchestration	Docker Compose	Manage multi-container setup

Setting Up the Project

Step 1: Clone the Repository

git clone https://github.com/cypher682/hng13-stage-3-devops.git
cd hng13-stage-3-devops

Step 2: Understand the Project Structure

hng13-stage-3-devops/
├── docker-compose.yml       # Container orchestration
├── nginx.conf.template      # Nginx configuration template
├── entrypoint.sh           # Nginx startup script
├── watcher.py              # Python log monitoring script
├── requirements.txt        # Python dependencies
├── .env.example            # Environment variables template
├── test-failover.sh        # Failover testing script
└── public/                 # Static HTML files

Step 3: Configure Environment Variables

Copy the example environment file:

cp .env.example .env

Edit .env with your configuration:

# Application Configuration
PORT=3000
ACTIVE_POOL=blue

# Docker Images
BLUE_IMAGE=yimikaade/wonderful:latest
GREEN_IMAGE=yimikaade/wonderful:latest

# Release Identifiers
RELEASE_ID_BLUE=blue-release-1.0.0
RELEASE_ID_GREEN=green-release-1.0.0

# Slack Integration
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL

# Alert Configuration
ERROR_RATE_THRESHOLD=2          # Percentage
WINDOW_SIZE=200                 # Number of requests
ALERT_COOLDOWN_SEC=300          # 5 minutes

Step 4: Set Up Slack Webhook

Go to Slack API
Create a new Slack App
Enable Incoming Webhooks
Create a webhook for your channel
Copy the webhook URL to .env

Step 5: Build and Start Services

# Build all containers
docker compose build

# Start all services in detached mode
docker compose up -d

# Verify all services are running
docker compose ps

Expected Output:

NAME              STATUS    PORTS
nginx_proxy       Up        0.0.0.0:8080->80/tcp
app_blue          Up        3000/tcp
app_green         Up        3000/tcp
alert_watcher     Up

Step 6: Verify the Deployment

Open your browser and navigate to:

http://localhost:8080/version

You should see:

{
  "pool": "blue",
  "release": "blue-release-1.0.0",
  "status": "healthy"
}

Understanding the Configuration

Nginx Configuration Template

The nginx.conf.template uses environment variable substitution:

upstream app {
    server app_blue:${PORT} max_fails=1 fail_timeout=5s;
    server app_green:${PORT} backup max_fails=1 fail_timeout=5s;
}

server {
    listen 80;

    location / {
        proxy_pass http://app;
        proxy_next_upstream error timeout http_502 http_503 http_504;

        # Timeouts for fast failure detection
        proxy_connect_timeout 2s;
        proxy_send_timeout 3s;
        proxy_read_timeout 3s;
    }
}

Key Configuration Explained

1. Proxy Next Upstream

proxy_next_upstream error timeout http_502 http_503 http_504;

What it does:
Automatically retry the request to the next upstream (Green) if:

Connection error occurs
Request times out
Upstream returns 502, 503, or 504

Result: User never sees these errors!

2. Aggressive Timeouts

proxy_connect_timeout 2s;
proxy_send_timeout 3s;
proxy_read_timeout 3s;

Why so short?

Detect failures fast
Failover happens in seconds, not minutes
Better user experience

3. Structured Logging

log_format detailed 
    'pool=$upstream_http_x_app_pool '
    'release=$upstream_http_x_release_id '
    'upstream_status=$upstream_status '
    'latency=$request_time ';

access_log /var/log/nginx/access.log detailed;

Log Example:

pool=blue release=blue-release-1.0.0 upstream_status=200 latency=0.045

How Failover Works

Normal Operation (Blue Active)

User Request → Nginx → App Blue → Success (200 OK)
                       ↓
                   Nginx Logs: pool=blue

Failure Scenario

Let's trace what happens when Blue crashes:

Step 1: User Makes Request

User → GET http://localhost:8080/

Step 2: Nginx Tries Blue

Nginx → app_blue:3000
        ↓
    Connection Refused (Blue is down)

Step 3: Nginx Detects Failure

Nginx marks app_blue as FAILED
(max_fails=1 threshold reached)

Step 4: Nginx Retries to Green

Nginx → app_green:3000
        ↓
    Success! (200 OK)

Step 5: User Receives Response

User ← 200 OK from Green
(User never knew Blue failed!)

Step 6: Alert Watcher Detects Failover

# watcher.py detects pool change
if pool != self.last_alerted_pool:
    send_slack_alert("Failover detected: blue → green")

Step 7: Slack Alert Sent

Failover detected: blue → green (to backup)
Release: green-release-1.0.0
Upstream: 172.18.0.4:3000
Request time: 0.05s

Real-Time Alerting System

Alert Watcher Architecture

The watcher.py script monitors Nginx logs in real-time:

class AlertWatcher:
    def __init__(self):
        # Track current state
        self.current_pool = ACTIVE_POOL
        self.last_alerted_pool = ACTIVE_POOL

        # Rolling window for error rate
        self.request_window = deque(maxlen=WINDOW_SIZE)

        # Cooldown timers
        self.last_failover_alert_time = 0
        self.last_error_alert_time = 0

How It Works

1. Log Tailing

def tail_log_file(self):
    with open(LOG_FILE, 'r') as log:
        log.seek(0, 2)  # Go to end of file
        while True:
            line = log.readline()
            if not line:
                time.sleep(0.1)
                continue
            self.process_log_line(line)

Explanation:

Opens log file in read mode
Seeks to end (like tail -f)
Continuously reads new lines
Processes each line in real-time

2. Failover Detection

def check_failover(self, pool):
    if pool == self.last_alerted_pool:
        return  # No change

    # Determine alert type
    if pool == ACTIVE_POOL:
        title = f"Recovery detected: back to {pool}"
    else:
        title = f"Failover detected: {self.last_alerted_pool} → {pool}"

    # Send alert
    self.send_slack_alert(title)

3. Error Rate Monitoring

def check_error_rate(self, all_statuses):
    # Check if any status is 5xx
    is_5xx = any(500 <= s < 600 for s in all_statuses)

    # Add to sliding window
    self.request_window.append(is_5xx)

    # Calculate error rate
    error_count = sum(self.request_window)
    total_requests = len(self.request_window)
    error_rate = (error_count / total_requests) * 100

    # Check threshold
    if error_rate > ERROR_RATE_THRESHOLD:
        self.send_slack_alert(
            f"High upstream error rate: {error_rate:.2f}%"
        )

Alert Types

1. Failover Alert

Failover detected: blue → green (to backup)
Release: green-release-1.0.0
Upstream: 172.18.0.4:3000
Request time: 0.05s

When: Primary instance fails, traffic switches to backup

2. Recovery Alert

Recovery detected: back to blue (primary)
Release: blue-release-1.0.0
Upstream: 172.18.0.3:3000
Request time: 0.03s

When: Primary instance recovers, traffic returns to primary

3. High Error Rate Alert

High upstream error rate
5xx in upstream attempts: 5.50% over last 200 requests (threshold 2%).
Current pool: green

When: Error rate exceeds threshold in sliding window

Testing the Deployment

Test 1: Verify Normal Operation

# Check which pool is active
curl http://localhost:8080/version

# Expected output:
{
  "pool": "blue",
  "release": "blue-release-1.0.0"
}

Test 2: Trigger Failover

Manual Failover Test:

# Stop Blue instance
docker compose stop app_blue

# Make requests
for i in {1..10}; do
  curl http://localhost:8080/version
  sleep 1
done

# You should see responses from Green
# Check Slack for failover alert

Test 3: Verify Zero Downtime

# In one terminal, continuously make requests
while true; do
  curl -s http://localhost:8080/version | jq -r '.pool'
  sleep 0.5
done

# In another terminal, stop Blue
docker compose stop app_blue

# Observe: No failed requests!
# Output switches from "blue" to "green" seamlessly

Test 4: Test Recovery

# Restart Blue instance
docker compose start app_blue

# Wait for health check to pass (10-15 seconds)
sleep 15

# Make requests
curl http://localhost:8080/version

# Should show Blue is active again
# Check Slack for recovery alert

Test 5: View Nginx Logs

# View real-time logs
docker compose exec nginx_proxy tail -f /var/log/nginx/access.log

# Example output:
pool=blue release=blue-release-1.0.0 upstream_status=200 latency=0.023
pool=green release=green-release-1.0.0 upstream_status=502, 200 latency=0.045

Common Issues and Troubleshooting

Issue 1: Failover Not Happening

Symptoms: Blue fails but traffic doesn't switch to Green

Solutions:

# Check Nginx config
docker compose exec nginx_proxy cat /etc/nginx/nginx.conf.processed

# Restart Nginx
docker compose restart nginx_proxy

# Verify Green is healthy
docker compose exec app_green wget -qO- http://localhost:3000/healthz

Issue 2: Alerts Not Sending to Slack

Symptoms: Failover happens but no Slack notification

Solutions:

Verify Webhook URL:

# Check .env file
cat .env | grep SLACK_WEBHOOK_URL

# Test webhook manually
curl -X POST YOUR_WEBHOOK_URL \
  -H 'Content-Type: application/json' \
  -d '{"text":"Test alert"}'

Restart Watcher:

docker compose restart alert_watcher

Issue 3: Both Instances Receiving Traffic

Symptoms: Requests alternate between Blue and Green

Solution:

# Ensure Green has "backup" directive
# Check nginx.conf.template
# Rebuild and restart
docker compose down
docker compose up -d --build

Debugging Commands Cheat Sheet

# View all container status
docker compose ps

# View all logs
docker compose logs

# View specific service logs
docker compose logs -f nginx_proxy

# Execute command in container
docker compose exec nginx_proxy sh

# Restart specific service
docker compose restart app_blue

# Clean up everything
docker compose down -v

Key Takeaways

What You've Learned

Blue-Green Deployment
- How to implement zero-downtime deployments
- Difference between blue-green and load balancing
- When to use blue-green vs. other strategies
Nginx as a Reverse Proxy
- Upstream configuration
- Failover mechanisms
- Health checks and timeouts
- Structured logging
Real-Time Monitoring
- Log tailing and parsing
- Event detection
- Sliding window calculations
- Alert deduplication
Slack Integration
- Webhook setup
- Alert formatting
- Error handling
Docker Orchestration
- Multi-container applications
- Service dependencies
- Volume management
- Health checks

Real-World Applications

This pattern is used by:

Netflix - Canary deployments with instant rollback
Amazon - Blue-green for critical services
Heroku - Platform-level blue-green deployments
GitHub - Zero-downtime deployments

Next Steps

Enhance the System
- Add database with replication
- Implement canary deployments
- Add A/B testing capabilities
Improve Monitoring
- Add Prometheus metrics
- Create Grafana dashboards
- Implement distributed tracing
Scale the Architecture
- Deploy to Kubernetes
- Use managed load balancers
- Implement auto-scaling

Additional Resources

Documentation

Tools

Nginx - High-performance web server
Docker - Containerization platform
Slack - Team communication

Conclusion

Congratulations! You've just learned how to implement a production-ready blue-green deployment system with automatic failover and real-time alerting. This is a critical skill for modern DevOps engineers.

Remember:

Test thoroughly before deploying to production
Monitor continuously - you can't fix what you can't see
Automate everything - manual processes lead to errors
Document your decisions - future you will thank you

Blue-green deployment is just one piece of the DevOps puzzle. The principles you've learned here - automation, monitoring, resilience, and rapid recovery - apply to all aspects of modern infrastructure.

Keep experimenting, keep learning, and most importantly, keep building!

Full Project Repository: HNG DevOps on GitHub

Happy deploying!

Complete Beginner's Guide to Building a Microservices TODO Application with Docker and Traefik

cypher682 — Tue, 09 Dec 2025 14:35:15 +0000

Introduction

Welcome! This guide will walk you through a real-world microservices application - a TODO app that demonstrates how different services written in different programming languages can work together seamlessly. If you're new to DevOps or microservices, don't worry - I'll explain everything step by step!

Repository: HNG DevOps on GitHub

What You'll Learn:

How microservices architecture works in practice
How to use Docker to containerize multiple services
How to set up Traefik as a reverse proxy
How different programming languages work together
How services communicate with each other
How to implement authentication across microservices

What is a Microservices Architecture?

The Traditional Way (Monolithic)

Imagine building a house where everything - kitchen, bedroom, bathroom - is one giant room. If you want to renovate the kitchen, you might affect the bedroom too. That's a monolithic application.

The Microservices Way

Now imagine a house where each room is a separate, self-contained unit. You can renovate the kitchen without touching the bedroom. Each room has its own entrance and can be accessed independently. That's microservices architecture.

Why Microservices?

Independent Development: Different teams can work on different services

Technology Flexibility: Use the best language/framework for each service

Scalability: Scale only the services that need it

Fault Isolation: If one service fails, others keep running

Easier Maintenance: Smaller codebases are easier to understand

Project Overview

This project is a TODO application that allows users to:

Register and log in (Authentication)
Create, read, update, and delete TODO items
View user profiles
Track operations through logging

The Magic Behind It

The application is split into 5 independent microservices:

Frontend (Vue.js) - The user interface
Auth API (Go) - Handles user authentication
TODOs API (Node.js) - Manages TODO items
Users API (Java/Spring Boot) - Manages user profiles
Log Message Processor (Python) - Processes and logs operations

Understanding the Components

Let's break down each component in simple terms:

1. Frontend (Vue.js - JavaScript)

What it does: This is what users see and interact with - the web interface.

Think of it as: The cashier at a restaurant who takes your order and brings you food.

Why Vue.js? It's a modern, lightweight JavaScript framework perfect for building interactive user interfaces.

2. Auth API (Go)

What it does: Handles user authentication and generates JWT (JSON Web Tokens) for secure access.

Think of it as: The security guard who checks your ID and gives you a badge to enter the building.

How JWT Works:

1. User logs in with username/password
2. Auth API verifies credentials
3. Auth API generates a JWT token (like a temporary pass)
4. User includes this token in future requests
5. Other services verify the token to confirm identity

Why Go? Go is fast, efficient, and excellent for building high-performance APIs.

3. TODOs API (Node.js)

What it does: Manages all TODO operations - create, read, update, delete.

Think of it as: The kitchen that prepares your food orders.

Redis Integration:
When you create or delete a TODO, the API sends a message to Redis (a message queue), which the Log Message Processor picks up and logs.

Why Node.js? Node.js is great for I/O-heavy operations and has a rich ecosystem of packages.

4. Users API (Java/Spring Boot)

What it does: Manages user profile information.

Think of it as: The HR department that maintains employee records.

Why Java/Spring Boot? Spring Boot is enterprise-grade, robust, and widely used in production environments.

5. Log Message Processor (Python)

What it does: Listens to Redis queue and logs TODO operations.

Think of it as: The security camera system that records all activities.

Why Python? Python is simple, readable, and perfect for quick scripting tasks.

6. Traefik (Reverse Proxy)

What it does: Routes incoming requests to the correct service and handles SSL/TLS certificates.

Think of it as: The receptionist who directs visitors to the right department.

Key Features:

Automatic service discovery
SSL/TLS certificate management (Let's Encrypt)
HTTP to HTTPS redirection
Path-based routing

Technology Stack

Here's a summary of all technologies used:

Component	Technology	Purpose
Frontend	Vue.js	User interface
Auth API	Go	Authentication service
TODOs API	Node.js	TODO management
Users API	Java/Spring Boot	User profile management
Log Processor	Python	Logging service
Message Queue	Redis	Inter-service communication
Reverse Proxy	Traefik	Request routing & SSL
Containerization	Docker	Package services
Orchestration	Docker Compose	Manage multiple containers

Architecture Overview

Visual Architecture

                    Internet
                       |
                       v
              +----------------+
              |    Traefik     |
              | (Port 80/443)  |
              +----------------+
                       |
    +------------------+------------------+
    |                  |                  |
    v                  v                  v
+----------+    +----------+    +----------+
| Frontend |    | Auth API |    |TODOs API |
| (Vue.js) |    |   (Go)   |    | (Node.js)|
+----------+    +----------+    +----------+
    |                  |                  |
    |                  v                  |
    |          +-------------+            |
    +--------->|  Users API  |<-----------+
               | (Java)      |
               +-------------+


+-------------+              +-------------+
| Log Message |<--Redis------|  TODOs API  |
| Processor   |              +-------------+
| (Python)    |
+-------------+

Request Flow Example

Let's trace what happens when a user creates a TODO:

User Action: User fills out TODO form and clicks "Create"
Frontend: Sends POST request to https://example.com/api/todos
Traefik: Receives request, checks routing rules, forwards to TODOs API
TODOs API:
- Validates JWT token
- Creates TODO in database
- Publishes "TODO created" message to Redis
- Returns success response
Redis: Stores message in queue
Log Message Processor:
- Reads message from Redis
- Logs the operation
Frontend: Receives success response, updates UI

Setting Up the Project

Prerequisites

Before you start, make sure you have:

Docker installed (Download here)
Docker Compose installed (usually comes with Docker Desktop)
Basic command line knowledge
A text editor (VS Code, Sublime, etc.)

Step 1: Clone the Repository

git clone https://github.com/cypher682/hng13-stage-3-devops.git
cd DevOps-Stage-6

Step 2: Understand the Project Structure

DevOps-Stage-6/
├── frontend/              # Vue.js application
│   ├── Dockerfile        # Instructions to build frontend container
│   └── src/              # Source code
├── auth-api/             # Go authentication service
│   ├── Dockerfile
│   └── main.go
├── todos-api/            # Node.js TODO service
│   ├── Dockerfile
│   └── server.js
├── users-api/            # Java Spring Boot service
│   ├── Dockerfile
│   └── src/
├── log-message-processor/ # Python logging service
│   ├── Dockerfile
│   └── processor.py
├── docker-compose.yml    # Orchestration configuration
└── .env                  # Environment variables

Step 3: Configure Environment Variables

The .env file contains configuration for all services:

# User credentials (for testing)
USER1_USERNAME=user1
USER1_PASSWORD=password1

# JWT Secret (for token generation)
JWT_SECRET=myfancysecret

# Redis Configuration
REDIS_HOST=redis-queue
REDIS_PORT=6379
REDIS_CHANNEL=log_channel

Important: In production, never commit .env files with real credentials!

Step 4: Build and Run

# Build all services
docker compose build

# Start all services
docker compose up -d

# Check if all services are running
docker compose ps

What -d means: Runs containers in "detached" mode (in the background).

Step 5: Verify Services

Check that all containers are running:

docker compose ps

You should see all services with status "Up" or "Running":

traefik
frontend
auth-api
todos-api
users-api
log-message-processor
redis-queue

Understanding Docker Compose

What is Docker Compose?

Docker Compose is a tool for defining and running multi-container Docker applications. Instead of running each container manually, you define everything in a docker-compose.yml file.

Key Sections Explained

1. Services Definition

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped

Explanation:

services: - Defines all containers
traefik: - Service name
image: - Docker image to use
container_name: - Name for the container
restart: - Restart policy

2. Port Mapping

ports:
  - "80:80"
  - "443:443"

Format: HOST_PORT:CONTAINER_PORT

Explanation:

Maps port 80 on your computer to port 80 in the container
Allows you to access services from your browser

3. Networks

networks:
  - app-network

Explanation:

All services on the same network can communicate with each other
Services can reference each other by service name (e.g., http://auth-api:8081)

4. Traefik Labels (Routing)

labels:
  - "traefik.enable=true"
  - "traefik.http.routers.frontend.rule=Host(`example.com`)"
  - "traefik.http.routers.frontend.entrypoints=websecure"

Explanation:

Tells Traefik to route requests for example.com to this service
Enables HTTPS with automatic certificate generation

How Services Communicate

Internal Communication (Service-to-Service)

Services communicate using service names as hostnames:

// In Frontend (Vue.js)
const AUTH_API_ADDRESS = 'http://auth-api:8081';
const TODOS_API_ADDRESS = 'http://todos-api:8082';

Why this works:

Docker Compose creates a network where services can find each other by name
No need for IP addresses

External Communication (User-to-Service)

Users access services through Traefik:

User → https://example.com → Traefik → Frontend
User → https://example.com/api/auth → Traefik → Auth API
User → https://example.com/api/todos → Traefik → TODOs API

Authentication Flow

1. User submits login form
   ↓
2. Frontend sends credentials to Auth API
   ↓
3. Auth API verifies with Users API
   ↓
4. Auth API generates JWT token
   ↓
5. Frontend stores token
   ↓
6. Frontend includes token in all future requests
   ↓
7. Each API validates token before processing

Testing the Application

1. Access the Frontend

Open your browser and navigate to:

http://localhost:8080

2. Test Authentication

Username: user1
Password: password1

3. Create a TODO

Click "Add TODO"
Enter a title and description
Click "Save"
Check that the TODO appears in the list

4. Check Logs

View logs from the Log Message Processor:

docker compose logs -f log-message-processor

You should see log entries for TODO creation.

5. Test API Endpoints Directly

Get all users:

curl http://localhost:8080/api/users

Login (get JWT token):

curl -X POST http://localhost:8080/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"user1","password":"password1"}'

Common Issues and Troubleshooting

Issue 1: Containers Won't Start

Symptoms: Containers show as "Exited" or "Restarting"

Solutions:

# Check logs for specific service
docker compose logs auth-api

# Rebuild containers
docker compose down
docker compose build --no-cache
docker compose up -d

Issue 2: Port Already in Use

Error: bind: address already in use

Solutions:

# Find what's using the port (Windows)
netstat -ano | findstr :80

# Change port in docker-compose.yml
ports:
  - "8000:80"  # Use port 8000 instead

Issue 3: Services Can't Communicate

Symptoms: Frontend can't reach APIs

Solutions:

# Verify all services are on the same network
docker network inspect devops-stage-6_app-network

# Restart networking
docker compose down
docker compose up -d

Debugging Commands Cheat Sheet

# View all running containers
docker compose ps

# View logs for all services
docker compose logs

# View logs for specific service
docker compose logs -f frontend

# Restart specific service
docker compose restart todos-api

# Stop all services
docker compose down

# View resource usage
docker stats

Key Takeaways

What You've Learned

Microservices Architecture
- How to split an application into independent services
- Benefits of using different languages for different services
- How services communicate with each other
Docker & Containerization
- How to containerize applications
- How to use Docker Compose for multi-container applications
- How to manage container networking
Reverse Proxy with Traefik
- How to route requests to different services
- How to automatically manage SSL certificates
- How to use labels for service discovery
Inter-Service Communication
- Synchronous communication (HTTP/REST)
- Asynchronous communication (Message Queues)
- Authentication across services (JWT)

Real-World Applications

This architecture pattern is used by companies like:

Netflix - Hundreds of microservices
Amazon - Service-oriented architecture
Uber - Microservices for different features
Spotify - Independent teams, independent services

Next Steps

To deepen your understanding:

Modify the Application
- Add a new microservice (e.g., Comments API)
- Implement a database for persistent storage
- Add user registration functionality
Improve the Infrastructure
- Add monitoring with Prometheus and Grafana
- Implement centralized logging
- Add CI/CD pipeline with GitHub Actions
Security Enhancements
- Implement rate limiting
- Add API gateway
- Use secrets management

Additional Resources

Documentation

Tools

Docker Desktop
Postman - API testing
Portainer - Docker management UI

Conclusion

Congratulations! You've just learned how to build, deploy, and manage a complete microservices application. This project demonstrates real-world DevOps practices used by major tech companies.

Remember:

Start small - Don't try to build everything at once
Understand each component - Know what each service does
Experiment - Break things and fix them (that's how you learn!)
Read logs - They're your best friend when debugging
Keep learning - DevOps is constantly evolving

The skills you've gained here - Docker, microservices, reverse proxies, and service orchestration - are highly valuable in the industry. Keep practicing, keep building, and most importantly, keep learning!

Full Project Repository: HNG DevOps on GitHub

Building Your Own Virtual Private Cloud on Linux: A Deep Dive into Network Namespaces

cypher682 — Sun, 09 Nov 2025 17:07:03 +0000

Introduction: Why Build a VPC from Scratch?

Amazon Web Services revolutionized cloud computing with Virtual Private Clouds (VPCs), allowing users to create isolated network environments in the cloud. But have you ever wondered how VPCs actually work under the hood?

In this project, I recreated AWS VPC functionality on a single Linux machine using native networking primitives. No Docker, no Kubernetes—just pure Linux networking: network namespaces, bridges, veth pairs, and iptables.

What You'll Learn

How network isolation works at the kernel level
Linux network namespaces as lightweight containers
Bridging and routing fundamentals
NAT implementation with iptables
Building infrastructure automation tools

Real-World Applications

Understanding cloud provider networking internals - See how AWS/Azure/GCP implement VPCs
Building custom network isolation for multi-tenant systems
DevOps and infrastructure automation skills - Create your own networking tools
Debugging complex network issues - Deep knowledge of Linux networking stack

Architecture Overview

A VPC in AWS provides:

Isolated network space with your own IP range (CIDR)
Subnets that partition your VPC into smaller networks
Internet Gateway for public subnet internet access
NAT Gateway for private subnet outbound access
Security Groups for firewall rules
VPC Peering for cross-VPC communication

My Implementation Stack

┌─────────────────────────────────────────────┐
│           Linux Host System                  │
│                                              │
│  ┌────────────────────────────────────────┐ │
│  │    VPC1 (10.0.0.0/16)                  │ │
│  │                                        │ │
│  │  br-vpc1 (Linux Bridge = VPC Router)  │ │
│  │        │                    │          │ │
│  │    veth-pair           veth-pair      │ │
│  │        │                    │          │ │
│  │  ┌─────▼────┐        ┌─────▼────┐    │ │
│  │  │ Public   │        │ Private  │    │ │
│  │  │ Subnet   │        │ Subnet   │    │ │
│  │  │ Namespace│        │ Namespace│    │ │
│  │  │10.0.1.2  │        │10.0.2.2  │    │ │
│  │  └─────┬────┘        └─────┬────┘    │ │
│  │        │                    X          │ │
│  │    NAT (iptables)      No Internet    │ │
│  └────────┼────────────────────────────────┘│
│           │                                  │
│      [eth0] ──► Internet                   │
└─────────────────────────────────────────────┘

Component Mapping

AWS VPC Concept	Linux Implementation
VPC	Linux Bridge (br-vpc1)
Subnet	Network Namespace
Subnet Connection	veth pair
Internet Gateway	iptables NAT MASQUERADE
Route Table	ip route commands
Security Group	iptables INPUT rules
VPC Peering	veth pair between bridges

Part 1: Understanding Network Namespaces

Network namespaces are Linux's way of creating isolated network stacks. Each namespace has its own:

Network interfaces
IP addresses
Routing tables
iptables rules

Why This Matters: This is the same technology Docker uses for container networking. Understanding this gives you deep insights into containerization.

Creating Isolation

# In vpcctl.py - SubnetManager.add()
run_command(f"ip netns add {ns_name}")  # Create isolated namespace

When you create a namespace, the Linux kernel creates a completely separate network stack. Processes inside can't see or access the host's network—perfect isolation!

Test It Yourself

# Create namespace
sudo ip netns add test-ns

# List interfaces in host
ip link

# List interfaces in namespace (only loopback!)
sudo ip netns exec test-ns ip link

You'll notice the namespace starts with only a loopback interface. It's completely isolated!

Part 2: Connecting Namespaces - The veth Pair Magic

Network namespaces are isolated, so we need a way to connect them. Enter veth pairs—virtual ethernet cables.

Conceptual Model

Think of a veth pair as a virtual ethernet cable with two ends:

One end plugs into the namespace
Other end plugs into the host or bridge

# Creating the connection
run_command(f"ip link add {veth_host} type veth peer name {veth_ns}")
run_command(f"ip link set {veth_host} master {bridge}")  # Connect to bridge
run_command(f"ip link set {veth_ns} netns {ns_name}")    # Move to namespace

Why This Works: Packets entering one end of the veth pair automatically come out the other end—like a wormhole for network traffic.

IP Address Assignment

net_info = NetworkUtils.get_network_info(cidr)
# Assign IP inside namespace
run_command(
    f"ip netns exec {ns_name} ip addr add "
    f"{net_info['first_host']}/{net_info['prefix']} dev {veth_ns}"
)

Key Insight: We use the first usable IP (.2) for the subnet, and the bridge gets the gateway IP (.1). This mirrors how AWS assigns IPs in subnets.

Part 3: The Bridge - Your VPC Router

A Linux bridge is like a virtual network switch. It forwards packets between connected interfaces at Layer 2.

Bridge Creation

bridge_name = f"br-{name}"
run_command(f"ip link add {bridge_name} type bridge")
run_command(f"ip addr add {net_info['gateway']}/{net_info['prefix']} dev {bridge_name}")
run_command(f"ip link set {bridge_name} up")

Critical Configuration

# Disable bridge netfilter - allows direct L2 forwarding
run_command("sysctl -w net.bridge.bridge-nf-call-iptables=0")

Why This Matters: By default, Linux bridges pass traffic through iptables. For intra-VPC communication, we want direct Layer 2 switching for performance—just like a real network switch.

Routing Setup

# Inside namespace: route everything through bridge gateway
run_command(f"ip netns exec {ns_name} ip route add default via {gateway_ip}")

This makes the bridge act as the default gateway—all traffic from namespaces flows through it.

Part 4: NAT Gateway - Internet Access

Private networks use RFC 1918 addresses (10.x.x.x, 172.16.x.x, 192.168.x.x) that aren't routable on the internet. NAT (Network Address Translation) solves this.

NAT Implementation

# Enable IP forwarding (routing between interfaces)
run_command("sysctl -w net.ipv4.ip_forward=1")

# MASQUERADE: Replace source IP with host's public IP
run_command(
    f"iptables -t nat -A POSTROUTING -s {cidr} -o {interface} -j MASQUERADE"
)

# Allow forwarding from VPC to internet
run_command(
    f"iptables -A FORWARD -s {cidr} -i {bridge} -o {interface} -j ACCEPT"
)

# Allow return traffic
run_command(
    f"iptables -A FORWARD -d {cidr} -i {interface} -o {bridge} "
    f"-m state --state RELATED,ESTABLISHED -j ACCEPT"
)

How MASQUERADE Works

Packet leaves namespace with source IP 10.0.1.2
Reaches host via bridge
iptables MASQUERADE rewrites source to host's public IP
Internet sees request from host, not internal IP
Response comes back, iptables rewrites destination back to 10.0.1.2
Packet forwarded to namespace

Key Insight: We only NAT public subnets. Private subnets remain isolated—they can reach other subnets within the VPC but not the internet.

Part 5: VPC Isolation & Peering

Default Isolation

Without configuration, VPCs can't communicate. Why?

Each VPC has its own bridge
No routes exist between bridges
iptables FORWARD policy is DROP by default

Testing Isolation

# From vpc1 namespace, try to reach vpc2
sudo ip netns exec vpc1-public ping 172.16.1.2
# Result: Network unreachable (no route to 172.16.0.0/16)

VPC Peering Implementation

The tricky part: packets from a namespace need to reach another VPC's bridge.

Solution:

# Create veth pair between bridges
run_command(f"ip link add {veth1} type veth peer name {veth2}")
run_command(f"ip link set {veth1} master {vpc1['bridge']}")
run_command(f"ip link set {veth2} master {vpc2['bridge']}")

# CRITICAL: Add routes in EACH namespace
for subnet in vpc1["subnets"]:
    ns_name = subnet["namespace"]
    run_command(
        f"ip netns exec {ns_name} ip route add {vpc2['cidr']} "
        f"via {vpc1['gateway']}"
    )

Why This Works

Namespace sends packet to VPC2 CIDR
Route points to its own gateway (bridge)
Bridge forwards to peering veth pair
Packet arrives at VPC2's bridge
VPC2 bridge forwards to destination namespace

Common Mistake I Made: Initially, I added routes on the host routing table. This doesn't work because packets originate from inside namespaces, which have their own routing tables!

Part 6: Firewall Rules (Security Groups)

AWS Security Groups are stateful firewalls. We simulate this with iptables inside namespaces.

JSON Policy Design

{
  "subnet": "10.0.1.0/24",
  "ingress": [
    {"port": 80, "protocol": "tcp", "action": "allow"},
    {"port": 22, "protocol": "tcp", "action": "deny"}
  ]
}

Implementation

# Essential: Allow established connections (stateful behavior)
run_command(
    f"ip netns exec {ns_name} iptables -A INPUT "
    f"-m state --state ESTABLISHED,RELATED -j ACCEPT"
)

# Apply custom rules
for rule in policy.get("ingress", []):
    port = rule["port"]
    protocol = rule["protocol"]
    action = "ACCEPT" if rule["action"] == "allow" else "DROP"
    run_command(
        f"ip netns exec {ns_name} iptables -A INPUT "
        f"-p {protocol} --dport {port} -j {action}"
    )

Why ESTABLISHED,RELATED Matters: Without this, responses to outbound connections would be blocked. The stateful rule tracks connections and allows return traffic automatically.

The CLI Tool: vpcctl

I built a Python CLI tool to automate all VPC operations.

Installation

git clone https://github.com/cypher682/hng13-stage-4-devops.git
cd hng13-stage-4-devops
chmod +x vpcctl.py

Usage Examples

Create VPC:

sudo ./vpcctl.py vpc create --name vpc1 --cidr 10.0.0.0/16

Add Subnets:

# Public subnet
sudo ./vpcctl.py subnet add \
  --vpc vpc1 \
  --name public \
  --cidr 10.0.1.0/24 \
  --type public

# Private subnet
sudo ./vpcctl.py subnet add \
  --vpc vpc1 \
  --name private \
  --cidr 10.0.2.0/24 \
  --type private

Enable NAT Gateway:

sudo ./vpcctl.py nat enable --vpc vpc1 --interface eth0

Deploy Web Server:

sudo ./vpcctl.py deploy webserver \
  --vpc vpc1 \
  --subnet public \
  --port 8080

Create VPC Peering:

sudo ./vpcctl.py peer create --vpc1 vpc1 --vpc2 vpc2

Apply Firewall Rules:

sudo ./vpcctl.py firewall apply \
  --vpc vpc1 \
  --subnet public \
  --policy policy.json

List All VPCs:

sudo ./vpcctl.py vpc list

Delete VPC:

sudo ./vpcctl.py vpc delete --name vpc1

Testing & Validation

Test 1: Intra-VPC Communication

# Create VPC with two subnets
sudo ./vpcctl.py vpc create --name vpc1 --cidr 10.0.0.0/16
sudo ./vpcctl.py subnet add --vpc vpc1 --name public \
  --cidr 10.0.1.0/24 --type public
sudo ./vpcctl.py subnet add --vpc vpc1 --name private \
  --cidr 10.0.2.0/24 --type private

# Test connectivity
sudo ip netns exec vpc1-public ping -c 3 10.0.2.2

✅ Expected: Success (same VPC, bridge routes traffic)

What's Happening:

Packet leaves public namespace (10.0.1.2)
Goes through veth to bridge
Bridge forwards to private subnet's veth
Arrives at private namespace (10.0.2.2)

Test 2: NAT Gateway

# Enable NAT
sudo ./vpcctl.py nat enable --vpc vpc1 --interface eth0

# Test public subnet internet access
sudo ip netns exec vpc1-public ping -c 3 8.8.8.8

✅ Expected: Success

# Test private subnet (should fail)
sudo ip netns exec vpc1-private ping -c 2 8.8.8.8

✅ Expected: Timeout (no NAT rule for private subnet)

Verification:

# Check NAT rules
sudo iptables -t nat -L POSTROUTING -n -v
# Should show MASQUERADE rule for 10.0.1.0/24 only

Test 3: VPC Isolation

# Create second VPC
sudo ./vpcctl.py vpc create --name vpc2 --cidr 172.16.0.0/16
sudo ./vpcctl.py subnet add --vpc vpc2 --name public \
  --cidr 172.16.1.0/24 --type public

# Try to communicate (should fail)
sudo timeout 3 ip netns exec vpc1-public ping 172.16.1.2

✅ Expected: Network unreachable

Test 4: VPC Peering

# Create peering
sudo ./vpcctl.py peer create --vpc1 vpc1 --vpc2 vpc2

# Now communication should work
sudo ip netns exec vpc1-public ping -c 3 172.16.1.2

✅ Expected: Success

# Verify routes were added
sudo ip netns exec vpc1-public ip route
# Should show: 172.16.0.0/16 via 10.0.0.1

Test 5: Firewall Rules

# Deploy web server
sudo ./vpcctl.py deploy webserver --vpc vpc1 --subnet public --port 8080

# Test before firewall
curl http://10.0.1.2:8080

✅ Works

# Apply restrictive policy
sudo ./vpcctl.py firewall apply \
  --vpc vpc1 \
  --subnet public \
  --policy test-policy.json

# Test allowed port
curl http://10.0.1.2:8080

✅ Still works (port 8080 is allowed in policy)

Complete Demo Script

Here's the full automated demo:

#!/bin/bash

# 1. Create VPC
sudo ./vpcctl.py vpc create --name vpc1 --cidr 10.0.0.0/16

# 2. Add subnets
sudo ./vpcctl.py subnet add --vpc vpc1 --name public \
  --cidr 10.0.1.0/24 --type public
sudo ./vpcctl.py subnet add --vpc vpc1 --name private \
  --cidr 10.0.2.0/24 --type private

# 3. Enable NAT
IFACE=$(ip route | grep default | awk '{print $5}' | head -1)
sudo ./vpcctl.py nat enable --vpc vpc1 --interface $IFACE

# 4. Deploy web servers
sudo ./vpcctl.py deploy webserver --vpc vpc1 --subnet public --port 8080
sudo ./vpcctl.py deploy webserver --vpc vpc1 --subnet private --port 8081

# 5. Test intra-VPC communication
sudo ip netns exec vpc1-public ping -c 3 10.0.2.2

# 6. Test NAT gateway
sudo ip netns exec vpc1-public ping -c 3 8.8.8.8
sudo timeout 3 ip netns exec vpc1-private ping -c 2 8.8.8.8

# 7. Create second VPC
sudo ./vpcctl.py vpc create --name vpc2 --cidr 172.16.0.0/16
sudo ./vpcctl.py subnet add --vpc vpc2 --name public \
  --cidr 172.16.1.0/24 --type public
sudo ./vpcctl.py nat enable --vpc vpc2 --interface $IFACE

# 8. Test VPC isolation
sudo timeout 3 ip netns exec vpc1-public ping -c 2 172.16.1.2

# 9. Create VPC peering
sudo ./vpcctl.py peer create --vpc1 vpc1 --vpc2 vpc2

# 10. Test after peering
sudo ip netns exec vpc1-public ping -c 3 172.16.1.2

# 11. Apply firewall rules
sudo ./vpcctl.py firewall apply --vpc vpc1 --subnet public \
  --policy test-policy.json

# 12. View logs
tail -30 /var/lib/vpcctl/vpcctl.log

# 13. List resources
sudo ./vpcctl.py vpc list

# 14. Cleanup
sudo ./vpcctl.py vpc delete --name vpc1
sudo ./vpcctl.py vpc delete --name vpc2

Cleanup Process

Proper cleanup is critical. Orphaned namespaces and iptables rules can cause issues.

# Delete VPC (automated cleanup)
sudo ./vpcctl.py vpc delete --name vpc1

What happens internally:

Kill all processes in namespaces
Delete namespaces
Remove veth pairs
Delete iptables rules
Remove bridge
Clean up state file

Emergency Cleanup

sudo ./cleanup.sh
# Removes ALL network resources created by vpcctl

Verification

# Should be empty
ip netns list
ip link show type bridge | grep br-

# Check iptables
sudo iptables -L FORWARD -n
sudo iptables -t nat -L POSTROUTING -n

Challenges & Solutions

Challenge 1: VPC Peering Not Working

Problem: Routes added to host routing table, but packets originate from namespaces.

Solution: Add routes inside each namespace pointing to their respective gateways.

# Wrong approach
run_command(f"ip route add {vpc2['cidr']} via {vpc2_peer_ip}")

# Correct approach
for subnet in vpc1["subnets"]:
    ns_name = subnet["namespace"]
    run_command(
        f"ip netns exec {ns_name} ip route add {vpc2['cidr']} "
        f"via {vpc1['gateway']}"
    )

Challenge 2: Firewall Blocking Everything

Problem: Applied rules but forgot to allow established connections.

Solution: Always add stateful rules first:

run_command(
    f"ip netns exec {ns_name} iptables -A INPUT "
    f"-m state --state ESTABLISHED,RELATED -j ACCEPT"
)

Challenge 3: Bridge Netfilter Interference

Problem: iptables was processing bridge traffic, causing performance issues.

Solution: Disable bridge netfilter:

sysctl -w net.bridge.bridge-nf-call-iptables=0

Key Takeaways

Network Namespaces provide true isolation—the foundation of containers
veth Pairs are the glue connecting isolated environments
Bridges act as virtual switches for Layer 2 forwarding
iptables is incredibly powerful for NAT, routing, and firewalling
Proper cleanup is essential for infrastructure automation

Real-World Applications

Container Networking: Docker/Kubernetes use the same primitives
Multi-tenant Systems: Isolate customer workloads
Cloud Provider Internals: Understanding how AWS VPC really works
Security: Network segmentation and isolation strategies

Architecture Deep Dive

Let me explain how packets flow through the system:

Scenario 1: Public Subnet → Internet

[Namespace] 10.0.1.2
     ↓ (veth pair)
[Bridge] br-vpc1 (10.0.0.1)
     ↓ (routing decision)
[iptables NAT] MASQUERADE (rewrites source IP)
     ↓
[eth0] → Internet

Scenario 2: Namespace → Namespace (Same VPC)

[Namespace A] 10.0.1.2
     ↓ (veth pair)
[Bridge] br-vpc1 (L2 switching)
     ↓ (veth pair)
[Namespace B] 10.0.2.2

Scenario 3: VPC Peering

[VPC1 Namespace] 10.0.1.2
     ↓ (veth pair)
[VPC1 Bridge] br-vpc1
     ↓ (peering veth pair)
[VPC2 Bridge] br-vpc2
     ↓ (veth pair)
[VPC2 Namespace] 172.16.1.2

Performance Considerations

Bridge vs Router

Using bridges instead of routing gives us:

Lower latency - Layer 2 switching is faster than Layer 3 routing
Higher throughput - No routing table lookups for intra-VPC traffic
Simpler configuration - Bridges handle MAC learning automatically

Namespace Overhead

Network namespaces are lightweight:

~1KB memory per namespace
Negligible CPU overhead for namespace switching
Near-native performance for network operations

Scalability Limits

On a typical Linux system:

~100,000+ namespaces possible
Limited by file descriptors and memory
iptables rules become the bottleneck at scale (~10,000+ rules)

Security Considerations

Isolation Guarantees

Network namespaces provide:

Complete network stack isolation
Separate iptables rules
Independent routing tables
Process isolation (can't see other namespace processes)

Attack Surface

Potential security concerns:

Host compromise affects all VPCs
Bridge vulnerabilities (MAC flooding, ARP spoofing)
iptables misconfigurations can leak traffic

Best Practices

Least privilege - Only enable NAT for public subnets
Default deny - Block all traffic, then allow specific flows
Audit logging - Log all iptables rules and changes
Regular cleanup - Remove unused resources

Future Enhancements

What's Missing?

Compared to AWS VPC, this implementation lacks:

IPv6 Support
- Current: IPv4 only
- Enhancement: Dual-stack networking
DNS Server
- Current: No internal DNS
- Enhancement: dnsmasq in each VPC
DHCP
- Current: Static IP assignment
- Enhancement: Dynamic IP allocation
Network ACLs
- Current: Security groups only
- Enhancement: Subnet-level firewalls
VPC Flow Logs
- Current: Basic logging
- Enhancement: Detailed traffic logging
Elastic IPs
- Current: No persistent public IPs
- Enhancement: Static IP mapping

Implementation Ideas

DNS Server:

def add_dns_server(vpc_name):
    # Start dnsmasq in namespace
    run_command(
        f"ip netns exec {vpc_name}-dns "
        f"dnsmasq --interface=lo --bind-interfaces "
        f"--listen-address=10.0.0.2"
    )

DHCP Server:

def add_dhcp_server(vpc_name, cidr):
    # Configure dnsmasq for DHCP
    run_command(
        f"ip netns exec {vpc_name}-dhcp "
        f"dnsmasq --dhcp-range={start_ip},{end_ip},12h"
    )

Comparison with Real Cloud VPCs

Feature	My Implementation	AWS VPC
Isolation	✅ Network namespaces	✅ Hypervisor-level
Subnets	✅ Multiple per VPC	✅ Multiple per VPC
NAT Gateway	✅ iptables MASQUERADE	✅ Managed NAT service
VPC Peering	✅ veth pairs	✅ Software-defined networking
Security Groups	✅ iptables rules	✅ Stateful firewall
Network ACLs	❌ Not implemented	✅ Subnet-level firewall
DNS	❌ Not implemented	✅ Route 53 integration
DHCP	❌ Static IPs only	✅ DHCP options
Flow Logs	⚠️ Basic logging	✅ Detailed flow logs
IPv6	❌ IPv4 only	✅ Dual-stack
Multi-region	❌ Single host	✅ Global infrastructure
HA/Redundancy	❌ Single point of failure	✅ Multi-AZ redundancy

Learning Resources

Official Documentation

Books

Linux Networking Cookbook by Carla Schroder
TCP/IP Illustrated by W. Richard Stevens
Linux Kernel Networking by Rami Rosen

Online Courses

Linux Foundation: Linux Networking and Administration
Pluralsight: Linux Networking Fundamentals
Udemy: Linux Networking Masterclass

Project Repository

GitHub: [https://github.com/cypher682/vpcctl-linux-networking]

Video Demo: [https://youtu.be/7LYUl3hc3xE]

Repository Structure

vpcctl-project/
├── README.md              # Complete documentation
├── vpcctl             # Main CLI tool
├── demo.sh                # Automated demo
├── cleanup.sh             # Emergency cleanup
├── docs/
│   ├── architecture-diagram.png

Conclusion

Building a VPC from scratch taught me more about networking than reading documentation ever could. Understanding these primitives—namespaces, bridges, veth pairs, and iptables—gives you superpowers when debugging container networking, understanding cloud provider internals, or designing multi-tenant systems.

Key Lessons:

Linux networking is powerful - You don't need specialized tools for complex networking
Cloud abstractions are implementations - AWS VPC is just well-packaged Linux networking
Isolation is achievable - Network namespaces provide true isolation
Automation is essential - Infrastructure as code makes everything reproducible

Got questions? Drop them in the comments below! 👇

Found this useful? Star the repo and share with fellow DevOps engineers! ⭐

GitHub:https://github.com/cypher682)

DEV Community: cypher682

Event-Driven AWS Security Automation: Real-Time Detection and Remediation with Terraform and Pulumi

Architecture

Detection and Remediation Coverage

Dual IaC: Terraform and Pulumi

The Finding Data Contract

DynamoDB Schema

Testing

Sprint Results — July 27, 2026

Repo

Building the Full LGTM Observability Stack from Scratch: Metrics, Logs, Traces, and Alerting

Architecture

The OpenTelemetry Collector Pipeline

Trace-to-Log Correlation

Custom Python Exporter

Dashboards as Code

Alertmanager: Noise Suppression

Kubernetes Transition

Repo

Modular AWS Infrastructure With Terraform: 2-Tier VPC, CIS-Hardened EC2, OIDC CI/CD, and a $6 Sprint Model

Architecture

Module Design

Security: No Bastion Host

Security: OIDC for GitHub Actions

Secrets: SSM Parameter Store

Configuration Management: Ansible

CI/CD Pipeline

Cost Model

Multi-Environment

What Comes Next

Building an Async Job Platform with BullMQ, Socket.io, and Webhooks

Architecture

The Job Lifecycle

Real-Time Updates: Cross-Process Socket.io

Webhooks: HMAC Signatures, Exponential Backoff, Circuit Breaking

HMAC-SHA256 Signatures

Exponential Backoff

Redis Circuit Breaker

Idempotency: Safe Request Replay

File Upload Pipeline

Cross-Cutting: Auth, Rate Limiting, Versioning

Testing and CI

Repo

Building an IAM Service with FastAPI: Refresh Token Families, TOTP MFA, and RBAC

Architecture

The Auth Flow

Refresh Token Rotation and Reuse Detection

MFA: Separating Password Verification from Full Authentication

Sessions

RBAC: Roles, Permissions, and Superuser Bypass

Rate Limiting and Brute-Force Lockout

Password Policy and Breach Checking

Email Verification and Password Reset

Implementation Notes Worth Calling Out

Testing and CI

Repo

Designing a Production-Grade Blue-Green ECS Platform on AWS with Terraform

Table of Contents

What Was Built

Architecture Overview

Terraform Module Structure

Security Design

1. Network Isolation (Security Groups)

2. IAM Role Separation

3. Secrets Management Flow

Blue-Green Deployment Mechanics

Initial State

Deployment Process

Why This Works

Data Persistence Validation

What Broke (And What I Learned)

Issue 1: RDS Connection Delay After "Available" Status

Issue 2: Docker HEALTHCHECK Missing Dependency

Issue 3: Git Bash Path Conversion on Windows

Issue 4: ALB Listener Syntax Constraint

Issue 5: PostgreSQL Minor Version Retirement

Issue 6: Terraform State Lock Timeout

Issue 7: ECR Image Pull Failures (Intermittent)

Issue 8: Terraform vs Manual Scaling Conflict

What I'd Change in Production