DEV Community: Victor Ojeje

I Built On-Prem Payment Infrastructure for a PCI DSS SAQ A-EP Audit. Here's Exactly How.

Victor Ojeje — Sat, 10 Oct 2026 07:42:30 +0000

Most engineers who deal with PCI DSS treat compliance as something that happens after the infrastructure exists: build the thing, then ask someone to audit it. That backwards process is why compliance engagements are painful and expensive.

For this project I reversed it. The architecture was designed specifically to qualify for SAQ A-EP first, and then every technical control was built and documented against the corresponding PCI DSS v4.0.1 requirement. The output is a completed Self-Assessment Questionnaire (SAQ A-EP) and a signed Attestation of Compliance (AOC), not just working infrastructure.

This post covers what SAQ A-EP actually requires, why the architecture I chose qualifies for it, and exactly how each control was implemented in code. AWS equivalents are called out throughout because the principles transfer directly even though the tooling differs.

What SAQ A-EP Is and Why It Matters

PCI DSS defines several SAQ forms based on how a merchant handles cardholder data. Most engineers are familiar with SAQ A (fully outsourced payment pages, simplest) and SAQ D (you store, process, or transmit PANs, heaviest). SAQ A-EP sits between them.

SAQ A-EP applies when all three of the following are true:

Your own server hosts the payment page
All actual card data entry happens inside a PCI-certified third party's environment (like Stripe's Embedded Checkout)
Your servers never receive, process, or store PANs, CVVs, or track data

WorkUp is a fictional B2B project-management SaaS charging $49/month per workspace with roughly 150,000 transactions per year. SAQ A-EP is the correct form. The payment page (billing.html) loads from WorkUp's own nginx. The card entry form is a Stripe-controlled iframe served from js.stripe.com. The Go API never receives card data. It only creates a Stripe Checkout Session and receives a webhook event after payment completes.

Because of this, Requirements 3 (stored card data), 4 (card data in transit), and 9 (physical access) are legitimately Not Applicable. Requirements 1, 2, 5, 6, 7, 8, 10, and 11 still apply in full to the systems hosting the payment page and supporting infrastructure.

That scope decision is the entire foundation of this project. Everything else builds on it.

Architecture: Four Isolated VMs, Zero LAN Traffic

The infrastructure runs on a self-hosted Proxmox node with four Ubuntu 22.04 VMs, each serving one segment:

Frontend VM  (192.168.123.21)  nginx serving static HTML/JS
Auth VM      (192.168.123.22)  Keycloak 26 + Postgres 18
Backend VM   (192.168.123.23)  Go/Gin API + Postgres 18
Obs VM       (192.168.123.24)  Grafana + Prometheus + Loki + Grafana Alloy

The key design choice: none of these VMs talk to each other over the internal LAN. Every inter-segment call goes through its public Cloudflare Tunnel URL. The frontend calls https://api.victorojeje.xyz, not http://192.168.123.23:8081. Auth tokens, API calls, log shipping: all of it exits through Cloudflare and comes back in through the tunnel on the destination VM.

This has a direct compliance consequence. PCI DSS Req 1 requires network segmentation. In most setups that means careful firewall rules to control east-west traffic between segments. Here, there is no east-west traffic. The default-DROP firewall policy on each VM is correct with zero application-level exceptions.

AWS parallel: In AWS this maps to placing each segment in its own subnet with a default-deny Network Access Control List (NACL), and using VPC Endpoints or PrivateLink for inter-service communication instead of direct subnet routing. The principle is the same: make the default behavior secure, then open only what's necessary.

Phase 1: Provisioning Infrastructure

Proxmox Template

Before Terraform provisions anything, a base VM template needs to exist. infra/template.sh handles this once. It downloads the Ubuntu 22.04 Jammy cloud image, uploads it to Proxmox, boots a VM, installs Docker and the QEMU guest agent, then converts it to a Proxmox template.

The script solves a real problem that breaks naively-written provisioning: Ubuntu cloud images fire apt-daily.timer with Persistent=true on first boot, which holds /var/lib/dpkg/lock-frontend and makes apt install fail immediately. The fix runs before touching apt:

sudo systemctl stop apt-daily.service apt-daily-upgrade.service \
  apt-daily.timer apt-daily-upgrade.timer unattended-upgrades.service 2>/dev/null || true
sudo systemctl mask apt-daily.service apt-daily-upgrade.service \
  apt-daily.timer apt-daily-upgrade.timer unattended-upgrades.service

sudo DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=120 update -y
sudo DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=120 install -y \
  qemu-guest-agent curl ca-certificates

After Docker installs, the VM shuts down and becomes a template with ciupgrade=0 set, which stops cloud-init from running package upgrades every time a VM is cloned from it.

Terraform VM Provisioning

Provider: bpg/proxmox v0.112.0. The older telmate/proxmox provider is unmaintained. bpg/proxmox has proper cloud-init support, a cleaner resource model, and active maintenance.

All four VMs follow the same pattern in clones.tf. They clone from the template with a full clone (not linked), get a static IP, and sized memory based on what each service actually needs: 768 MB for nginx, 1536 MB for Keycloak's JVM, 1024 MB for Go, 1536 MB for the full observability stack.

The single most important line in the VM definition:

network_device {
  bridge   = "vmbr0"
  model    = "virtio"
  firewall = true   # without this, firewall.tf rules exist but never enforce
}

firewall = true tells Proxmox to attach the NIC-level firewall policy to that specific network interface. Without this flag, you can define all the rules you want in firewall.tf and they will be stored in Proxmox's API but never evaluated. This was a real misconfiguration found mid-project; the auth, backend, and observability VMs were missing it.

Proxmox Firewall as Code

The firewall runs at three levels: Datacenter, Node, and per-VM. All three are declared in firewall.tf.

# Define who is allowed to manage things
resource "proxmox_virtual_environment_firewall_ipset" "management" {
  name = "management"
  cidr {
    name    = "192.168.123.132/32"
    comment = "Lab / management network"
  }
}

# Datacenter-level: drop everything by default
resource "proxmox_virtual_environment_cluster_firewall" "cluster" {
  enabled       = true
  input_policy  = "DROP"
  output_policy = "ACCEPT"
}

# Cluster-level rules: Proxmox GUI + SSH from management only
resource "proxmox_virtual_environment_firewall_rules" "cluster_mgmt" {
  rule { type = "in"; action = "ACCEPT"; source = "+management"; dport = "8006"; proto = "tcp" }
  rule { type = "in"; action = "ACCEPT"; source = "+management"; dport = "22";   proto = "tcp" }
  rule { type = "in"; action = "ACCEPT"; source = "+management"; proto = "icmp" }
}

# Reusable security groups attached to each VM
resource "proxmox_virtual_environment_cluster_firewall_security_group" "mgmt_ssh" {
  name = "mgmt-ssh"
  rule { type = "in"; action = "ACCEPT"; source = "+management"; dport = "22"; proto = "tcp" }
}

# Enable DROP on each VM, attach security groups
resource "proxmox_virtual_environment_firewall_options" "frontend" {
  node_name    = proxmox_virtual_environment_vm.frontend.node_name
  vm_id        = proxmox_virtual_environment_vm.frontend.vm_id
  enabled      = true
  input_policy = "DROP"
}

resource "proxmox_virtual_environment_firewall_rules" "frontend" {
  node_name = proxmox_virtual_environment_vm.frontend.node_name
  vm_id     = proxmox_virtual_environment_vm.frontend.vm_id
  rule { security_group = proxmox_virtual_environment_cluster_firewall_security_group.mgmt_ssh.name; iface = "net0" }
  rule { security_group = proxmox_virtual_environment_cluster_firewall_security_group.mgmt_icmp.name; iface = "net0" }
}

Result: all four VMs drop all inbound traffic except SSH and ICMP from the management machine. No application ports are reachable from the LAN. Public traffic reaches services through Cloudflare Tunnels only.

AWS parallel: This maps to a layered defense using NACLs (stateless, subnet-level, applied to all traffic entering the subnet) and Security Groups (stateful, instance-level, applied to specific ENIs). The three-level Proxmox hierarchy mirrors that model: cluster policy is the NACL equivalent, per-VM policy is the Security Group equivalent, and security groups in Proxmox are reusable rule sets like AWS managed prefix lists attached to SGs.

Phase 2: Hardening Every VM With Ansible

Four playbooks, one per VM segment. Every playbook runs the same task sequence before deploying its application: host_firewall.yml, os_hardening.yml, auditd.yml, aide.yml.

Guest OS Firewall (ufw)

The Proxmox NIC firewall is one layer. ufw inside each guest OS is a second, independent layer:

# tasks/host_firewall.yml
- name: Reset ufw to defaults
  community.general.ufw:
    state: reset

- name: Set default incoming policy to deny
  community.general.ufw:
    direction: incoming
    policy: deny

- name: Allow SSH from management network only
  community.general.ufw:
    rule: allow
    port: "22"
    proto: tcp
    src: "{{ management_network }}"

- name: Enable ufw
  community.general.ufw:
    state: enabled

There is no ICMP rule in ufw. The community.general.ufw module doesn't handle ICMP cleanly, and the Proxmox firewall already allows management ICMP. Adding a poorly-written ICMP rule in ufw adds complexity without benefit.

The Docker Port Publishing Problem

Docker bypasses ufw by writing directly to iptables PREROUTING. Any ports: mapping in a Compose file exposes that port to the LAN regardless of ufw's default-deny. This means a ports: ["8081:8081"] entry on the API container makes port 8081 reachable from any machine on the LAN, even with ufw default deny incoming in place.

The fix: no ports: block on any application service. cloudflared reaches the API container via Docker's internal service DNS (http://api:8081) without the port being published. The Keycloak container uses 127.0.0.1:8080:8080 to bind to the host loopback only for local bootstrap access, and cloudflared reaches it via internal DNS anyway.

OS Hardening

# tasks/os_hardening.yml (condensed)

# Scope ubuntu's sudo: removes cloud-init's dangerous NOPASSWD: ALL default
- name: Scope ubuntu sudo to required commands only
  ansible.builtin.copy:
    dest: /etc/sudoers.d/90-ubuntu-scoped
    content: |
      ubuntu ALL=(ALL) NOPASSWD: /usr/bin/docker, /usr/bin/systemctl, /usr/sbin/ufw
    mode: "0440"
    validate: "visudo -cf %s"   # validate before writing, prevents sudo lockout

# SSH hardening
- name: Harden sshd_config
  ansible.builtin.lineinfile:
    path: /etc/ssh/sshd_config
    regexp: "{{ item.regexp }}"
    line: "{{ item.line }}"
    validate: /usr/sbin/sshd -t -f %s   # test config before writing
  loop:
    - { regexp: '^#?PermitRootLogin',        line: 'PermitRootLogin no' }
    - { regexp: '^#?PasswordAuthentication', line: 'PasswordAuthentication no' }
    - { regexp: '^#?MaxAuthTries',           line: 'MaxAuthTries 3' }
    - { regexp: '^#?X11Forwarding',          line: 'X11Forwarding no' }
    - { regexp: '^#?AllowTcpForwarding',     line: 'AllowTcpForwarding no' }
    - { regexp: '^#?ClientAliveInterval',    line: 'ClientAliveInterval 300' }
    - { regexp: '^#?ClientAliveCountMax',    line: 'ClientAliveCountMax 2' }
  notify: Restart ssh

# Kernel hardening via sysctl
- name: Apply sysctl hardening
  ansible.posix.sysctl:
    name: "{{ item.name }}"
    value: "{{ item.value }}"
    sysctl_file: /etc/sysctl.d/99-hardening.conf
  loop:
    - { name: 'kernel.randomize_va_space',           value: '2' }  # ASLR
    - { name: 'net.ipv4.tcp_syncookies',             value: '1' }  # SYN flood protection
    - { name: 'net.ipv4.conf.all.rp_filter',         value: '1' }  # reverse path filtering
    - { name: 'net.ipv4.conf.all.accept_redirects',  value: '0' }  # no ICMP redirects
    - { name: 'net.ipv4.conf.all.send_redirects',    value: '0' }
    - { name: 'fs.suid_dumpable',                    value: '0' }  # no setuid core dumps
    - { name: 'kernel.dmesg_restrict',               value: '1' }  # non-root can't read dmesg
    - { name: 'kernel.kptr_restrict',                value: '2' }  # hide kernel pointers

The validate on sshd_config changes is not optional. Running sshd -t before writing the file ensures you never write a config that breaks SSH access. That validation step has saved this project from a lockout at least once.

AWS parallel: Ansible playbooks here are functionally equivalent to AWS Systems Manager (SSM) State Manager associations. SSM State Manager applies configurations on a schedule to EC2 instances, and SSM Parameter Store holds secrets the same way vars/secrets.yml (Ansible Vault) does here. The sysctl and sshd hardening tasks map directly to EC2 user data scripts or SSM Run Command documents.

Phase 3: Audit Logging

Two controls here. Both are required by PCI DSS. Both ship their output to the central observability stack.

auditd: Kernel-Level Audit (PCI Req 10.2.1)

auditd is installed on all four VMs and configured with rules that map directly to PCI DSS Req 10.2.1 sub-requirements:

# infra/ansible/playbooks/files/pci-audit.rules

## 10.2.1.6: detect tampering with the audit subsystem itself
-w /etc/audit/ -p wa -k audit_config
-w /sbin/auditctl -p x -k audit_tools
-w /sbin/auditd -p x -k audit_tools

## 10.2.1.5: identity and authentication file changes
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k identity
-w /etc/sudoers.d/ -p wa -k identity
-w /etc/ssh/sshd_config -p wa -k sshd_config

## 10.2.1.2: privileged command execution
-w /usr/bin/sudo -p x -k privileged
-w /usr/bin/su -p x -k privileged
-w /usr/sbin/useradd -p x -k privileged
-w /usr/sbin/userdel -p x -k privileged
-w /usr/sbin/usermod -p x -k privileged

## 10.2.1.3: access to the audit log itself
-w /var/log/audit/ -p rwa -k audit_log_access

## 10.2.1.4: failed login tracking
-w /var/log/faillog -p wa -k logins
-w /var/run/faillock/ -p wa -k logins

The -p wa flag watches for writes and attribute changes. -p x watches for execution. -k is the key used to filter audit events when searching logs. These map one-to-one to the SAQ A-EP sub-requirements, which matters when filling in the questionnaire; you can point to a specific rule file, not just say "yes we log things."

The Ansible task that deploys these rules uses a handler to reload them:

# tasks/auditd.yml
- name: Deploy PCI audit rules
  ansible.builtin.copy:
    src: files/pci-audit.rules
    dest: /etc/audit/rules.d/pci-dss.rules
    owner: root
    group: root
    mode: "0640"
  notify: Reload auditd rules   # handler: augenrules --load

AIDE: File Integrity Monitoring (PCI Req 11.5.2)

AIDE (Advanced Intrusion Detection Environment) runs on all four VMs. The configuration template defines what to monitor and what to exclude:

# aide.conf.j2 (relevant portions)
FULL   = p+i+n+u+g+s+b+m+c+sha256   # perms+inode+links+user+group+size+blocks+mtime+ctime+hash
BINLIB = p+u+g+sha256                # for binaries: just identity + hash
CONFIG = p+u+g+n+sha256              # for config files: + symlinks

/boot           BINLIB
/bin            BINLIB
/usr/bin        BINLIB
/usr/sbin       BINLIB
/etc            CONFIG
/opt            BINLIB     # where Docker Compose stacks live

# Exclude volatile paths: these change constantly and would flood alerts
!/var/lib/docker
!/var/log
!/var/cache
!/proc
!/run
!/tmp

The exclusions are as important as the inclusions. Without excluding /var/log and /var/lib/docker, every Docker layer pull and every log rotation would trigger a false positive. The point is detecting unexpected changes to binaries and configuration, not logging every file write.

The Ansible task initializes the baseline and installs the daily cron:

# tasks/aide.yml
- name: Initialize AIDE database (first run only)
  ansible.builtin.command: aideinit
  args:
    creates: /var/lib/aide/aide.db   # idempotent: skip if db already exists

- name: Daily AIDE integrity check
  ansible.builtin.cron:
    name: "AIDE daily integrity check"
    minute: "17"
    hour: "3"
    user: root
    job: "/usr/bin/aide --check > /var/log/aide/aide-check.log 2>&1 || true"

The cron runs at 03:17, not 03:00. Running at exactly :00 means competing with every other cron job that defaults to that time. The || true prevents a non-zero exit from AIDE (which reports differences, not errors) from looking like a cron failure.

Grafana Alloy ships the AIDE log file to Loki. Changes to monitored files show up in Grafana immediately.

AWS parallel: In AWS, file integrity monitoring at this level combines AWS Config (detects configuration drift on EC2 instances), AWS CloudTrail with CloudTrail Insights (detects unusual API patterns), and Amazon GuardDuty (anomaly detection). For a direct equivalent of AIDE's file hash comparison, Amazon Inspector also includes a reachability and package vulnerability component. None of these is a 1:1 match for AIDE's low-level file monitoring, which is why on-prem compliance often runs AIDE even in hybrid environments.

Phase 4: Identity and Access

Keycloak 26 runs on the Auth VM in production mode (start, not start-dev). Everything is configured through a bootstrap script that calls the Keycloak Admin REST API, with no manual GUI steps after the first deploy.

Why REST API instead of a realm JSON export? An exported JSON file becomes a maintenance burden. When one setting needs updating, you diff and re-import the whole thing. The bootstrap script is idempotent: it checks whether each resource exists, creates it if missing, and updates it if present. The same REALM_SETTINGS block handles both the initial create (POST) and subsequent updates (PUT).

# app/auth/bootstrap.sh: realm settings (key security parameters shown)
REALM_SETTINGS=$(cat <<EOF
{
  "realm": "workup",
  "enabled": true,
  "accessTokenLifespan": 300,
  "ssoSessionIdleTimeout": 900,
  "ssoSessionMaxLifespan": 28800,
  "verifyEmail": true,
  "passwordPolicy": "length(12) and upperCase(1) and lowerCase(1) and digits(1) and specialChars(1) and notUsername and passwordHistory(5)",
  "bruteForceProtected": true,
  "failureFactor": 5,
  "maxFailureWaitSeconds": 1800,
  "smtpServer": {
    "host": "smtp.resend.com",
    "port": "465",
    "ssl": "true",
    "auth": "true",
    "user": "resend",
    "password": "${RESEND_API_KEY}"
  }
}
EOF
)

These values map directly to PCI DSS requirements: password complexity maps to Req 8.3.6, brute-force protection maps to Req 8.3.4, session timeouts map to Req 8.2.8.

TOTP is configured as a required action on the realm:

curl -X PUT \
  "http://localhost:8080/admin/realms/workup/authentication/required-actions/CONFIGURE_TOTP" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "alias": "CONFIGURE_TOTP",
    "enabled": true,
    "defaultAction": true,
    "priority": 10
  }'

Setting defaultAction: true means every new user is prompted to configure an authenticator app on first login. This satisfies PCI DSS Req 8.4 (MFA for all accounts with access to the cardholder data environment).

The RBAC model is deliberately minimal. One realm role: admin. It is only meaningful in systems that read it from the JWT; currently only Grafana (role_attribute_path = "contains(realm_access.roles[*], 'admin')" with role_attribute_strict = true). An early design had four roles: admin, billing-ops, support, developer. billing-ops maps to Stripe, developer maps to GitLab. Neither system authenticates through Keycloak. Creating Keycloak roles for them produces dead configuration that looks good on paper but enforces nothing. The correct approach is to document each system's own access controls separately, which is what the evidence docs do.

Three OIDC clients are created by the bootstrap: workup-frontend (public, PKCE S256, standard flow only), cloudflare-access (confidential, used as the IdP for Cloudflare Zero Trust), and grafana (confidential, for Grafana OAuth). Direct access grants are disabled on all three.

AWS parallel: Amazon Cognito User Pools is the managed equivalent of Keycloak here. It supports OIDC/OAuth2, PKCE, built-in MFA (TOTP via software tokens), and password policies configured in the pool settings. The bootstrap.sh pattern maps to Cognito's API or Terraform's aws_cognito_user_pool resource. One meaningful difference: Cognito doesn't have an equivalent of Keycloak's Admin REST API for idempotent realm bootstrapping; you use Terraform or CDK instead, which has the same idempotency property.

Phase 5: The Payment Flow

This is the architectural core of the SAQ A-EP eligibility. The flow:

Authenticated user opens billing.html (served by nginx on the Frontend VM)
billing.js calls POST /billing/create-checkout-session with a Keycloak JWT in the Authorization header
The Go API validates the JWT, creates a Stripe Checkout Session, returns a clientSecret
billing.js calls stripe.initEmbeddedCheckout({ clientSecret }) and mounts the checkout UI
The card entry iframe is served from js.stripe.com (Stripe's servers, not WorkUp's)
The user submits card details directly to Stripe
Stripe calls the backend's webhook endpoint after payment

Step 5 is the entire basis for the N/A decisions on Req 3, 4, and 9. Card data (PAN, CVV, expiry) is submitted directly from the browser to Stripe's servers. WorkUp's servers are not in that data path.

The backend's JWT validation uses github.com/MicahParks/keyfunc/v3 to fetch and cache the JWKS from Keycloak's public endpoint:

// app/backend/internal/auth/auth.go
jwks, err := keyfunc.NewDefaultCtx(ctx, []string{
    cfg.KeycloakIssuerURL + "/protocol/openid-connect/certs",
})

token, err := jwt.Parse(tokenString, jwks.Keyfunc,
    jwt.WithValidMethods([]string{"RS256"}),
    jwt.WithIssuer(cfg.KeycloakIssuerURL),
    jwt.WithAudience(cfg.KeycloakClientID),
)

It validates algorithm (RS256), signature (against Keycloak's public keys), issuer, and audience. A rejected token is logged and counted as a Prometheus metric.

The Stripe webhook endpoint at POST /billing/webhook verifies the Stripe-Signature header before processing any event. A Cloudflare Custom Rule also blocks any POST to /billing/webhook that arrives without the stripe-signature header; unsigned payloads are rejected at the CDN edge before reaching the backend.

The database schema confirms the scope in code:

-- app/backend/db/schema.sql
CREATE TABLE workspaces (
  id            BIGSERIAL PRIMARY KEY,
  keycloak_sub  TEXT NOT NULL UNIQUE,
  name          TEXT NOT NULL
);

CREATE TABLE subscriptions (
  id                     BIGSERIAL PRIMARY KEY,
  workspace_id           BIGINT NOT NULL REFERENCES workspaces(id),
  stripe_customer_id     TEXT NOT NULL,
  stripe_subscription_id TEXT NOT NULL,
  status                 TEXT NOT NULL
);

No PAN fields. No card_last4. No CVV. No expiry date. Not even a placeholder column. This was confirmed by direct code review, not assumed from the Stripe integration model. That distinction matters when filling in the SAQ; you have to be able to point to evidence, not just assert.

The Stripe secret key in use is a Restricted key scoped to only the operations the backend performs. The project started with a Standard key and rotated to a Restricted key mid-project when the gap was identified. Standard keys grant full Stripe API access; a restricted key for checkout sessions and webhook processing is the correct least-privilege approach.

The Go backend runs in a multi-stage Docker container. Builder stage uses golang:1.27-alpine with CGO_ENABLED=0 GOOS=linux and -ldflags="-s -w" to produce a statically linked binary with no debug symbols. The final stage uses alpine:3.20 with a non-root user:

FROM golang:1.27-alpine AS builder
WORKDIR /build
RUN apk add --no-cache git ca-certificates
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /workup-backend ./cmd/server

FROM alpine:3.20
RUN apk add --no-cache ca-certificates
RUN addgroup -g 1000 workup && adduser -u 1000 -G workup -D workup
WORKDIR /app
COPY --from=builder /workup-backend /app/workup-backend
RUN chown -R workup:workup /app
ENTRYPOINT ["/app/workup-backend"]

AWS parallel: In AWS, this payment architecture would put the API behind an ALB with WAF rules (equivalent to the Cloudflare Custom Rules). The Stripe webhook signature verification is identical. Secrets (Stripe keys, Keycloak client credentials) would live in AWS Secrets Manager rather than Ansible Vault-encrypted files, with IAM roles granting the EC2 instance access. The JWT validation library and logic are the same regardless of where Keycloak runs.

Phase 6: Observability

Grafana Alloy runs on all four VMs. On the three non-observability VMs, it ships four log sources to the central Loki instance: container stdout/stderr (via Docker socket), systemd journal, auditd output, and AIDE check reports.

# app/auth/alloy-config.river (pattern used on all non-obs VMs)

# Ship container logs
discovery.docker "containers" {
  host = "unix:///var/run/docker.sock"
}

loki.source.docker "default" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.containers.output
  forward_to = [loki.write.central.receiver]
}

# Ship auditd logs
local.file_match "audit" {
  path_targets = [{"__path__" = "/var/log/audit/audit.log"}]
}
loki.source.file "audit" {
  targets    = local.file_match.audit.targets
  forward_to = [loki.relabel.audit.receiver]
}

# Ship AIDE check reports
local.file_match "aide" {
  path_targets = [{"__path__" = "/var/log/aide/*.log"}]
}
loki.source.file "aide_logs" {
  targets    = local.file_match.aide.targets
  forward_to = [loki.relabel.aide.receiver]
}

# All remote writes include Cloudflare Access service token headers
loki.write "central" {
  endpoint {
    url = "https://logs.victorojeje.xyz/loki/api/v1/push"
    headers = {
      "CF-Access-Client-Id"     = env("CF_ACCESS_CLIENT_ID"),
      "CF-Access-Client-Secret" = env("CF_ACCESS_CLIENT_SECRET"),
    }
  }
}

The logs.victorojeje.xyz URL is a Cloudflare Tunnel that reaches Loki on the observability VM. The CF_ACCESS headers carry a Cloudflare service token that authenticates Alloy to the tunnel; random internet traffic trying to write to Loki over that URL gets rejected.

The observability VM's own Alloy writes locally (same Docker network as Loki and Prometheus). No CF_ACCESS headers needed.

Loki's retention is set to exactly 365 days:

# app/observability/loki-config.yaml
limits_config:
  retention_period: 8760h   # 365 days: PCI Req 10.5.1 requires 12 months minimum

compactor:
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h

8760h is 365 days. Not "approximately 12 months": exactly 365 days. PCI Req 10.5.1 requires 12 months. Any rounding that lands below 365 is a compliance gap.

Grafana is accessed via Cloudflare Access Auth Proxy, not a second OIDC login. When a user hits grafana.victorojeje.xyz, Cloudflare Access authenticates them (via Keycloak or OTP). If they pass, Cloudflare forwards the request to Grafana with the Cf-Access-Authenticated-User-Email header set. Grafana reads that header and signs the user in:

# app/observability/docker-compose.yml
grafana:
  environment:
    GF_AUTH_PROXY_ENABLED: "true"
    GF_AUTH_PROXY_HEADER_NAME: "Cf-Access-Authenticated-User-Email"
    GF_AUTH_PROXY_AUTO_SIGN_UP: "true"
    GF_AUTH_DISABLE_LOGIN_FORM: "true"
    GF_AUTH_ANONYMOUS_ENABLED: "false"
    GF_AUTH_BASIC_ENABLED: "false"

The earlier implementation used Keycloak OIDC for Grafana. It caused a double-login: Cloudflare Access prompts for credentials, then Grafana's OIDC redirect triggers another Keycloak login. Auth Proxy eliminates the redundant prompt while keeping the access gate at the Cloudflare edge.

AWS parallel: Amazon CloudWatch Logs is the managed equivalent of Loki here. Set the log group retention policy to 365 days (aws logs put-retention-policy --log-group-name /workup/audit --retention-in-days 365). AWS CloudTrail provides API-level audit logging equivalent to what auditd captures at the OS level. Amazon OpenSearch Service (or the Grafana integration with CloudWatch) replaces the local Grafana + Loki stack. The Grafana Auth Proxy pattern maps to Cognito-based authentication in AWS CloudWatch dashboards or directly in managed Grafana on AWS.

Phase 7: CI/CD Security

GitHub is the code source of truth. On every push, a GitHub Actions workflow mirrors the code to GitLab, where the full CI pipeline runs.

The mirror strips .github/ before pushing; GitHub Actions config doesn't belong on GitLab:

# .github/workflows/mirror-to-gitlab.yml
- name: Push to GitLab without .github
  run: |
    git checkout -b mirror-temp
    git rm -r .github
    git commit -m "temp: remove .github for GitLab mirror"
    git push gitlab mirror-temp:main --force

The GitLab CI pipeline runs five jobs across three stages:

# .gitlab-ci.yml (structure)
stages: [test, build, scan]

include:
  - template: Security/SAST.gitlab-ci.yml        # Semgrep: Go source analysis
  - template: Security/Secret-Detection.gitlab-ci.yml

govulncheck:
  stage: test
  image: golang:1.27
  script:
    - go install golang.org/x/vuln/cmd/govulncheck@latest
    - govulncheck ./...    # checks Go deps against the Go vuln database

build-backend:
  stage: build
  image: docker:24.0
  services: [docker:24.0-dind]
  script:
    - docker build --tag "$BACKEND_IMAGE:$CI_COMMIT_SHORT_SHA" --tag "$BACKEND_IMAGE:latest" \
        -f app/backend/Dockerfile app/backend
    - docker push "$BACKEND_IMAGE:$CI_COMMIT_SHORT_SHA"
    - docker push "$BACKEND_IMAGE:latest"
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

trivy-container:
  stage: scan
  needs: [build-backend]
  script:
    - trivy image --severity HIGH,CRITICAL \
        --format json --output gl-container-scanning-report.json \
        "$BACKEND_IMAGE:$CI_COMMIT_SHORT_SHA"
  artifacts:
    paths: [gl-container-scanning-report.json]
    expire_in: 30 days

govulncheck is specifically for Go. It checks the module's dependencies against the Go vulnerability database (vuln.go.dev), not just generic CVEs. A dependency can appear clean in a general scanner like Trivy but show up in govulncheck because the vulnerability is in a code path the module actually calls.

The built image is stored in the GitLab Container Registry. The Backend VM pulls it at deploy time using a GitLab deploy token scoped to read_registry only (not a personal access token, not the CI runner token).

AWS parallel: This pipeline maps to AWS CodePipeline triggering CodeBuild on push. The govulncheck step becomes a CodeBuild phase. The Docker build and push targets Amazon ECR. Trivy is replaced by Amazon Inspector V2, which integrates natively with ECR and scans images on push. GitLab's Secret Detection maps to AWS CodeGuru Security's secret detection. The registry pull uses an IAM role attached to the EC2 instance with ecr:GetAuthorizationToken and ecr:BatchGetImage permissions, equivalent to the scoped deploy token.

What the SAQ A-EP Actually Required

The SAQ itself is a 100-page questionnaire. Most of it is not surprising if you understand the scope. A few decisions worth explaining:

Req 3, 4, 9: Not Applicable.

These requirements cover protecting stored card data, card data in transit, and physical access to card data. None of them apply because WorkUp never has card data. The N/A justifications are written documents (docs/evidence/na-justifications.md) that explain why each requirement doesn't apply, not just check a box.

Req 5: Anti-malware, Not Applicable (justified).

PCI DSS Req 5.1 allows anti-malware to be declared not applicable for systems that are "not commonly affected by malware." These VMs have no interactive logins (SSH from management only), no browsers, no file uploads, and no email. They run containerized workloads with images scanned by Trivy in CI. The N/A justification documents the reasoning.

Req 6.4.3: Payment page scripts.

This requirement asks for a list of all scripts on the payment page, the purpose of each, and either an integrity mechanism or a written justification for scripts that don't support it. The payment page (billing.html) loads five scripts. Four are first-party. The fifth is https://js.stripe.com/v3/. Stripe.js explicitly does not support SRI (Subresource Integrity); their documentation states the script updates dynamically for fraud detection. The justification for the SRI exception is Stripe's own PCI Level 1 certification status. All five scripts are inventoried in docs/evidence/payment-page-script-inventory.md.

Req 8.4: MFA for all access to the cardholder data environment.

TOTP via Keycloak covers human users. Cloudflare Access provides OTP as a backup login method. Machine-to-machine access (Alloy to Loki/Prometheus, backend to Keycloak JWKS endpoint) uses service tokens and JWT validation respectively.

Things That Went Wrong

firewall = true missing on VM network interfaces.

The auth, backend, and observability VMs were defined in Terraform without firewall = true on their network_device blocks. The Proxmox firewall rules for those VMs existed in Terraform state but were never evaluated. Caught when checking the Proxmox GUI and noticing the shield icon was greyed out on three of four VMs.

Docker published ports bypassing ufw.

An early version of the backend's docker-compose.yml had ports: ["8081:8081"] on the api service. Port 8081 was reachable from the LAN even with ufw default deny incoming. Docker writes iptables PREROUTING rules that the INPUT chain (where ufw operates) never sees. Fix: remove the ports: block entirely.

Grafana double-login.

Using both Cloudflare Access (with Keycloak as IdP) and Grafana's own Keycloak OIDC integration created two separate login prompts. The fix was to disable Grafana's own OAuth and switch to Auth Proxy mode, where Cloudflare passes the authenticated user's email in a trusted header and Grafana reads it directly.

Cloudflare Access session duration set to "No duration, expires immediately" on the Grafana application.

This is correct for machine-to-machine service token applications. It is wrong for user-facing applications. Every page load triggered a Cloudflare Access re-authentication. Setting the session to 24h fixed it.

Postgres 18 volume mount path changed.

The volume for Postgres 18 mounts at /var/lib/postgresql (not /var/lib/postgresql/data as in earlier versions). Using the old path created a nested directory structure where Postgres thought the data directory was uninitialized on every container restart.

.gitignore accidentally hiding non-secret variable files.

An early .gitignore entry used vars/*, which matched management.yml and os_hardening.yml in addition to secrets.yml. Those files were never committed, and Ansible runs that referenced them failed silently. Changed to vars/secrets.yml specifically.

The Result

One completed SAQ A-EP, one signed AOC. The infrastructure runs. The audit trail is in Loki. AIDE runs nightly on all four VMs. The CI pipeline scans every push. The payment flow routes card data through Stripe without touching WorkUp.

The point of building this was not to demonstrate that you can pass a PCI audit. The point was to demonstrate that compliance is an architectural decision, not a retrofit. When the architecture is correct, when card data genuinely never reaches your servers, the audit becomes a documentation exercise rather than a gap-remediation project.

If you want to see the implementation, the repo is at github.com/escanut/on-prem-workup-pci-dss.

Victor Ojeje is a Cloud and DevOps Engineer based in Lagos. He holds AWS SAA-C03, CompTIA Security+, and CEH certifications.

LinkedIn | Dev.to | ojejevictor@gmail.com

PayCore v2: I Removed My Own High Availability, Added a Ledger, and Finally Added Monitoring

Victor Ojeje — Thu, 18 Jun 2026 11:49:39 +0000

What was wrong with v1?

Three things.

The networking was over-engineered. I built a two-node WireGuard setup where a CloudWatch alarm would trigger a Lambda to move an Elastic IP from the primary EC2 node to a standby node on failure. Looked great on paper. Then I found a gap I had documented myself: the failover Lambda moved the Elastic IP but never updated the VPC route table. The route table still pointed at EC2 node 0's ENI after failover. That means traffic from Lambda would still hit the old interface. What I had was not high availability. It was a more complicated single point of failure.

The application had no real financial data model. PayCore had one table: transaction. You could queue a payment, Lambda would validate it, status would get patched back. That demonstrates async processing. It does not demonstrate a payment backend. A payment backend needs accounts, ledger entries, and idempotency. Without those three things you are building a webhook relay, not a financial system.

There was no observability. I could deploy the thing but I could not show I was able to operate it. Every Cloud and SRE role I have applied to at financial institutions, telcos, or larger tech companies lists Prometheus and Grafana. My project had nothing.

v2 fixes all three.

The networking got simpler

I removed the dual-EC2 failover setup entirely. PayCore now runs a single EC2 instance as the WireGuard gateway, one public subnet, one route table. The Elastic IP is still there for address stability. There is no failover automation.

This is the honest call. At homelab scale, the two-node setup added operational complexity without actually delivering HA because of the route table gap. At real enterprise scale, the right answer is AWS Site-to-Site VPN or AWS Transit Gateway with redundant tunnels. Both handle route propagation automatically. AWS Site-to-Site VPN is around $0.05 per hour per connection, roughly $36 a month. The cost of managing WireGuard HA yourself, patching it, monitoring tunnel health, scripting route updates, and testing failover paths, does not make sense when a managed service exists that handles all of that.

I am genuinely curious how people running hybrid infrastructure in financial services or telco handle this in practice. If you have done it at scale, I want to know what the actual operational overhead looked like.

The app now has a proper financial data model

Three tables got added: accounts, ledger_entry, and idempotency_keys.

accounts is scoped per merchant per currency. Before a merchant can submit an NGN payment, they need an NGN account. Currency isolation enforced at the data layer, not just the API.

ledger_entry records every credit and debit against an account. Balance is computed by summing across entries, not stored as a column. Storing a running balance as a column creates race conditions under concurrent writes and makes it harder to audit individual movements.

def get_balance(db: Session, account_id: str):
    rows = db.execute(
        select(LedgerEntry).where(LedgerEntry.account_id == account_id)
    ).scalars().all()

    balance = 0.0
    for row in rows:
        if row.movement_direction == LedgerMovement.credit:
            balance += row.amount
        else:
            balance -= row.amount

    return balance

idempotency_keys stores the response body against a client-supplied header. If the same key arrives twice from the same merchant, the stored response comes back without touching the database a second time. Network retries are real. A duplicate charge is a support incident. This is not optional in payment systems.

Prometheus and Grafana, finally

Four exporters running in the same Docker Compose stack:

prometheus-fastapi-instrumentator instruments the FastAPI app at startup and exposes /metrics. Prometheus scrapes it every 10 seconds. You get request count, latency histograms, and error rates per endpoint without writing custom metric code.

node-exporter for host-level metrics. CPU, memory, disk, network. It runs with pid: host and sees the host's process tree from inside the container.

cAdvisor (Container Advisor) for per-container resource usage. Reads directly from the Docker daemon.

postgres-exporter for database internals. Connection counts, query duration, table sizes.

Grafana is provisioned entirely through code. Datasource and dashboard JSON are mounted via Docker volumes at startup. No manual UI setup. Tear it down, redeploy, dashboards come back.

volumes:
  - grafana_data:/var/lib/grafana
  - ./monitoring/grafana/provisioning:/etc/grafana/provisioning

Four alerts in alerts.yml: API down, PostgreSQL down, host CPU above 85% for five minutes, disk below 15% free. Alertmanager is not deployed in the dev setup so these do not deliver anywhere right now. In production this routes to PagerDuty or SNS.

The reason I cared about this specifically: most of the Cloud and SRE roles I have looked at in finance, oil and gas, and telco all list observability. The day to day work in those environments is not architecture. It is watching dashboards, chasing slow queries, correlating an API error spike with a disk event. This stack is proof I can do that, not just deploy infrastructure.

What is still not production-ready

Terraform state is local, on disk. Production uses S3 backend with DynamoDB locking.
No alertmanager target configured as to keep things simple.
No settlement layer. The ledger tracks movements. There is no reconciliation job, no settlement engine, no balance sheet. The goal of this project was the infrastructure, implementing the ledger application layer was a good opportunity to delve deeper than I normally would.
Secrets Manager recovery window is 0. Fast teardown for dev. Production minimum is 7 days.

Code is public: github.com/escanut/paycore-hybrid-payment-infra

Victor Ojeje, Cloud and DevOps Engineer

LinkedIn | Dev.to | ojejevictor@gmail.com

I Built a Hybrid Cloud Payment Backend: WireGuard, FastAPI, AWS Lambda, and a Proxmox VM

Victor Ojeje — Sun, 24 May 2026 18:46:39 +0000

It has been a while since I posted here. One of my previous article on GitHub Actions OIDC got picked up and reposted by Dev.to's own X account, which I did not expect. That was a good signal to keep writing, but between job applications, AWS SAA-C03 prep, and a few client things, the writing fell off.

The reason I am back now is a specific realisation I had while reviewing my own portfolio: every single project I have built is greenfield AWS. Clean VPCs, Lambda functions, managed services from day one. There is nothing wrong with that, but the real world is not greenfield. Most companies are running something on-premises, in a data centre, or on a server that someone's CTO refused to migrate five years ago. If I cannot architect across that boundary, I am limited.

So I built Paycore: a hybrid cloud B2B payment middleware that runs a FastAPI application on an on-premises Proxmox VM, bridged to AWS serverless infrastructure via WireGuard VPN. This is not a tutorial scaffold. The infrastructure, secrets management, networking, and deployment pipeline are all production-pattern. I will walk through the architecture, the security decisions, and the parts I deliberately left imperfect and why.

What This Project Actually Is

PayCore accepts payment requests from merchants over HTTPS, tokenises the card number, writes the transaction to a PostgreSQL database, queues it to AWS SQS (Simple Queue Service), and returns a queued status immediately. An AWS Lambda function picks up the message, archives the raw payload to S3, runs fraud detection logic, and PATCHes the transaction status back to the on-premises API over the VPN tunnel.

The on-prem VM has no public IP. AWS Lambda never touches the public internet to reach the API. The only path in is the WireGuard tunnel through the EC2 nodes acting as VPN gateways.

This is a hybrid cloud architecture of the kind most enterprises actually run, not the simplified "everything in AWS" patterns you see in certifications. It covers secure cross-boundary networking, IAM (Identity and Access Management) scoping, event-driven async processing, secrets management, and automated failover. The constraints are real: PAN tokenisation so raw card numbers never hit the database or message queue, callback authentication to prevent spoofed status updates, and a KMS (Key Management Service) Customer Master Key encrypting both S3 objects and Secrets Manager entries.

One core problem i encountered was getting Lambda to reach a private API on a Proxmox VM with no public IP. The solution is a two-node WireGuard setup where EC2 instances act purely as VPN gateways in which no application logic runs on EC2. Lambda sits inside a VPC (Virtual Private Cloud) with a route table entry pointing 10.10.0.0/24 at EC2 node 0's ENI (Elastic Network Interface). The API port binds exclusively to the WireGuard interface on the VM.

Network Topology

A merchant sends an HTTPS request which is first handled by Cloudflare. The request passes through a Cloudflare Tunnel into an Nginx reverse proxy, which forwards it to a FastAPI application running on a Proxmox virtual machine with no public IP address.

When the FastAPI service processes the request, it publishes a message to Amazon Simple Queue Service (SQS). This triggers an AWS Lambda function running inside a Virtual Private Cloud (VPC).

The Lambda function needs to communicate back to the FastAPI service on the Proxmox VM. To achieve this, the VPC route table contains a rule that directs traffic for the subnet 10.10.0.0/24 to the Elastic Network Interface (ENI) of the currently active EC2 instance.

This EC2 instance acts as a WireGuard gateway. Traffic from Lambda is routed to this instance, then forwarded through a WireGuard tunnel to the Proxmox VM. The FastAPI service receives the request and processes the PATCH call to update the transaction status.

Failover Mechanism

Two EC2 instances are configured as WireGuard gateways:

EC2 node 0 is the primary and holds the Elastic IP
EC2 node 1 is the standby and does not hold the Elastic IP unless failover occurs

Amazon CloudWatch monitors the health of the primary node. When it detects two consecutive failed health checks, it triggers a failover Lambda function.

This function detaches the Elastic IP from the primary node and attaches it to the standby node. Because the WireGuard configuration uses the Elastic IP as its endpoint, the tunnel automatically reconnects to the new active node.

The VPC route table does not change during this process. It continues to route traffic to the same ENI target logic, while the public-facing identity shifts between EC2 instances.

Key Design Behavior

The system separates two concerns:

Private routing inside the VPC remains constant and does not require updates during failover
Public connectivity is abstracted through the Elastic IP, which can be reassigned between instances

This design avoids modifying route tables during failure events and allows faster recovery since only the public endpoint changes while internal routing stays stable.

Node Roles

Proxmox VM at 10.10.0.1 runs the FastAPI application and initiates the WireGuard tunnel
EC2 node 0 at 10.10.0.2 acts as the primary WireGuard gateway with the Elastic IP
EC2 node 1 at 10.10.0.3 acts as the standby gateway and takes over during failover

Security Decisions Worth Explaining

PAN Tokenisation

The PAN (Primary Account Number, the 16-digit card number) is tokenised on arrival before it touches the database or SQS. What gets stored and queued is a UUID token and a masked string like **** **** **** 1111. The raw number is used once, validated, masked, and discarded. This is the minimum floor for anything handling card data.

def tokenise_pan(pan: str) -> tuple[str, str]:
    token = str(uuid.uuid4())
    masked = "**** **** **** " + pan[-4:]
    return token, masked

Dual Authentication: JWT and API Key

Two separate auth flows exist intentionally. Merchant-facing endpoints use JWT (JSON Web Token) with HS256, issued on login and validated via FastAPI dependency injection. The Lambda callback endpoint (PATCH /transactions/{token}/status) uses a separate X-API-Key header, validated against a value fetched from AWS Secrets Manager at Lambda startup.

These are intentionally different because they protect against different threats. A compromised JWT does not give an attacker the ability to spoof transaction statuses. A leaked API key cannot be used to enumerate merchant transactions.

bcrypt_sha256 Instead of bcrypt

Standard bcrypt silently truncates passwords at 72 bytes. A password longer than 72 characters hashes identically to its first 72 characters. bcrypt_sha256 from passlib pre-hashes with SHA-256 before bcrypt, eliminating the truncation issue entirely. This is a real vulnerability, not a theoretical one. It affects production systems using bcrypt with long or Unicode-heavy passwords.

IAM Scoping

The Lambda validator role has explicit, resource-scoped permissions:

SQS permissions scoped to the single queue ARN
S3 PutObject scoped to arn:aws:s3:::${bucket_name}/transactions/*
Secrets Manager access scoped to paycore/internal/config*

The EC2 permissions (DescribeInstances, DescribeNetworkInterfaces) are wildcarded. That is a known gap, documented in the README. Those should be scoped to specific resource ARNs in production.

KMS Encryption

A single Customer Master Key (CMK) with key rotation enabled covers both S3 server-side encryption and Secrets Manager entries. The KMS key policy explicitly grants decrypt access to the Lambda IAM role, rather than relying on root account inheritance. The S3 bucket also has bucket_key_enabled = true, which caches the data key at the bucket level and reduces KMS API call volume against your monthly quota.

The Deployment Pipeline

Provisioning runs in three layers.

1. Build the Proxmox golden image

A cloud-init Ubuntu 22.04 VM is provisioned with Ansible: Docker Engine, Docker Compose plugin, WireGuard tools, SSH hardening, QEMU guest agent. Once provisioned, it is converted to an immutable Proxmox template with qm template. Any VM cloned from that template is pre-hardened and Docker-ready without running a single extra apt command.

2. Provision AWS infrastructure

Terraform manages the full AWS side: VPC, subnets, route tables, security groups, EC2 WireGuard nodes, Elastic IP, SQS queue with DLQ (Dead Letter Queue), SNS (Simple Notification Service) topic, S3 bucket, KMS key, Lambda functions, Secrets Manager secrets, and CloudWatch alarm. The layout is modular across five directories: networking, compute, kms, messaging, storage. Each module has its own variables and outputs. aws-infra/dev/main.tf wires them together.

3. Configure on-prem and deploy the app

A single Ansible playbook pair (configure.yml then deploy_app.yml) handles WireGuard setup on the VM, fetches secrets from Secrets Manager, git-pulls the repository, and brings up the Docker Compose stack. The final health check hits http://10.10.0.1:80/api/health over the VPN tunnel. The playbook does not exit clean until that endpoint returns 200.

Fraud Detection Flow

Lambda processes one SQS record per invocation (batch_size = 1). Before any fraud check runs, the raw payload is archived to S3. The archive happens before the decision, not after, so both flagged and processed transactions have an immutable record regardless of outcome.

def check_fraud(body: dict):
    if body["amount"] > 10000000:  # 10M NGN threshold
        return True
    if body.get("currency") not in ["NGN", "USD", "EUR"]:
        return True
    return False

If fraud is detected, Lambda publishes to the SNS topic (email alert) and PATCHes the status to flagged. Otherwise it PATCHes processed. The SQS queue has a 3-retry DLQ policy. If Lambda fails three times on a given message, the message moves to the DLQ for investigation rather than being silently dropped.

What I Deliberately Left Imperfect

Honest projects document their gaps. These are in the README and I will repeat them here.

Local Terraform state. dev.tfstate is on disk. Production requires an S3 backend with DynamoDB state locking to prevent concurrent apply conflicts.

Route table failover gap. The failover Lambda moves the Elastic IP to the standby EC2 node, but the VPC route table entry still points at EC2 node 0's ENI. Full failover requires additional Lambda logic to update the route table. There is currently a window where the Elastic IP is on node 1 but VPN traffic is still routing through node 0's interface.

S3 Object Lock disabled. force_destroy = true for fast teardown in dev. Production would use COMPLIANCE mode with a minimum 365-day retention period, consistent with PCI DSS Requirement 10.7 on audit log retention.

No ledger engine. PayCore validates and processes payment events. It does not maintain merchant balances, settlement records, or fund movement. A production payment platform needs double-entry bookkeeping, reconciliation, and settlement logic sitting on top of this.

Secrets Manager recovery window is 0. Immediate deletion enabled for fast iteration in dev. Production minimum is 7 days.

What the Stack Demonstrates

This project covers hybrid cloud networking (WireGuard VPN bridging AWS and on-prem), event-driven async processing (SQS to Lambda with DLQ), layered authentication (JWT plus API key separation), KMS-backed secrets management, immutable infrastructure patterns (Proxmox golden image via cloud-init and Ansible), and automated failover via CloudWatch alarms. The architecture decisions are reasoned out, and the gaps are explicitly documented, not hidden.

The code is public: github.com/escanut/paycore-hybrid-payment-infra

Victor Ojeje, Cloud and DevOps Engineer LinkedIn | Dev.to | ojejevictor@gmail.com

I Passed AWS SAA-C03. Here's What Actually Mattered.

Victor Ojeje — Fri, 03 Apr 2026 21:16:46 +0000

I sat the SAA-C03 recently and passed. Before I forget what actually worked, here's the honest breakdown.

The exam is not a memory test

AWS publishes the exam guide openly. Four domains, weighted percentages: Secure Architectures is 30%, Resilient Architectures 26%, High-Performing 24%, Cost-Optimized 20%. The questions drop you inside a scenario with a compliance constraint, a budget ceiling, or a failure condition, then ask you to pick the right architecture.

If you study by memorising service features, you will fail on the scenario questions because you have never practiced making the decision. I knew this going in because I had already built a fraud detection pipeline with SQS, Lambda, DynamoDB, and SNS. When a question asked me to choose between Lambda and Fargate for a PCI-constrained workload, I had a real reference point. That matters more than any flashcard.

Tutorials Dojo was the core resource, not a supplement

I used Jon Bonso's practice exams as the primary study mechanism, not a final check before sitting. The reason it works is the explanations. When you get a question wrong, it does not just show you the correct answer. It tells you exactly why each wrong answer is wrong.

That distinction is the whole exam. You are not picking the right option out of five. You are eliminating three plausible options. If you do not understand why the wrong ones fail, you will keep making the same mistakes on different question wording.

Community benchmark on Tutorials Dojo is consistent 80% before sitting the real exam. I used that as my gate.

Every wrong answer became an architecture pattern

When I got a question wrong I did not just note the correct service. I mapped the whole scenario.

Wrong on a DynamoDB composite key question? That becomes a working mental model for when a Global Secondary Index is the actual fix versus just restructuring the partition key. Wrong on an S3 lifecycle transition? That maps to the 30-day minimum billing rule for Standard-IA and why an aggressive lifecycle policy can cost more than doing nothing and oh boy does the exam love S3.

You build a schema that generalises to questions you have never seen. That is the only way to handle the 65 questions in 130 minutes without running out of time on edge cases.

Breadth over depth, done deliberately

SAA-C03 tests roughly 35 to 40 services with meaningful depth. Going deep on everything is a waste. Going shallow on the high-weight domains is how people fail.

Secure Architectures is 30% of the exam. If you cannot immediately distinguish when to use a Customer Managed KMS key versus an AWS Managed Key for an audit requirement, you are losing points on the biggest single domain. I spent more time on IAM policies, KMS, VPC endpoints, and CloudTrail than on anything else, specifically because the exam weights said to.

What I would skip next time

I spent too long on ECS versus EKS distinctions early on and very hyper focused portions of services like lambda, Eventbridge and Config for example. That is not where the exam goes. The exam goes into architectural tradeoffs including Multi-AZ RDS versus Read Replicas versus Aurora Global Database. That type of decision appears repeatedly.

If you're preparing for SAA-C03, start with Tutorials Dojo's AWS Certified Solutions Architect Associate Practice Exams from day one, not after you've finished a course. The feedback loop from testing yourself early is faster than any other method.

Follow me on Reddit: reddit.com/user/Escanut/
GitHub: github.com/escanut

How I Built a PCI-Ready Merchant Onboarding API on AWS for Under $5/Month

Victor Ojeje — Sun, 22 Mar 2026 17:40:23 +0000

Payment processors get rejected from enterprise contracts because it wasn't auditable. PCI DSS cares whether every data access event is logged, every record is encrypted with a key you control, and whether you can recover a merchant record to a specific second if a dispute arises.

This post walks through a serverless merchant onboarding API built to those standards, with full cost transparency.

What This Solves

A payment processor needs to register merchants: collect business details, store KYC documents, and expose that data to internal admin systems. Straightforward on the surface. The compliance overhead is where most implementations fall apart:

PCI DSS Requirement 3.5.1 — stored data must be encrypted with a key the organization controls, not a cloud-provider default
PCI DSS Requirement 8 — every API call must be authenticated and tied to an identity
PCI DSS Requirement 10 — every data access event must produce audit evidence

Building infra that can survive a compliance audit without emergency rework is the goal to strive for.

Architecture

The full flow:

Merchant authenticates with email and password via Amazon Cognito, receives a JWT (JSON Web Token)
Merchant calls API Gateway with that JWT in the Authorization header
API Gateway validates the JWT against Cognito ensuring Lambda never receives an unauthenticated request
Lambda processes the request and creates merchant profile in DynamoDB, stores KYC documents in S3 if present
DynamoDB stores profiles encrypted at rest with a customer-managed KMS key, PITR enabled
S3 stores KYC documents encrypted with SSE-KMS, Bucket Key enabled
CloudTrail captures every API call, Lambda invocation, and DynamoDB operation as mandated by PCI DSS Requirement 10
CloudWatch monitors operational metrics

Assumptions and Traffic Model

Dimension	Value	Basis
Monthly Active Users (MAU)	10,000	Initial operating scale
Projected new merchants/month	500	Growth estimate
DynamoDB writes/month	500	One write per new merchant registration
DynamoDB reads/month	330,000	10,000 users × 3 reads + 500 new records × 20 reads × 30 days
S3 PUT requests/month	2,000	500 merchants × 4 KYC documents each
S3 GET requests/month	6,000	2,000 PUTs × 3 retrieval average
API Gateway + Lambda requests	330,500	Matches read/write totals

They were derived from real behavioral assumptions: how often a merchant logs in, how often admins pull records, document retrieval patterns.

Security Decisions and Why They Were Made

1. JWT Validation at API Gateway, Not Lambda

Authentication is enforced at the API Gateway layer using a Cognito authorizer. This means Lambda is never invoked for an unauthenticated request meaning there is no compute cost and no attack surface for unauthenticated traffic.

If you push JWT validation into Lambda, you pay for Lambda invocations even for rejected requests, and you've moved your authentication boundary inward, increasing blast radius.

2. Customer-Managed KMS Key on DynamoDB (Not AWS-Owned)

The DynamoDB table uses a customer-managed KMS key

AWS-Owned Keys (the free default) fail PCI DSS Requirement 3.5.1 for a specific reason: you cannot produce key usage evidence for an auditor because you have no visibility into key operations. CloudTrail does not log AWS-Owned Key usage. Customer-managed keys generate CloudTrail entries on every encrypt and decrypt operation. That's the audit evidence.

3. S3 SSE-KMS with Bucket Key Enabled

KYC documents in S3 are encrypted with SSE-KMS. Bucket Key is enabled, which reduces the number of KMS API calls by batching envelope encryption at the bucket level rather than generating a unique data key per object.

Without Bucket Key, every S3 object PUT and GET generates a KMS API call. At 8,000 S3 requests/month, that's 8,000 additional KMS calls on top of DynamoDB operations. Bucket Key collapses that significantly.

4. Point-in-Time Recovery (PITR) on DynamoDB

PITR is enabled on the merchants table. This provides continuous backups with one-second granularity for the last 35 days.

In a payment dispute or fraud investigation, you may need to prove what a merchant record looked like at a specific timestamp. Without PITR, you either rebuild from CloudTrail (tedious) or you can't answer the question. PITR makes this a 30-second console operation.

5. GSI for Read Efficiency

The GET /merchants endpoint (admin listing) queries a Global Secondary Index (GSI) called entity-type-index rather than scanning the full table:

query_kwargs = {
    "IndexName": GSI_NAME,
    "KeyConditionExpression": Key("entity_type").eq("MERCHANT"),
    "Limit": limit,
    "ScanIndexForward": False,
}

DynamoDB Scan reads every item in the table regardless of what you need. At scale, that becomes expensive. A GSI query reads only matching items. With 300,000+ reads per month, using Scan instead of a GSI query would multiply your read capacity consumption by the full table size factor.

Cursor-based pagination is implemented using ExclusiveStartKey and base64-encoded tokens, which is the correct DynamoDB pagination pattern.

6. Duplicate Registration Prevention at the Database Level

table.put_item(
    Item=item,
    ConditionExpression="attribute_not_exists(cac_number)",
)

A merchant's CAC (Corporate Affairs Commission) registration number is unique to their business. This condition expression makes DynamoDB reject the write if a record with that CAC number already exists. This is enforced at the storage layer, not in application logic, so it cannot be bypassed by a Lambda bug or a parallel request race condition.

7. CloudTrail for PCI DSS Requirement 10

CloudTrail is configured to log management events and Lambda data events. Every PutItem, GetItem, and Query operation on the merchants table appears in CloudTrail. Every S3 object PUT and GET is logged.

PCI DSS Requirement 10.2 specifies the exact event types that must be logged: user access to cardholder data, invalid logical access attempts, use of privilege escalation mechanisms. CloudTrail at this configuration covers all of them.

The Full Lambda Function

This single function handles all three API routes. Every design decision in the security section above maps directly to something in this code.

import json
import uuid
import boto3
import os
import re
import base64
from datetime import datetime, timezone
from boto3.dynamodb.conditions import Key

dynamodb = boto3.resource("dynamodb")
TABLE_NAME = os.environ.get("TABLE_NAME", "merchants")
table = dynamodb.Table(TABLE_NAME)

# GSI (Global Secondary Index) used for GET /merchants — avoids a full table
# Scan which reads every item regardless of what you need. At 300,000 admin
# reads/month, Scan would multiply read capacity consumption by the full
# table size factor.
GSI_NAME = "entity-type-index"
DEFAULT_PAGE_SIZE = 20
MAX_PAGE_SIZE = 100
EMAIL_REGEX = re.compile(r"^[\w\.-]+@[\w\.-]+\.\w+$")


def respond(status_code, body):
    return {
        "statusCode": status_code,
        "headers": {"Content-Type": "application/json"},
        "body": json.dumps(body),
    }


def validate_post_body(body):
    required = ["business_name", "cac_number", "contact_email"]
    missing = [f for f in required if not body.get(f, "").strip()]
    if missing:
        return None, f"Missing or empty fields: {', '.join(missing)}"

    email = body["contact_email"].strip().lower()
    if not EMAIL_REGEX.match(email):
        return None, "Invalid contact_email format"

    # CAC = Corporate Affairs Commission registration number (Nigeria).
    # Exactly 6 digits — validated here before it ever touches DynamoDB.
    cac = body["cac_number"].strip()
    if not cac.isdigit() or len(cac) != 6:
        return None, "cac_number must be exactly 6 digits"

    return {
        "business_name": body["business_name"].strip(),
        "cac_number": cac,
        "contact_email": email,
    }, None


def post_merchant(event):
    try:
        body = json.loads(event.get("body") or "{}")
    except json.JSONDecodeError:
        return respond(400, {"error": "Request body must be valid JSON"})

    validated, error = validate_post_body(body)
    if error:
        return respond(400, {"error": error})

    merchant_id = str(uuid.uuid4())
    created_date = datetime.now(timezone.utc).isoformat()

    item = {
        "merchant_id": merchant_id,
        "created_date": created_date,
        "entity_type": "MERCHANT",
        "business_name": validated["business_name"],
        "cac_number": validated["cac_number"],
        "contact_email": validated["contact_email"],
        "status": "active",
    }

    # ConditionExpression enforces uniqueness at the storage layer.
    # If a record with this CAC number already exists, DynamoDB rejects
    # the write with ConditionalCheckFailedException — no application-layer
    # race condition can bypass this.
    table.put_item(
        Item=item,
        ConditionExpression="attribute_not_exists(cac_number)",
    )

    return respond(201, {"merchant_id": merchant_id, "created_date": created_date})


def get_all_merchants(event):
    params = event.get("queryStringParameters") or {}

    try:
        limit = min(int(params.get("limit", DEFAULT_PAGE_SIZE)), MAX_PAGE_SIZE)
    except ValueError:
        return respond(400, {"error": "limit must be an integer"})

    query_kwargs = {
        "IndexName": GSI_NAME,
        "KeyConditionExpression": Key("entity_type").eq("MERCHANT"),
        "Limit": limit,
        "ScanIndexForward": False,  # newest first
    }

    # Cursor-based pagination using DynamoDB's ExclusiveStartKey.
    # The cursor is base64-encoded JSON of the last evaluated key — safe
    # to pass to clients without exposing internal DynamoDB key structure.
    cursor = params.get("cursor")
    if cursor:
        try:
            exclusive_start_key = json.loads(
                base64.b64decode(cursor.encode()).decode()
            )
            query_kwargs["ExclusiveStartKey"] = exclusive_start_key
        except Exception:
            return respond(400, {"error": "Invalid cursor"})

    response = table.query(**query_kwargs)

    next_cursor = None
    if "LastEvaluatedKey" in response:
        next_cursor = base64.b64encode(
            json.dumps(response["LastEvaluatedKey"]).encode()
        ).decode()

    return respond(200, {
        "merchants": response["Items"],
        "count": len(response["Items"]),
        "next_cursor": next_cursor,
    })


def get_merchant_by_id(merchant_id):
    response = table.query(
        KeyConditionExpression=Key("merchant_id").eq(merchant_id),
        Limit=1,
    )
    items = response.get("Items", [])
    if not items:
        return respond(404, {"error": "Merchant not found"})
    return respond(200, items[0])


# Route table — static dispatch is faster than regex matching at Lambda
# invocation frequency and easier to read during a code review.
ROUTES = {
    ("POST", "/merchants"): post_merchant,
    ("GET", "/merchants"): get_all_merchants,
}


def lambda_handler(event, context):
    method = event.get("httpMethod", "")
    path = event.get("path", "")
    path_params = event.get("pathParameters") or {}

    if (method, path) in ROUTES:
        return ROUTES[(method, path)](event)

    # Dynamic route: /merchants/{id}
    if method == "GET" and path_params.get("id"):
        return get_merchant_by_id(path_params["id"])

    return respond(404, {"error": "Route not found"})

Three things worth noting in this code that matter for regulated environments:

Input validation happens before any AWS SDK call. Email format, CAC number format, and required field presence are all checked before Lambda touches DynamoDB. Malformed requests never generate KMS decrypt operations, which keeps the KMS call count accurate to your cost model.

entity_type: "MERCHANT" is set by the server, not the client. The GSI partition key cannot be spoofed by a bad actor sending a crafted payload. If it were client-supplied, a request sending entity_type: "ADMIN" could pollute the index.

The route table (ROUTES dict) replaces a chain of if/elif checks. At 330,500 invocations per month this is not a performance concern, but during a code review at a financial institution, explicit dispatch tables are easier to audit than nested conditionals.

How Authentication Was Tested

# Generate SECRET_HASH (required for Cognito app clients with secret)
import hmac, hashlib, base64
message = username + client_id
secret_hash = base64.b64encode(
    hmac.new(client_secret.encode(), message.encode(), hashlib.sha256).digest()
).decode()

# Get JWT from Cognito
aws cognito-idp initiate-auth \
  --client-id 3einfu39fvd2opcoldicn0fh2q \
  --auth-flow USER_PASSWORD_AUTH \
  --auth-parameters USERNAME=<username>,PASSWORD=<password>,SECRET_HASH=<hash>

# Call the API with the JWT
curl -X POST https://93z9qxnyv4.execute-api.us-east-1.amazonaws.com/prod/merchants \
  -H "Authorization: <IdToken>" \
  -H "Content-Type: application/json" \
  -d '{"business_name": "Jiji", "cac_number": "128957", "contact_email": "jiji@example.com"}'

# List all merchants
curl -X GET https://93z9qxnyv4.execute-api.us-east-1.amazonaws.com/prod/merchants \
  -H "Authorization: <IdToken>"

# Retrieve a specific merchant
curl -X GET https://93z9qxnyv4.execute-api.us-east-1.amazonaws.com/prod/merchants/01fcd787-47d1-4db5-a396-f2adcf0c3e18 \
  -H "Authorization: <IdToken>"

Requests without a valid JWT are rejected at the API Gateway layer before Lambda is invoked.

Cost Breakdown

Source: AWS Pricing Calculator export, 03/22/2026, us-east-1. 10,000 MAU, 330,500 requests/month.

Service	Monthly Cost	% of Bill	Notes
KMS	$2.01	41.3%	669,000 symmetric API calls
API Gateway	$1.16	23.8%	330,500 requests, avg 34KB
Lambda	$0.76	15.6%	330,500 invocations, 512MB
CloudTrail	$0.34	7.0%	Lambda data events at volume
CloudWatch	$0.32	6.6%	0.63GB log ingestion
DynamoDB	$0.24	4.9%	On-demand, PITR, KMS
S3	$0.04	0.8%	1GB storage, 8,000 requests
Cognito	$0.00	0.0%	10,000 MAU = within free tier
Total	$4.87/month

What most engineers miss about this bill:

KMS is 41% of the monthly cost at this scale. The moment you add customer-managed encryption to DynamoDB and S3 (which PCI DSS requires), KMS becomes the dominant cost driver. Every DynamoDB read and write generates KMS API calls. Without Bucket Key on S3, every object operation would add to that count.

At 10,000 MAU this is $4.87/month. At 100,000 MAU, KMS call volume scales roughly linearly with request volume. Planning for this in the initial architecture prevents a compliance requirement from arriving as a surprise bill.

Cognito at this scale is free. The first 10,000 MAU are within the AWS free tier perpetually.

What Breaks at Scale

This architecture is appropriate for the stated scale. Three things need revisiting before 10x growth:

1. CloudTrail cost at high Lambda data event volume

CloudTrail logs Lambda data events at $0.10 per 100,000 events. At 330,500 Lambda invocations, that's $0.34/month. At 3.3M invocations, it's $3.30/month. At 33M, $33/month. Worth modelling before reaching that range.

2. GSI read capacity under admin query load

300,000 admin reads per month assumes 20 reads per admin per day. If admin tooling increases query frequency, the GSI read cost scales. Add DynamoDB DAX (DynamoDB Accelerator) at that point, DAX adds $150+/month at minimum instance size.

3. KMS request limits

KMS has a default quota of 5,500 symmetric requests per second in us-east-1. At the traffic levels modelled here, this is not a constraint. At high-throughput payment processing scale, it becomes one. Request a quota increase before hitting the ceiling, not after.

Why This Architecture Over EC2-Based Alternatives

The comparable EC2 architecture (t3.micro + RDS + ALB) runs approximately $50-80/month before adding KMS, CloudTrail, and WAF. Serverless brings this to $4.87/month at 10,000 MAU.

More importantly: there are no servers to patch. A common PCI DSS finding in EC2-based environments is unpatched OS vulnerabilities. Lambda eliminates that finding category entirely with AWS managing the runtime, and the function execution environment is ephemeral.

The tradeoff is cold starts. Lambda cold starts for Python average 100-300ms. For a merchant onboarding API (not a real-time payment path), this is acceptable. For a transaction authorization endpoint, it is not.

Stack

Amazon Cognito (authentication, JWT issuance)
Amazon API Gateway (JWT validation, request routing)
AWS Lambda (Python 3.14, business logic)
Amazon DynamoDB (merchant profiles, SSE-KMS, PITR, on-demand)
Amazon S3 (KYC documents(for demonstration), SSE-KMS, Bucket Key, versioning)
AWS CloudTrail (audit logging, PCI DSS Requirement 10)
Amazon CloudWatch (operational monitoring)
AWS KMS (customer-managed encryption keys)

If you're hiring for Cloud Engineer, DevOps, or SRE roles with infrastructure security depth, I'm open to remote opportunities.

LinkedIn | GitHub | ojejevictor@gmail.com

I Built a Serverless Fraud Detection Pipeline on AWS. Here's What It Actually Costs to Do It Right.

Victor Ojeje — Tue, 17 Mar 2026 22:19:14 +0000

Most cloud portfolio projects look like this: spin up an EC2 instance, deploy a web app, take a screenshot. Done.

That tells a hiring manager you can follow a tutorial.
This is not that.

I built a real-time transaction screening pipeline modeled after how financial institutions actually handle suspicious activity routing.
The entire processing backbone costs $2.48/month to run at 500 000 transactions.
Here is how it works, why every decision was made, and what it actually costs to secure it properly.

The Problem This Solves

Payment processors and banks screen every transaction before it clears. A transaction above a defined threshold, or matching a suspicious pattern, needs to be flagged, stored, and routed to an analyst in near real-time. Latency here is not a UX problem. It is a compliance and fraud exposure problem.
The architecture needs to handle three things without failing:

Ingest transactions reliably, even under burst load
Evaluate and persist every transaction regardless of outcome
Alert immediately when a transaction crosses the threshold, with zero alert loss

A traditional EC2-based approach handles this with always-on servers. You pay for compute whether transactions are flowing or not. You manage patching, availability zones, and process monitoring yourself.

The serverless approach inverts this entirely.

Architecture

Every transaction enters through SQS. Lambda processes each message, writes the full record to DynamoDB regardless of whether it is flagged, and fires an SNS alert only if the amount exceeds the threshold. If Lambda fails to process a message three times, SQS routes it to a Dead Letter Queue for manual investigation. Nothing is silently dropped.

Lambda was deployed inside a VPC across two availability zones. All connections to SQS, DynamoDB, SNS, and CloudWatch run through VPC interface and gateway endpoints. No traffic touches the public internet at any point in the pipeline.

Lambda Function

import json
import boto3
import os
from decimal import Decimal

dyn = boto3.resource('dynamodb')
sns = boto3.client('sns')

TABLE = os.environ.get("TABLE", "fraud-detections")
SNS_ARN = os.environ.get("SNS_ARN", "arn:aws:sns:us-east-1:461840362463:fraud-alerts")

# Threshold represents configurable screening rule for the institution
THRESHOLD = 500000

def lambda_handler(event, context):
    for record in event["Records"]:
        body = json.loads(record["body"])
        txn_id = body["transaction_id"]
        amount = Decimal(str(body["amount"]))
        merchant = body["merchant"]

        flagged = amount > THRESHOLD

        dyn.Table(TABLE).put_item(
            Item={
                "transaction_id": txn_id,
                "amount": amount,
                "merchant": merchant,
                "flagged": flagged
            }
        )

        if flagged:
            sns.publish(
                TopicArn=SNS_ARN,
                Message=f"Suspicious transfer Flagged: {txn_id} — NGN {amount} — {merchant}",
                Subject="Fraud Alert"
            )
    return {'statusCode': 200}

Every transaction is written to DynamoDB first, before the flag check. This matters. In a real system you need a complete audit trail. A transaction that does not trigger an alert is not necessarily clean. It needs to exist in the record. Regulators do not accept gaps.

The threshold is a variable, not a hardcoded business rule. In production this would be pulled from a configuration store or Parameter Store, allowing compliance teams to adjust screening rules without a code deployment.

Decimal is used instead of float for the amount. DynamoDB does not accept Python floats. Using float here causes silent precision errors on large transaction amounts. This is a real production bug that shows up in systems built by engineers who have not actually handled financial data before.

Verified Results

Two transactions were sent through the pipeline.

TXN-001 crossed the NGN 500,000 threshold. The SNS alert fired immediately to email with the subject line "Fraud Alert" and the full transaction details in the message body.
TXN-002 was written to DynamoDB with flagged: false. No alert sent. Complete audit record preserved.

Both records confirmed live in DynamoDB

Email alert received from SNS Topic.

What It Actually Costs

Service	Monthly Cost
AWS Lambda (500k requests)	$1.14
Amazon SQS (1.5M requests)	$0.60
Amazon DynamoDB (1GB)	$0.56
Amazon SNS (10k alerts)	$0.18
Pipeline Total	$2.48
VPC Interface Endpoints (3x)	$43.81
Secured Total	$46.29/month

The pipeline itself costs $2.48/month to process 500,000 transactions.

The VPC endpoints cost $43.81/month.
That is 94.6% of the total bill for a security control, not compute.

This is an intentional tradeoff. Without VPC endpoints, Lambda communicates with SQS, DynamoDB, SNS, and CloudWatch over the public internet. In a financial context that is not a configuration choice. It is an audit finding. VPC endpoints keep all traffic private within the AWS network, satisfy network isolation requirements common in PCI-DSS and SOC 2 environments, and eliminate the data exfiltration risk that comes with public endpoint exposure.

If this were a cost-only conversation, you would skip the endpoints and save $43.81/month. In a production fintech environment, that decision gets flagged in the first security review.

What This Is Not

This pipeline has a single threshold rule. Production fraud detection at scale uses ML-based anomaly scoring, velocity checks across merchant categories, device fingerprinting, and graph-based relationship analysis. Those are layered on top of an event-driven backbone exactly like this one.

This is the infrastructure layer. It is the part that has to work before any model or rule engine gets plugged in. Getting this layer wrong means no amount of ML sophistication above it recovers cleanly.

Why Serverless for This

Three concrete reasons:

Burst handling without pre-provisioning. Payment transaction volume is not linear. End-of-month salary runs, Black Friday, public holiday spikes. Lambda scales to concurrency automatically. An EC2-based system requires capacity planning or auto-scaling lag measured in minutes.
No idle cost. An EC2 t3.micro running 24/7 costs approximately $7.59/month before you add storage, monitoring, or patching overhead. Lambda at 500,000 transactions costs $1.14/month. At zero transactions it costs nothing.
Operational surface reduced to code. There is no OS to patch, no SSH access to harden, no process monitor to configure. The attack surface is the IAM role and the function code. Both are auditable and version-controlled.

What Carries Into Production

The patterns here are not demo-specific:

SQS as a durable ingest buffer decouples transaction producers from the processing layer. A downstream Lambda failure does not lose the transaction.
DLQ after three failures means no silent message loss. Every failed transaction is recoverable.
VPC-private connectivity means the pipeline satisfies network isolation requirements out of the box.
IAM roles scoped to least privilege mean Lambda cannot touch anything outside its defined resource set.

These are the same architectural decisions you find in production payment infrastructure. The scale is different. The patterns are not.

Victor Ojeje is a Cloud and Infrastructure Engineer based in Lagos. He builds production-grade AWS infrastructure with a focus on security, automation, and cost-conscious design.

LinkedIn: linkedin.com/in/victorojeje | GitHub: github.com/escanut

How I went from Docker Compose to production EKS without burning AWS budget on mistakes

Victor Ojeje — Wed, 18 Feb 2026 06:12:33 +0000

The Problem with Learning Kubernetes Directly on AWS

Most tutorials for Kubernetes on AWS tell you to:

Spin up an EKS cluster
Apply your manifests
Debug why nothing works
Watch your bill climb while you figure it out

EKS charges $0.10 per hour just for the control plane, before a single EC2 node is added. That is about $72/month for a cluster sitting idle while you troubleshoot ingress routing, secret injection failures, and container networking issues.

That is fine for a company budget. It hurts when you are learning on your own.

There is a better workflow. Test locally, prove it works, then pay for AWS.

What I Built

A production-grade containerized web application with a staged deployment workflow:

Stage 1: Docker Compose for fast local iteration
Stage 2: Minikube to validate Kubernetes behavior before touching AWS
Stage 3: EKS with full CI/CD, AWS Secrets Manager, and automated ALB provisioning

App repo: https://github.com/escanut/fastapi-k8s-project

Infra repo: https://github.com/escanut/fastapi-aws-infra

The app is a FastAPI backend with async request handling and connection pooling connected to PostgreSQL, plus a vanilla JS frontend. Simple product catalog with create, retrieve, delete items. Nothing fancy on purpose. The focus is the infrastructure pattern and delivery workflow.

Stage 1: Docker Compose

Goal: instant feedback on code changes with zero cluster overhead.

services:
  postgres:
    image: postgres:15-alpine
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U admin -d products"]

  backend:
    build: ./backend
    environment:
      DB_HOST: postgres
    volumes:
      - ./backend:/app

  frontend:
    image: nginx:alpine
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf

Nginx routes /api traffic to FastAPI. Credentials are plain env variables here, intentionally different from production. That contrast is the lesson.

Validates: app logic, queries, routing, frontend-backend communication.
Does not validate: Kubernetes behavior.

Stage 2: Minikube

Goal: catch Kubernetes failures at zero cost before provisioning AWS.

minikube start
eval $(minikube docker-env)
minikube addons enable ingress
kubectl apply -f dev/k8s/

Setup mirrors production closely:

Secrets from Kubernetes Secrets, not env vars

Postgres as a pod with a PVC

Ingress configured like production

Broken ingress, wrong ports, and secret issues show up here instead of on a paid cluster. Minikube is not identical to ALB, but it is close enough to surface real problems early.

The CI/CD Pipeline

On every push to main:

  Configure AWS Credentials
  This uses aws-actions/configure-aws-credentials@v4

  Build and push frontend & backend image
  This runs docker build + push

  Updates deployment
  This runs kubectl set image ...

  Verify
  This runs kubectl rollout status ...

Each image is tagged with commit SHA. Rollback just means redeploying a previous SHA. No guesswork.

OIDC is used for short-lived credentials. No static keys stored.

VPC Endpoints: Why They Matter Beyond Security

Without endpoints:

Worker → NAT → Internet → AWS API → Internet → NAT → Worker

NAT data processing costs $0.045/GB. Image pulls, logs, and secrets traffic add up fast.

Eight endpoints route AWS-internal traffic privately. NAT remains only for external access during bootstrap.

This results to lower cost and reduced exposure.

Secrets Management Progression

Stage 1: Plain env var  
Stage 2: K8s Secret  
Stage 3: ExternalSecret synced from Secrets Manager

Production password is randomly generated and stored in Secrets Manager. Terraform state is encrypted. Password is not floating around in plaintext.

IAM Design: Least Privilege

GitHub Actions: scoped OIDC role
Node role: read-only ECR
Backend pods: read specific secret via IRSA
ALB Controller: ALB-only permissions

Each component gets only what it needs to perform its task.

Post-Install: Why Not Everything in Terraform

post-install.sh installs:

AWS Load Balancer Controller
External Secrets Operator

CRDs plus dependency ordering make Terraform Helm messy here. This approach is predictable and aligns with upstream guidance.

Infrastructure Cost Snapshot

Service	Monthly
EKS	$72
EC2 nodes	~$30
RDS	~$12
NAT	~$65
VPC endpoints	~$11.60
ALB	~$16

Endpoints reduce NAT data charges on AWS API traffic. In active pipelines, that tradeoff makes sense.

What This Demonstrates

Cost-aware Kubernetes learning path
OIDC instead of static keys
IRSA for pod-level IAM
VPC endpoints for cost and security
External Secrets integration
Fully automated ALB provisioning

Try it

Repos are public and documented. Start local, move to Minikube, then EKS.

If something breaks or you want to discuss design choices, reach out. I am always refining this setup myself.

Open to Work

Currently seeking remote roles in Cloud, DevOps, and Platform Engineering.

LinkedIn: https://linkedin.com/in/victorojeje
Email: ojejevictor@gmail.com
GitHub: https://github.com/escanut

I built auto-scaling infrastructure that actually survives failures

Victor Ojeje — Sat, 31 Jan 2026 21:19:07 +0000

Most deployment tutorials get you to "it works on my machine" status. Few show you how to build infrastructure that survives database connection failures, instance crashes during deployment, or availability zone outages.

I deployed a production-grade Flask API on AWS with auto-scaling that maintains availability through failures. Here's what separates infrastructure that works in demos from infrastructure that works in production.

What I Built

A three-tier Flask API deployment on AWS with PostgreSQL, designed to survive common failure scenarios.

Live Repository: github.com/escanut/terraform-aws-flask-oidc

Architecture:

Application Load Balancer (Multi-AZ)
        ↓
Auto Scaling Group (2-4 instances)
        ↓
RDS PostgreSQL (Multi-AZ with automatic failover)

Key Stats:

Zero-downtime deployments (50% minimum healthy capacity)
6-8 minute deployment from commit to live
380-second instance warmup prevents premature failures
Multi-AZ redundancy across database and networking
Runtime credential retrieval (no secrets in code or Docker images)
Monthly cost: $158 (us-east-1)

Health Check Integration: The Critical Path

Health checks are where most auto-scaling setups fail. You need three layers working together:

1. Docker Container Health Check

HEALTHCHECK --interval=30s --timeout=3s --start-period=40s --retries=3

40-second start period accounts for ECR image pull (15-20 seconds) and application initialization.

2. Flask Application Health Check

@app.route('/health')
def health():
    # Executes real database query
    conn = get_db()
    cur.execute('SELECT 1')
    return jsonify({'status': 'healthy'}), 200

Validates the application can actually reach RDS. An instance passing Docker checks but unable to connect to the database is useless.

3. ALB Health Check

health_check {
  healthy_threshold   = 2
  unhealthy_threshold = 3
  interval            = 10
  path                = "/health"
}

2 consecutive successes (20 seconds) confirms stability. 3 consecutive failures (30 seconds) removes flapping instances.

Deregistration delay: 30 seconds allows in-flight requests to complete before removing unhealthy instances.

Instance Warmup: Preventing Cascading Failures

This configuration prevented deployment failures:

instance_refresh {
  strategy = "Rolling"
  preferences {
    min_healthy_percentage = 50
    instance_warmup        = 380
  }
}

Instance warmup: 380 seconds (6+ minutes)

What happens during warmup:

EC2 instance launches (30-60 seconds)
User data script: apt updates, Docker install, AWS CLI install, ECR login, image pull (4-5 minutes)
Docker health checks stabilize (40 seconds)
ALB health checks pass 2 consecutive times (20 seconds)

What happens if warmup is too short (e.g., 120 seconds):

Auto Scaling Group marks instance "healthy" before Docker finishes pulling the image
ASG immediately terminates old instances
New instances aren't ready for traffic
ALB routes traffic to instances returning connection refused
502 Bad Gateway errors cascade to users

380-second warmup ensures:

Old instances stay alive until new instances are genuinely ready
Zero user-facing errors during deployments
Rolling updates complete without capacity loss

This number was determined empirically by monitoring actual instance launch times, not guessing.

Multi-AZ Architecture: Eliminating Single Points of Failure

RDS Multi-AZ Configuration:

resource "aws_db_instance" "rds" {
  multi_az              = true
  instance_class        = "db.t4g.micro"
  storage_encrypted     = true
}

Cost: $24.82/month (vs $12.41 for single-AZ)

What Multi-AZ provides:

Synchronous replication to standby in different availability zone
Automatic failover in 1-2 minutes (zero manual intervention)
Zero data loss during AZ failure

What happens during AZ failure without Multi-AZ:

Database becomes unreachable
Application returns 503 errors to all users
Manual recovery required (10-30+ minutes)
Potential data loss depending on last backup

AWS availability zone failures occur 2-3 times per year per region. Production systems justify the cost.

Dual NAT Gateways:

resource "aws_nat_gateway" "nat" {
  count = 2  # One per AZ
}

Cost: $65.70/month (vs $32.85 for single NAT)

EC2 instances in private subnets use NAT Gateways for internet access (apt updates, Docker installation, security patches). Single NAT Gateway creates single point of failure. If the NAT Gateway's AZ fails, instances in the other AZ can't bootstrap or update.

Total redundancy cost: $45/month ($12.41 for Multi-AZ RDS + $32.85 for second NAT Gateway)

This is insurance against extended outages, manual recovery procedures, and potential data loss.

Security: Runtime Credentials, Not Hardcoded Secrets

Database credentials are retrieved at runtime from AWS Secrets Manager:

secret_name = os.getenv('DB_SECRET_NAME')
client = boto3.client('secretsmanager')
secret = json.loads(client.get_secret_value(SecretId=secret_name)['SecretString'])

Where credentials are NOT:

Application code
Docker image layers
Environment variables (visible in process listings)

Where credentials ARE:

AWS Secrets Manager (encrypted at rest, access logged via CloudTrail)
EC2 instance memory only (ephemeral, destroyed on termination)

IAM instance profile grants read-only access to one specific secret. Even if an instance is compromised, the attacker can't access other secrets.

Rolling Deployment Strategy

Configuration:

min_healthy_percentage = 50

Deployment flow:

GitHub Actions builds Docker image, pushes to ECR (tagged with commit SHA)
Triggers Auto Scaling Group instance refresh
ASG replaces 50% of instances at a time
New instances pull latest image, complete 380-second warmup
ALB confirms new instances are healthy
Old instances terminated
Process repeats for remaining instances

Deployment time: 6-8 minutes

Docker build/push: 3 minutes
Rolling update with warmup: 3-5 minutes

Zero downtime: ALB continues routing traffic to healthy instances throughout deployment.

Failure Scenarios and Recovery

EC2 instance crash:

ALB health checks fail after 30 seconds
ALB stops routing traffic to failed instance
Auto Scaling Group replaces unhealthy instance
New instance completes warmup and joins rotation
Impact: Zero user-facing errors

Availability zone failure:

Multi-AZ RDS fails over to standby (1-2 minutes)
EC2 instances in failed AZ marked unhealthy
Auto Scaling Group launches replacements in healthy AZ
NAT Gateway redundancy ensures new instances bootstrap successfully
Impact: 1-2 minutes degraded performance (reduced capacity)

Docker image pull failure:

Instance warmup prevents premature traffic routing
Failed instance never receives traffic
Auto Scaling Group retries launch
Impact: Zero (deployment time increases, no user errors)

Key Lessons

Instance warmup must account for real bootstrapping time

I initially set warmup to 120 seconds. Deployments failed because Docker image pull took 40-60 seconds and health checks needed 20 seconds to stabilize. 380 seconds was determined by monitoring actual instance launch times.

Health checks must validate actual functionality

A health check that only verifies "the container is running" is useless. The /health endpoint executes a real database query. If the database is unreachable, the instance is genuinely unhealthy.

Multi-AZ costs are insurance premiums

The extra $45/month for Multi-AZ RDS and dual NAT Gateways prevents manual failover procedures, potential data loss, and extended outages. For production systems, this is a rounding error compared to downtime costs.

Conclusion

Production infrastructure is measured by how it handles failures, not how it performs in demos.

Multi-layer health checks, proper warmup periods, Multi-AZ redundancy, and zero-downtime deployments separate infrastructure that works in tutorials from infrastructure that works when databases fail, instances crash, or entire availability zones go offline.

Repository: github.com/escanut/terraform-aws-flask-oidc

Reach out: linkedin.com/in/victorojeje

I'm seeking remote opportunities in Cloud Engineering, DevOps, and Infrastructure Engineering. If your team needs someone who thinks about failure scenarios and production resilience, let's connect.

How I eliminated access keys from my deployment pipeline with OIDC, Terraform, and GitHub Actions

Victor Ojeje — Fri, 23 Jan 2026 21:45:10 +0000

The Problem with Traditional CI/CD

Most tutorials for deploying static sites to AWS tell you to:

Generate AWS access keys
Store them as GitHub secrets
Hope nobody compromises your repository

This approach has serious security implications. Long-lived credentials in CI/CD pipelines are a common attack vector since if your repository is compromised or secrets are accidentally exposed, attackers gain full AWS access with those credentials.

There's a better way: OpenID Connect (OIDC) federation.

What I Built

A fully automated static website deployment pipeline that eliminates access keys entirely by using temporary credentials exchanged through OIDC.

Source Code: GitHub Repository

Architecture Overview

Developer commits → GitHub Actions → OIDC token exchange 
→ Temporary AWS credentials → Deploy to S3 → CloudFront invalidation

Key Components:

S3 for static website hosting
CloudFront for global CDN
Terraform for infrastructure as code
GitHub Actions for CI/CD
OIDC for secure authentication

Why OIDC Over Access Keys?

Traditional Approach (Access Keys):

# Potential security risk

- name: Configure AWS Credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Problems:

Keys are long-lived (valid until manually rotated)
If leaked, provide persistent AWS access
Require manual rotation and management
Stored as static secrets in GitHub

OIDC Approach:

# This is secure

- name: Configure AWS Credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
    aws-region: ${{ secrets.AWS_REGION }}

Benefits:

Credentials expire in 1 hour (temporary)
No static secrets to leak
Zero credential rotation needed
AWS manages the trust relationship

Implementation Deep Dive

1. Setting Up the OIDC Provider in Terraform

First, create an OIDC identity provider that trusts GitHub:

resource "aws_iam_openid_connect_provider" "github" {
  url = "https://token.actions.githubusercontent.com"

  client_id_list = ["sts.amazonaws.com"]

  thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}

The thumbprint verifies GitHub's signing certificate authenticity.

2. Creating the IAM Role with Trust Policy

For this to work, we define exactly which repositories can assume this role:

resource "aws_iam_role" "github_actions" {
  name = "github-actions-s3-deployment"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = aws_iam_openid_connect_provider.github.arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
        }
        StringLike = {
          "token.actions.githubusercontent.com:sub" = "repo:${var.github_username}/${var.repo_name}:ref:refs/heads/main"
        }
      }
    }]
  })
}

Critical security feature: The StringLike condition restricts access to:

Your specific GitHub username
Your specific repository
Only the main branch

Even if someone forks your repo, they can't assume this role.

3. Least-Privilege IAM Permissions

The role only gets permissions it absolutely needs:

resource "aws_iam_role_policy" "github_actions" {
  role = aws_iam_role.github_actions.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject",
          "s3:ListBucket"
        ]
        Resource = [
          aws_s3_bucket.website_storage.arn,
          "${aws_s3_bucket.website_storage.arn}/*"
        ]
      },
      {
        Effect = "Allow"
        Action = "cloudfront:CreateInvalidation"
        Resource = aws_cloudfront_distribution.website_storage.arn
      }
    ]
  })
}

No overly broad permissions.

4. CloudFront Configuration Challenge

Here's something that tripped me up: S3 website endpoints require custom origin configuration, not standard S3 origin configuration.

# This doesn't work for S3 website endpoints

origin {
  domain_name = aws_s3_bucket.website_storage.bucket_regional_domain_name
  origin_id   = "S3-Website"

  s3_origin_config {
    origin_access_identity = aws_cloudfront_origin_access_identity.default.cloudfront_access_identity_path
  }
}

# This works for the s3 website endpoints

origin {
  domain_name = aws_s3_bucket.website_storage.website_endpoint
  origin_id   = "S3-Website"

  custom_origin_config {
    http_port              = 80
    https_port             = 443
    origin_protocol_policy = "http-only"
    origin_ssl_protocols   = ["TLSv1.2"]
  }
}

Why? S3 website endpoints function as HTTP servers, supporting redirects and error documents. Standard S3 origins access the S3 API directly and don't support these features.

5. Remote State Management

For production infrastructure, always use remote state with locking:

terraform {
  backend "s3" {
    bucket         = "escanut-tf-state"
    key            = "s3-website/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "tf-state-lock"
    encrypt        = true
  }
}

This will enable:

Team collaboration without conflicts
State encryption at rest
Prevention of concurrent modifications

The GitHub Actions Workflow

The deployment workflow is remarkably simple:

name: Deploy to S3

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write  # Required for OIDC
      contents: read

    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: ${{ secrets.AWS_REGION }}

      - name: Sync to S3
        run: |
          aws s3 sync ./build s3://${{ secrets.S3_BUCKET }} --delete

      - name: Invalidate CloudFront
        run: |
          aws cloudfront create-invalidation \
            --distribution-id ${{ secrets.CLOUDFRONT_ID }} \
            --paths "/*"

      - name: Deployment Overview
        run: |
          echo "Website deployment successful" 
          echo "https://${{ secrets.CLOUDFRONT_URL }}"

Deployment time: Approximately 30 seconds from commit to live site (measured from workflow logs).

Cost Analysis

One of the best parts? This infrastructure costs almost nothing to run.

Monthly costs (based on January 2026 AWS pricing):

Service	Usage	Cost
S3 Storage	<1GB	$0.02
S3 Requests	10k/month	$0.05
CloudFront	10GB transfer	$0.85
Total		~$0.92/month

For comparison, a single t3.micro EC2 instance costs $8-15/month and requires active management.
.

AWS's eventually consistent nature means public access settings must be applied before the bucket policy.

Security Wins

This architecture eliminates several attack vectors:

No long-lived credentials - temporary credentials expire in 1 hour

Repository-scoped access - only your specific repo can deploy

Branch-specific trust - only main branch can trigger deployments

Least-privilege permissions - role can only write to specific S3 bucket

Encrypted state - Terraform state encrypted at rest in S3

No credential rotation - AWS handles credential lifecycle automatically

The current implementation shows:

Infrastructure as Code proficiency
Cloud security best practices
CI/CD automation
Cost optimization awareness

Conclusion

OIDC federation with GitHub Actions is the modern standard for CI/CD authentication to AWS. It's more secure, requires zero maintenance, and is easier to implement than managing long-lived access keys.

If you're still using AWS access keys in your deployment pipelines, it's time to upgrade.

Try it yourself: Fork the repository, update the variables, and deploy your own infrastructure in under 5 minutes.

Drop a comment below or reach out on LinkedIn.

If this was helpful, consider giving the GitHub repo a ⭐!

Deploying Secure WordPress on AWS: A Multi-Tier Architecture Approach

Victor Ojeje — Tue, 20 Jan 2026 18:07:20 +0000

This project started as a learning exercise and ended up looking very close to how small production workloads are actually deployed. The goal was simple: run WordPress on AWS with proper network isolation, least-privilege access, and basic monitoring, without spending money or cutting corners.

Everything here was built inside Free Tier limits, but the design decisions follow the same patterns used in real environments.

Architecture Overview

I deployed WordPress using a classic two-tier setup inside a custom VPC.

Core components:

Custom VPC: 10.0.0.0/16
Public subnet: 10.0.1.0/24 for the web server
Private subnets: 10.0.10.0/24 and 10.0.11.0/24 for RDS
EC2 t2.micro running Ubuntu 24.04
RDS MySQL db.t4g.micro
Elastic IP for stable external access
Apache 2.4, PHP 8.1, WordPress

Traffic flow:

Internet → Internet Gateway → EC2 Compute (public subnet) → RDS (private subnets)

[Architecture and traffic flow]

The database has no public IP and no route to the internet. All access is controlled with security groups. I intentionally did not rely on NACLs or host firewalls to keep the design aligned with how AWS environments are usually managed.

Security Implementation

Security Groups

The security group setup is minimal and intentional.

Web tier security group:

SSH (22): allowed only from my admin IP
HTTP (80) and HTTPS (443): open to the internet

Database tier security group:

MySQL (3306): allowed only from the web tier security group

No IP-based rules between tiers. Only security group references.

[EC2 security group details]

[RDS security group details]

This matters. The first version used the EC2 private IP as the database allow list. That failed immediately after a reboot when the IP changed. Switching to security group IDs fixed the issue permanently and reflects how AWS expects environments to scale and change.

WordPress Hardening

I applied basic but meaningful WordPress security controls:

Disabled file editing in the admin dashboard
Limited login attempts to reduce brute-force risk
Disabled XML-RPC
Enforced sane file permissions (755 for directories, 644 for files)
Enabled HTTPS using a self-signed certificate

Let's Encrypt does not issue certificates for AWS public DNS names. Using a self-signed cert keeps traffic encrypted while avoiding extra costs. Which is fine in this case, as its a demo, not production.

Monitoring and Alerting

Both EC2 and RDS are monitored with CloudWatch.

Alarm configuration:

CPU utilization threshold: 70 percent
Action: SNS email notification

I chose 70 percent instead of waiting for 80 to 90 percent. On small instance types, sustained CPU spikes quickly translate into performance problems.

[CloudWatch dashboard]

[CloudWatch alarms]

During normal usage, EC2 CPU sat around 12 to 18 percent. During load testing, alarms triggered correctly and SNS emails arrived within about 2 to 3 minutes. That confirms the alerting path actually works, not just that it exists.

Real Problems Encountered

Security Group Misconfiguration

Using IP addresses for internal trust broke the moment the instance restarted. This is a common mistake in AWS environments and a good lesson. In AWS, security groups are the trust boundary, not IPs.

WordPress URL Handling

WordPress stores absolute URLs inside the database. Changing the Elastic IP meant internal links broke even after updating wp-config.php.

The fix required a full database search and replace using WP-CLI. This isn't documented in most WordPress migration guides but breaks production migrations regularly.

Free Tier Cost Awareness

AWS does not stop you from spending money. An unattached Elastic IP started generating charges within hours.

Billing alarms should be set up immediately, even for learning projects. Free Tier does not mean free by default.

Design Trade-offs

Single-AZ RDS

Multi-AZ was skipped to stay within Free Tier. This means backups exist, but there is no automatic failover. Recovery requires restoring from a snapshot and takes several minutes. That is acceptable here, not for systems with tight recovery objectives.

Apache instead of Nginx

Apache was chosen for simplicity and WordPress compatibility. Nginx would scale better under high concurrency, but that advantage does not matter at this size.

Self-signed SSL

Encrypted traffic without a domain cost. Browser warnings are expected. This was a deliberate trade-off, not an oversight.

Results

[WordPress live test]

WordPress reachable over HTTPS via Elastic IP
RDS fully isolated in private subnets
Database access restricted to the web tier only
CloudWatch alarms and SNS notifications verified
Automated RDS backups tested via snapshot restore
Total cost stayed at $0 within Free Tier limits

Skills Demonstrated

Cloud and Infrastructure:

VPC and subnet design
Security group based access control
EC2 and RDS provisioning
Elastic IP management

Linux and Systems:

Ubuntu server setup
Apache and PHP configuration
MySQL client usage
Service and permission management

Security:

Network isolation
Least-privilege enforcement
TLS configuration
Application-level hardening

Operations:

CloudWatch monitoring
SNS alerting
Backup validation
Cost awareness

Final Notes

This project reflects how small but real AWS environments are built. The focus was fully on correct fundamentals including isolation, access control, monitoring, and understanding trade-offs.

Victor Ogechukwu Ojeje

Email: ojejevictor@gmail.com

LinkedIn: linkedin.com/in/victorojeje

GitHub: github.com/escanut

Deployment screenshots and configuration details available in the project repository.