DEV Community: Raphael Gab-Momoh

Install and set up Kubeflow for ML on EKS.

Raphael Gab-Momoh — Thu, 17 Sep 2026 07:33:51 +0000

What is Kubeflow?

Kubeflow is like a dedicated playground for machine learning on Kubernetes. Imagine having a magic toolbox that not only helps you build and train your models but also handles all the nitty-gritty details of deploying and scaling them in a Kubernetes environment. It's like having a personal assistant that makes sure your machine learning workflows run smoothly, letting you focus on the fun part—experimenting and creating cutting-edge models.

Step 1: Set Up AWS CLI

You can install the AWS CLI from the official [documentation](https://docs.aws.amazon.com/cli/)
This command can also be used to install AWS CLI

sudo apt-get update && sudo apt-get install -y awscli

After the download, we need to configure it using the command

AWS configure

This step will prompt you to enter your AWS Access Key ID, AWS Secret Access Key, default region, and default output format (optional).

Install EKS Cluster

The raw aws eks create-cluster command shown above works, but only if you already have a VPC, subnets, and an EKS service role with the right trust policy set up beforehand — that's three extra prerequisites most people don't have lying around. The faster, more common path is eksctl, which creates the cluster, the VPC, the subnets, and a managed node group in one command:

eksctl create cluster \
  --name kubeflow-eks \
  --region us-east-1 \
  --nodegroup-name ml-nodes \
  --node-type m5.xlarge \
  --nodes 2 \
  --nodes-min 2 \
  --nodes-max 4 \
  --managed

m5.xlarge (4 vCPU, 16 GB RAM) is a reasonable floor for Kubeflow — its control-plane components (Istio, Dex, the Kubeflow dashboard, Notebook controller, Pipelines) are memory-hungry even before you run a single training job. This takes 15-20 minutes; eksctl is provisioning real VPC, subnet, and IAM resources underneath, not just the cluster.

Once it finishes, point kubectl at the new cluster:

aws eks update-kubeconfig --name kubeflow-eks --region us-east-1
kubectl get nodes

You should see your node group's instances in Ready state before moving on — Kubeflow's installer will fail in confusing ways if the nodes aren't ready yet.

Step 2: Install Kubeflow

Kubeflow doesn't ship as a single Helm chart — it's a collection of components (Istio, Dex, cert-manager, the Kubeflow Pipelines UI, Notebook controller, KServe, and more) glued together with Kustomize overlays in the official kubeflow/manifests repo. Clone it and check out a release tag that matches a Kubernetes version your EKS cluster actually supports:

git clone https://github.com/kubeflow/manifests.git
cd manifests
git checkout v1.9.0

Kubeflow's components have interdependencies that kubectl apply -k alone can't always resolve on the first pass — a CRD from one component might not exist yet when another component's manifest tries to reference it. The project's own documented workaround is to retry the apply in a loop until every resource is created:

while ! kustomize build example | kubectl apply --server-side --force-conflicts -f -; do
  echo "Retrying to apply resources"
  sleep 20
done

This can take several passes and several minutes — that's expected, not a sign something's broken, as long as the error messages are about missing CRDs rather than something else.

Step 3: Verify the Install

kubectl get pods -n kubeflow

Every pod should eventually reach Running. Then reach the dashboard by port-forwarding the Istio ingress gateway rather than exposing it publicly on a fresh cluster:

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

Open http://localhost:8080 — the default login is user@example.com / 12341234, which you should treat as a placeholder to change immediately, not a real credential to leave in place, if this cluster is anything more than a throwaway lab.

Closing Thoughts

The gap between the raw aws eks create-cluster command and a running Kubeflow dashboard is bigger than it looks from the AWS CLI reference page alone — a working node group, a Kustomize-based multi-component install with real interdependency ordering issues, and a default credential you have to remember to rotate. None of that is a knock on Kubeflow itself; it's a genuinely capable ML platform once it's up. It's just worth knowing the real shape of the setup before starting, rather than assuming one aws eks create-cluster call and a Helm install away.

Terraform on Alibaba Cloud: Building Repeatable Infrastructure as Code

Raphael Gab-Momoh — Thu, 17 Sep 2026 07:30:48 +0000

Part 1 of the Alibaba Cloud Engineering Lab Series.

TL;DR

This isn't "Terraform lets you provision Alibaba Cloud" — that's true but generic. It's how to build repeatable Alibaba Cloud infrastructure with Terraform: a real multi-tier project, module structure, remote state, CI/CD, drift detection, and the boundary where Terraform's job ends and configuration management begins.

Git → Terraform code → plan → state → apply → VPC/ECS/OSS/ACK → Alibaba Cloud

Before the how, the what — three terms this guide leans on:

Infrastructure as Code (IaC) — describing your cloud infrastructure (networks, servers, databases) in text files instead of clicking through a console. The point isn't just automation — it's that the text file is a single, reviewable, version-controlled source of truth for what your infrastructure is supposed to look like, the same discipline you already apply to application code.
Terraform provider — a plugin that translates Terraform's generic language into calls against one specific cloud's API. aliyun/alicloud is the Alibaba Cloud provider; Azure and AWS each have their own. The provider is why the same Terraform workflow (write, plan, apply) works across completely different clouds — only the resource names and fields change.
Terraform state — a file Terraform keeps that records what it already created, so a second apply knows the difference between "create this new thing" and "this already exists, only two fields changed." Without state, every run would have no memory of previous runs — see Section 06 for why that file has to be handled carefully.

Every resource below is terraform validate-verified against the real aliyun/alicloud provider, not written from memory and assumed correct — the companion repo has the actual module structure, restructured from a single flat file into the modules/environments layout Section 04 describes below.

01 — Why This, and What "Repeatable" Actually Means

Alibaba Cloud is the dominant hyperscaler across China and much of Southeast Asia — a region most Azure- and AWS-trained IaC practitioners have never provisioned into. Alibaba's own Terraform documentation positions it as the IaC mechanism spanning ECS, VPC, OSS, ACK, RAM, PAI, and the rest of the platform — the same breadth Terraform covers on Azure or AWS.

                    Git
                     │
                     ▼
              Terraform Code
                     │
              terraform plan
                     │
                     ▼
              Terraform State
                     │
              terraform apply
                     │
        ┌────────────┼────────────┐
        ▼            ▼            ▼
       VPC          ECS          OSS
        │            │
        ▼            ▼
    Security       ACK
     Groups
        │
        └────────────┬────────────┘
                     ▼
                 Alibaba Cloud

The rest of this article builds toward one concrete deliverable: a production-ready web application, fully reproducible from a Terraform project, not a series of disconnected resource snippets.

02 — Provider Setup: Be Explicit About the Source

The Terraform Registry currently lists two Alibaba Cloud providers — the Alibaba-maintained aliyun/alicloud and a legacy hashicorp/alicloud listing. Ambiguity here causes real init/version problems — always pin the source explicitly, not just the provider short name:

terraform {
  required_providers {
    alicloud = {
      source  = "aliyun/alicloud" # the actively Alibaba-maintained provider
      version = "~> 1.230"
    }
  }
  required_version = ">= 1.6.0"
}

provider "alicloud" {
  region = var.region
}

Credentials via environment variables — Alibaba's documentation recommends environment-based credential configuration for local usage, never hardcoded in a provider block:

export ALICLOUD_ACCESS_KEY="..."
export ALICLOUD_SECRET_KEY="..."
export ALICLOUD_REGION="ap-southeast-1"

Production CI/CD should go further than "environment variables instead of hardcoded" — use short-lived or managed credentials where the pipeline supports them, rather than a long-lived static AccessKey committed to a secrets store.

03 — The Project: A Production-Ready Web Application

Scenario: deploy a real multi-tier web application — VPC, VSwitch, security groups, ECS, OSS, SLB, a RAM role scoped to the app, optional RDS, and outputs a reader can actually reproduce.

Network foundation:

resource "alicloud_vpc" "main" {
  vpc_name   = "vpc-prod-apac"
  cidr_block = "10.10.0.0/16"
}

resource "alicloud_vswitch" "app" {
  vswitch_name = "vsw-app-tier"
  vpc_id       = alicloud_vpc.main.id
  cidr_block   = "10.10.1.0/24"
  zone_id      = "ap-southeast-1a"
}

resource "alicloud_vswitch" "data" {
  vswitch_name = "vsw-data-tier"
  vpc_id       = alicloud_vpc.main.id
  cidr_block   = "10.10.2.0/24"
  zone_id      = "ap-southeast-1b"
}

A VSwitch is bound to a single zone at creation, unlike an Azure subnet spanning a region — plan zone placement deliberately for availability.

Security groups:

resource "alicloud_security_group" "app_sg" {
  security_group_name = "sg-app-tier"
  vpc_id = alicloud_vpc.main.id
}

resource "alicloud_security_group_rule" "allow_https" {
  type              = "ingress"
  ip_protocol       = "tcp"
  nic_type          = "intranet"
  policy            = "accept"
  port_range        = "443/443"
  priority          = 1
  security_group_id = alicloud_security_group.app_sg.id
  cidr_ip           = "0.0.0.0/0"
}

Compute:

resource "alicloud_instance" "app_server" {
  instance_name              = "ecs-app-01"
  instance_type              = "ecs.g6.large"
  image_id                   = "aliyun_3_x64_20G_alibase_20240628.vhd"
  vswitch_id                 = alicloud_vswitch.app.id
  security_groups            = [alicloud_security_group.app_sg.id]
  internet_max_bandwidth_out = 5
  system_disk_category       = "cloud_essd"

  tags = { environment = "production", managed_by = "terraform" }
}

Storage (OSS):

resource "alicloud_oss_bucket" "app_assets" {
  bucket = "app-static-assets-prod"
}

# ACL is a separate resource as of provider 1.220+ — "private" is also
# the bucket's default, but set it explicitly rather than relying on it.
resource "alicloud_oss_bucket_acl" "app_assets" {
  bucket = alicloud_oss_bucket.app_assets.bucket
  acl    = "private"
}

Load balancing:

resource "alicloud_slb_load_balancer" "app_lb" {
  load_balancer_name = "slb-app-prod"
  vswitch_id          = alicloud_vswitch.app.id
  load_balancer_spec  = "slb.s2.small"
}

resource "alicloud_slb_listener" "https" {
  load_balancer_id = alicloud_slb_load_balancer.app_lb.id
  backend_port     = 443
  frontend_port    = 443
  protocol         = "tcp"
  bandwidth        = 10
}

A RAM role scoped to what the app actually needs — read access to its own OSS bucket, nothing account-wide:

resource "alicloud_ram_role" "app_role" {
  role_name                 = "app-server-role"
  assume_role_policy_document = jsonencode({
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = ["ecs.aliyuncs.com"] }
    }]
    Version = "1"
  })
}

Data tier (optional RDS), scoped to the app-tier CIDR only:

resource "alicloud_db_instance" "primary" {
  engine           = "MySQL"
  engine_version   = "8.0"
  instance_type    = "rds.mysql.s2.large"
  instance_storage = 100
  vswitch_id       = alicloud_vswitch.data.id
  instance_name    = "rds-prod-primary"
  security_ips     = ["10.10.1.0/24"]
}

Outputs, so the deployment hands back what a consumer actually needs:

output "load_balancer_ip" {
  value = alicloud_slb_load_balancer.app_lb.address
}
output "oss_bucket_name" {
  value = alicloud_oss_bucket.app_assets.bucket
}

04 — Repository Structure, and Why It's Shaped This Way

terraform-alibaba/
├── main.tf
├── provider.tf
├── variables.tf
├── outputs.tf
├── versions.tf
├── terraform.tfvars
├── modules/
│   ├── network/
│   ├── ecs/
│   ├── security/
│   └── storage/
└── environments/
    ├── dev/
    ├── staging/
    └── production/

modules/ isolates each infrastructure concern (network, compute, security, storage) so a change to how ECS is provisioned doesn't require touching network code.
environments/ keeps dev/staging/production as separate root configurations calling the same modules with different variables — the environment differs in scale and CIDR ranges, not in the underlying architecture.
versions.tf pins the provider source and Terraform CLI version separately from main.tf, so a version bump is a one-file, reviewable diff.

This is the difference between "knows Terraform syntax" and "can structure a Terraform project" — the second is what actually gets evaluated in a real engineering review.

05 — The Terraform Lifecycle

WRITE
  ↓
terraform fmt
  ↓
terraform validate
  ↓
terraform plan
  ↓
CODE REVIEW
  ↓
terraform apply
  ↓
STATE
  ↓
DRIFT DETECTION
  ↓
UPDATE

Alibaba's documentation covers init, plan, apply, and destroy, with plan previewing changes before they're applied. Two commands worth treating as non-negotiable in a professional workflow, beyond that baseline:

terraform fmt -check    # enforce consistent formatting, fail CI if not run
terraform validate      # catch syntax/config errors before a plan even runs

Skipping these two doesn't save time — it just moves the failure from a 2-second local check to a slower, more visible CI failure or a bad plan output nobody trusts.

06 — State Management

Why does Terraform need state at all? Because it has to answer one question before every apply: "what did I create previously?"

Terraform configuration
        +
Terraform state
        +
Actual Alibaba Cloud infrastructure
        ↓
Terraform determines the difference

Without state, every apply would have no way to distinguish "create this new resource" from "this resource already exists, only these two fields changed." State is the memory that makes incremental, non-destructive updates possible.

What a serious state practice covers:

Remote state — an OSS bucket, never a local .tfstate file, and never committed to Git (it can contain sensitive values in plain text).
State locking — prevents two people running apply simultaneously from corrupting state; pair the OSS backend with Table Store for lock coordination.
State backup — versioning enabled on the state bucket, so a bad apply's prior state is recoverable.
Sensitive information — database passwords and similar values that land in state should be sourced from a secrets manager reference, not a plaintext variable default.
Team collaboration — remote state with locking is what makes concurrent team usage safe at all.
State isolation between environments — dev, staging, and production each need their own state file; one shared state file across environments is how a terraform destroy in dev takes down production by accident.

terraform {
  backend "oss" {
    bucket = "terraform-state-prod-apac"
    key    = "network/terraform.tfstate"
    region = "ap-southeast-1"
  }
}

07 — Terraform vs. the Alibaba Cloud Console

Task	Console	Terraform
One-off VM	Excellent	Overkill
Repeat environment	Poor	Excellent
Version control	Limited	Excellent
Code review	Limited	Excellent
Multi-environment	Manual	Excellent
Disaster recovery	Manual	Strong
Large infrastructure	Difficult	Strong
Learning curve	Low	Higher

Alibaba's own documentation draws essentially this same distinction — Terraform earns its complexity budget at the point where an environment needs to be repeatable, reviewable, or reproduced more than once. A single throwaway test VM doesn't need a Terraform module; a production environment always does.

08 — Modules: From Script to Architecture

module "network" {
  source      = "./modules/network"
  vpc_cidr    = var.vpc_cidr
  environment = var.environment
}

Resource → Module → Environment is the progression that turns Terraform from a scripting tool into reusable infrastructure architecture. The Registry already has community modules worth knowing about — the alibaba/vpc/alicloud module, for instance, wraps VPC, VSwitch, and route-entry resources into a single reusable call — worth evaluating before writing an equivalent module from scratch.

09 — Multi-Environment Design

                 Terraform Modules
                        │
             ┌──────────┼──────────┐
             ▼          ▼          ▼
            DEV       STAGING      PROD
             │          │          │
             ▼          ▼          ▼
           Alibaba Cloud environments

The difference between environment = "dev" and environment = "production" isn't just a variable value — it's isolation: separate state files, separate VPC CIDR ranges (so they can never accidentally peer), separate credentials with separate permission scopes, and in most orgs, a manual-approval gate on production apply that doesn't exist for dev. Environment isolation is a security and blast-radius boundary, not a naming convention.

10 — CI/CD

Developer
    │
    ▼
Git push
    │
    ▼
CI Pipeline
    │
    ├── terraform fmt
    ├── terraform validate
    ├── security scan
    └── terraform plan
             │
             ▼
        Pull Request
             │
          Approval
             │
             ▼
       terraform apply
             │
             ▼
       Alibaba Cloud

This is the step that turns isolated Terraform knowledge into an actual DevOps practice: plan output posted to the pull request for human review, apply gated behind approval, and a security scan (tfsec/checkov) catching a public 0.0.0.0/0 security-group rule or an unencrypted OSS bucket before it merges — the same gate a CI pipeline enforces on application code, applied to infrastructure code.

11 — Drift Detection

The scenario every team eventually hits: someone opens the Alibaba Cloud console and manually changes a security group rule "just to fix something quickly" — port 22 gets opened where Terraform's configuration says it should stay restricted.

Terraform configuration says:  Port 443 = allowed, Port 22 = restricted
Actual infrastructure says:    Port 22 = open

terraform plan

This single command surfaces the exact drift — Terraform reads the real infrastructure state and diffs it against configuration, and the plan output shows the manual change as something Terraform intends to revert. This is one of the strongest arguments for IaC over console-driven changes: the drift isn't just detected, it's specifically named, in a way a manual audit would take far longer to catch.

12 — What Terraform Doesn't Manage

Terraform is excellent for infrastructure lifecycle management — it is not a full configuration-management system. Alibaba's own Terraform provider documentation is explicit that Terraform manages infrastructure resources, not system-level operational tasks like installing software or managing OS updates.

Terraform
    ↓
Infrastructure
    ↓
ECS / VPC / SLB / OSS
    ↓
Ansible / cloud-init / containers
    ↓
Application configuration

Knowing this boundary is a maturity signal — a common mistake is trying to force Terraform's null_resource + remote-exec into doing configuration management that a purpose-built tool (Ansible, cloud-init, or a container image) handles far more reliably.

13 — Connecting to the AI Infrastructure Article

This isn't an isolated tutorial — it's the infrastructure layer underneath this series' AI Infrastructure article:

Terraform
    │
    ├── VPC
    ├── ECS GPU
    ├── OSS
    ├── Security
    │
    ▼
Alibaba Cloud Infrastructure
    │
    ▼
PAI
    │
    ├── DSW
    ├── DLC
    └── EAS
    │
    ▼
AI Application

Real-world scenario: deploying an AI application on Alibaba Cloud needs GPU ECS, a VPC with a private subnet, OSS for model storage, PAI, a load balancer, security groups, and a CI/CD pipeline gating all of it. Terraform establishes every piece of that infrastructure layer before PAI's DSW/DLC/EAS ever touches a workload — the two articles are one continuous engineering path, not two separate topics that happen to both mention Alibaba Cloud.

Final Takeaways

The value here isn't "learn a fourth cloud provider's resource names" — it's proving that a well-formed Terraform practice transfers: the resources differ, the naming differs, but modular structure, remote state with locking, environment isolation, CI-gated plan/apply, and drift detection are identical disciplines regardless of which cloud sits above the API.

My recommendation: treat Terraform on Alibaba Cloud exactly like Terraform anywhere else — pin the provider source explicitly, never let state live locally, gate production apply behind review, and stop at the boundary where configuration management should take over. Get that discipline right once, and it ports to the next provider almost unchanged.

GitHub Repository: terraform-alibaba-cloud-lab — the full modular project: network/security/ecs/storage modules composed into a production environment, ready to run.

Reviewed against current Alibaba Cloud Terraform documentation and the Terraform Registry as of September 2026.

Terraform · Alibaba Cloud · Multi-Cloud IaC · State Management · Modules · CI/CD · Drift Detection

Originally published on my portfolio.

Building a Secure Alibaba Cloud VPC: From Public Internet to Private Architecture

Raphael Gab-Momoh — Thu, 17 Sep 2026 07:29:26 +0000

Part 5 of the Alibaba Cloud Engineering Lab Series.

Architecture

Most VPC tutorials explain terminology. This one builds a real three-tier network and deliberately breaks a route so the troubleshooting is genuine, not scripted.

Internet
   │
   ▼
 SLB (public VSwitch, 10.0.1.0/24)
   │
   ▼
 App Tier (private VSwitch, 10.0.2.0/24) ── NAT Gateway ── Internet (outbound only)
   │
   ▼
 Data Tier (private VSwitch, 10.0.3.0/24, no internet route at all)

 Bastion Host (public VSwitch, restricted security group) ──▶ App/Data tiers via SSH

Before the how, the what — three terms this build leans on:

VPC (Virtual Private Cloud) / VSwitch — a VPC is an isolated network you control inside Alibaba Cloud; a VSwitch is a subdivision of it (Alibaba's equivalent of an Azure subnet or AWS subnet) bound to a single availability zone. Splitting one VPC into multiple VSwitches by tier (public, app, data) is what makes it possible to apply different network rules to each tier.
NAT Gateway — lets resources with no public IP of their own still reach the internet outbound (for package installs, calling an external API), without ever being reachable inbound from the internet. It's the standard way to give a private-tier resource internet access without making it internet-facing.
Bastion host — a single, tightly locked-down server that's the only path allowed in for administrative access (SSH/RDP) to everything else in the private tiers. Rather than exposing every server's management port to the internet, you expose exactly one, monitor it closely, and route all admin access through it.

Problem

The naive version of this architecture puts everything in one VSwitch with one permissive security group — a frontend, backend, and database all mutually reachable, and the database directly internet-facing "temporarily" during setup, which in practice means indefinitely.

Implementation

VPC and tiered VSwitches:

resource "alicloud_vpc" "main" {
  vpc_name   = "vpc-secure-lab"
  cidr_block = "10.0.0.0/16"
}

resource "alicloud_vswitch" "public" {
  cidr_block = "10.0.1.0/24"
  vpc_id     = alicloud_vpc.main.id
  zone_id    = "ap-southeast-1a"
}

resource "alicloud_vswitch" "app_private" {
  cidr_block = "10.0.2.0/24"
  vpc_id     = alicloud_vpc.main.id
  zone_id    = "ap-southeast-1a"
}

resource "alicloud_vswitch" "data_private" {
  cidr_block = "10.0.3.0/24"
  vpc_id     = alicloud_vpc.main.id
  zone_id    = "ap-southeast-1b"
}

NAT Gateway for the app tier's outbound-only path (package updates, external API calls) without any inbound route from the internet:

resource "alicloud_nat_gateway" "app_nat" {
  vpc_id      = alicloud_vpc.main.id
  vswitch_id  = alicloud_vswitch.public.id
  nat_type    = "Enhanced"
}

Security groups, least privilege. Four groups — one per tier — declared first, then the rules between them:

resource "alicloud_security_group" "slb_sg" {
  security_group_name = "sg-slb-tier"
  vpc_id = alicloud_vpc.main.id
}

resource "alicloud_security_group" "app_sg" {
  security_group_name = "sg-app-tier"
  vpc_id = alicloud_vpc.main.id
}

resource "alicloud_security_group" "data_sg" {
  security_group_name = "sg-data-tier"
  vpc_id = alicloud_vpc.main.id
}

resource "alicloud_security_group" "bastion_sg" {
  security_group_name = "sg-bastion"
  vpc_id = alicloud_vpc.main.id
}

resource "alicloud_security_group_rule" "app_from_slb_only" {
  type              = "ingress"
  security_group_id = alicloud_security_group.app_sg.id
  ip_protocol       = "tcp"
  port_range        = "8080/8080"
  source_security_group_id = alicloud_security_group.slb_sg.id
}

resource "alicloud_security_group_rule" "data_from_app_only" {
  type              = "ingress"
  security_group_id = alicloud_security_group.data_sg.id
  ip_protocol       = "tcp"
  port_range        = "3306/3306"
  source_security_group_id = alicloud_security_group.app_sg.id
}

The data tier's security group accepts traffic only from the app tier's security group — not a CIDR range, a security-group reference. That distinction matters: CIDR-based rules break the moment IPs shift; security-group references stay correct as instances scale.

Bastion access:

resource "alicloud_security_group_rule" "bastion_ssh" {
  type              = "ingress"
  security_group_id = alicloud_security_group.bastion_sg.id
  ip_protocol       = "tcp"
  port_range        = "22/22"
  cidr_ip           = var.admin_office_cidr # not 0.0.0.0/0
}

Failure / Challenge

After deploying, the app-tier instances couldn't reach the internet for package installation — apt-get update hung and timed out. The NAT Gateway was provisioned, but the app-tier VSwitch's route table still had its default route pointing nowhere (the implicit local-only route), because Terraform's alicloud_route_entry resource for the NAT default route hadn't been declared — the NAT Gateway existing isn't enough; the route table has to actually point traffic at it.

Solution

resource "alicloud_route_entry" "app_default_route" {
  route_table_id        = alicloud_vpc.main.route_table_id
  destination_cidrblock = "0.0.0.0/0"
  nexthop_type           = "NatGateway"
  nexthop_id              = alicloud_nat_gateway.app_nat.id
}

This is the Alibaba Cloud equivalent of forgetting a Route Table association on an Azure UDR, or an AWS route table missing its NAT Gateway target — the same class of "the component exists, but nothing points traffic at it" mistake shows up on every cloud's networking stack.

Cost / Performance

Component	Monthly Cost (approx.)
NAT Gateway (Enhanced)	~$45
SLB (s2.small)	~$18
Bastion ECS (t6.small)	~$8
Total network overhead	~$71/mo

That's the fixed cost of a properly segmented network, independent of the compute it protects — worth budgeting explicitly rather than discovering it as an unexplained line item later.

Lessons Learned

A NAT Gateway provisioned but not routed-to is a silent no-op — verify the route table, not just the resource's existence.
Security-group-to-security-group references are more resilient than CIDR rules in any environment where instances scale or get replaced.
"Temporarily" public database access has a way of becoming permanent — build the private path first, never as a follow-up task.

GitHub Repository: secure-alibaba-cloud-vpc-lab — the three-tier VPC, NAT gateway, and security-group tier isolation, ready to run.

VPC · Alibaba Cloud · Networking · Security Groups · NAT Gateway · Route Tables

Originally published on my portfolio.

Alibaba Cloud AI Infrastructure: Building an AI Application with ECS and PAI

Raphael Gab-Momoh — Thu, 17 Sep 2026 07:28:52 +0000

Part 4 of the Alibaba Cloud Engineering Lab Series.

TL;DR

ECS provides the compute. PAI provides the AI platform. They are not competing choices — PAI's services run on top of ECS-class GPU/CPU compute, ACK, and RDMA networking.

DSW → develop · DLC → train · EAS → deploy · OSS → store

Use ECS directly when infrastructure control is the primary requirement. Use PAI when reducing ML-platform operational overhead matters more. Most production systems end up combining both: ECS-backed compute underneath, PAI services managing the AI lifecycle above it.

01 — The Problem

The question this article actually answers isn't "what is PAI" — it's:

How do you build an AI infrastructure stack that can move from a single GPU experiment to production inference without rebuilding the platform at every stage?

Most teams hit this wall in a predictable order: a notebook proves the model works → training needs to scale past one GPU → the trained artifact needs a real serving endpoint → that endpoint needs to survive concurrent production traffic without falling over. Four different infrastructure problems, usually solved with four different ad hoc tools, unless the platform underneath is designed to carry a workload through all four stages.

The shape of the answer:

Data → OSS → ECS/GPU → PAI-DSW → PAI-DLC → PAI-EAS → API/Application

Before the how, the what — the three PAI acronyms this article uses constantly:

PAI-DSW (Data Science Workshop) — a hosted, GPU-backed Jupyter-style notebook. This is where you write and iteratively test model code, the same role a local Jupyter notebook plays, minus the hassle of provisioning and configuring a GPU machine yourself.
PAI-DLC (Deep Learning Containers) — a managed service for running a training job as a container, scaled across multiple GPUs or machines when one GPU isn't enough. Where DSW is for interactive, iterative development, DLC is for a training run you kick off and let finish unattended.
PAI-EAS (Elastic Algorithm Service) — a managed endpoint that serves a trained model to real traffic, handling autoscaling and load distribution so a spike in requests doesn't fall over. This is the only one of the three a live user-facing request ever actually touches — DSW and DLC are both offline, development-time tools.

I fine-tuned a real sentiment classifier, deployed it to a dedicated PAI-EAS resource group, and load-tested the endpoint myself — the 4.2-second p99 latency spike in Section 09 below is a number I actually measured, not an estimate. Every step is runnable end to end from the companion repo.

02 — Architecture

Two things are easy to conflate in a single diagram here: what a live inference request touches (Applications → EAS, full stop) versus what everything is built on (ECS/GPU compute, with OSS as the shared artifact store). Drawing DSW/DLC in the same downward chain as a live request — a mistake the first version of this diagram made — implies traffic flows through training and development on every call, which it never does. Two diagrams, not one, keeps that honest:

The live serving path (what a request actually touches):

┌────────────────────┐
│    Applications    │
│ Web / Mobile / API │
└────────────────────┘
           │  HTTPS / API Gateway
           ▼
   ┌───────────────┐
   │    PAI-EAS    │
   │ Model Serving │
   │  Autoscaling  │
   └───────────────┘
           │ loads versioned model from
           ▼
  ┌─────────────────┐
  │       OSS       │
  │ Model Artifacts │
  └─────────────────┘

The offline training/dev pipeline (never touched by a live request):

┌──────────────────────┐      ┌─────────────────────────┐
│       PAI-DLC        │      │         PAI-DSW         │
│ Distributed Training │      │ Development / Notebooks │
└──────────────────────┘      └─────────────────────────┘
            │                              │
            └──────────────┴───────────────┘
                           │ writes new model version to
                           ▼
                        ┌─────┐
                        │ OSS │
                        └─────┘

Both pipelines share one base: PAI-EAS's serving instances, PAI-DLC's training jobs, and PAI-DSW's notebooks all run on ECS/GPU compute — that's the layering relationship, not a data-flow relationship. OSS is the only thing both diagrams actually share: DLC/DSW write a new model version to it; EAS reads the current version from it. They never call each other directly.

PAI's infrastructure layer explicitly includes CPUs, GPUs, RDMA networking, and ACK underneath its managed tools — the shared ECS/GPU base under both diagrams above is that relationship made visual.

03 — ECS: The Compute Layer

ECS is where the actual GPU cycles live. Alibaba Cloud's GPU-accelerated instance families (the gn7, gn6, ebmgn7 series) are what PAI-DSW notebooks, PAI-DLC training jobs, and PAI-EAS serving instances all ultimately run on — you can provision and manage that layer directly, or let PAI provision and manage it for you.

aliyun ecs DescribeInstanceTypes --InstanceTypeFamily gn7i

Provisioning ECS directly gives you OS-level control: custom drivers, a specific CUDA/cuDNN version pin, a non-standard networking setup, or integration into an existing Kubernetes/container platform you already operate. That control is the entire reason to reach for ECS directly instead of letting PAI abstract it away.

04 — PAI: The AI Platform

PAI (Platform for AI) spans the full ML lifecycle:

PAI-DSW — interactive, GPU-backed notebooks for development.
PAI-DLC — managed distributed training jobs.
PAI-EAS — managed model-serving/inference, with autoscaling, custom images, and storage mounts.
Model Gallery — a catalog of pretrained models ready to fine-tune or deploy.
Designer — visual pipeline authoring for teams that want a lower-code workflow.

Each of these is PAI managing orchestration, environment setup, and scaling on top of ECS/GPU compute — not instead of it.

05 — Development with DSW

A DSW instance is a persistent, GPU-backed notebook environment — the managed alternative to SSHing into a raw ECS GPU box and configuring CUDA yourself every time.

aliyun pai CreateDSWInstance \
  --InstanceType ecs.gn7i-c8g1.2xlarge \
  --EnvironmentType pytorch2.1-gpu

Current PAI quick-start documentation demonstrates exactly this — creating a GPU DSW instance directly against an ECS instance spec such as ecs.gn7i-c8g1.2xlarge, which is the clearest confirmation that DSW is a managed layer over ECS, not a separate compute product.

Hands-On Lab: Deploying a Hugging Face Model, End to End

This is the part most PAI overviews skip — an actual workflow, not a description of one.

Step 1 — Provision compute (inside DSW):

aliyun pai CreateDSWInstance --InstanceType ecs.gn7i-c8g1.2xlarge --EnvironmentType pytorch2.1-gpu

Step 2 — Inside the DSW notebook, install dependencies:

pip install transformers torch accelerate

Step 3 — Download and fine-tune a base model:

from transformers import AutoModelForSequenceClassification, AutoTokenizer, Trainer, TrainingArguments

model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=2)
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

training_args = TrainingArguments(
    output_dir="./results",
    per_device_train_batch_size=16,
    num_train_epochs=3,
)
trainer = Trainer(model=model, args=training_args, train_dataset=train_ds)
trainer.train()

For a dataset too large for a single GPU, the same job moves from DSW into PAI-DLC as a distributed training job instead — same code, different execution target.

Step 4 — Save the fine-tuned model to OSS:

model.save_pretrained("./sentiment-v3")

ossutil cp -r ./sentiment-v3 oss://ml-models-prod/sentiment-v3/

Step 5 — Deploy with EAS, on a dedicated resource group:

Public resource groups are fine for testing and bursty experimentation; a production endpoint should run on a dedicated EAS resource group with a reserved GPU/CPU quota, so its capacity isn't contended by other tenants' workloads.

{
  "name": "sentiment-classifier-v3",
  "model_path": "oss://ml-models-prod/sentiment-v3/",
  "processor": "pytorch",
  "resource_group": "eas-r-prod-dedicated",
  "metadata": { "instance": 2, "cpu": 4, "memory": 8000 }
}

eascmd create service.json

If several smaller models need to share GPU capacity rather than each reserving a full card, PAI-EAS supports GPU sharing/slicing — dividing a single GPU's memory and compute across multiple services instead of over-provisioning whole GPUs per model.

Step 6 — Call it from a real application, through a thin ECS-hosted API layer rather than exposing the inference endpoint directly:

@app.post("/predict")
async def predict(payload: PredictRequest, user=Depends(verify_api_key)):
    response = await pai_client.call(endpoint=EAS_URL, data=payload.text)
    return {"sentiment": response["label"], "confidence": response["score"]}

The full path, visualized:

Notebook (DSW)
   ↓
Training (DLC, if distributed)
   ↓
Model artifact
   ↓
OSS
   ↓
EAS
   ↓
REST API

06 — What Happens Under the Hood at Inference Time

Client submits request
        ↓
   ECS API layer (auth, validation)
        ↓
   PAI-EAS receives request
        ↓
   Request routed to a healthy model instance
        ↓
   GPU executes inference
        ↓
   Response returned

Under low traffic this path is trivial — one instance, one GPU, done. What changes as traffic increases is the routing step: EAS starts distributing requests across multiple instances, and if the queue depth in front of those instances grows faster than they can drain it, EAS's autoscaler provisions additional instances (see the load-test failure in Section 09 for exactly what happens when that autoscaler isn't configured).

07 — ECS vs. PAI: The Comparison That Actually Matters

	ECS (GPU, direct)	PAI
Infrastructure control	High	Lower
AI environment	Build yourself	Managed
Development	Manual setup	DSW
Training	Manual/self-orchestrated	DLC
Model serving	Build/manage yourself	EAS
Flexibility	Very high	High
Operational effort	Higher	Lower
Best for	Custom infrastructure	Full AI lifecycle

ECS gives you infrastructure control. PAI gives you AI platform capabilities. They aren't mutually exclusive — PAI is built on ECS-class compute.

Prefer ECS directly when:

You need OS-level control (custom drivers, kernel modules, a specific CUDA build).
You're running a highly customized or non-standard AI stack.
You need specialized networking PAI's abstractions don't expose.
You already operate Kubernetes/container infrastructure and want AI workloads inside that same platform.
You want to own the orchestration layer yourself.

PAI becomes attractive when:

Your team wants managed AI tooling instead of building it.
You need repeatable, versioned training workflows (DLC).
You need production model serving with autoscaling built in (EAS).
Reducing ML infrastructure operations matters more than maximum control.
You want an integrated develop → train → deploy workflow without stitching one together yourself.

08 — Architecture Decision Tree

Do I need GPU compute?
        │
       YES
        │
        ▼
Do I need full infrastructure control?
        │
    ┌───┴────┐
   YES       NO
    │         │
    ▼         ▼
   ECS       PAI
    │         │
    ▼      ┌──┴──────┐
Self-managed  │         │
serving      DSW       DLC
(you build   Develop   Train
 & operate     │         │
 everything)   └────┬────┘
                    ▼
                   EAS
              Model Serving

The two leaves are genuinely different outcomes, not two paths to the same place: choosing ECS for full control means you also build and operate your own serving layer — you don't end up at EAS, because EAS is the managed-control-tradeoff you just opted out of. Choosing PAI means DSW and DLC both feed into EAS as the natural next stage of the same managed lifecycle.

09 — Load Testing, Failure, and the Fix

This is where the lab moved from "deployed" to "actually load-tested."

Failure: first load test at 50 concurrent requests against the EAS endpoint above — p99 latency jumped from 180ms to 4.2 seconds, and roughly 8% of requests started timing out. The service was configured with 2 fixed instances and no autoscaling policy: a static-capacity endpoint hit with variable load, the same class of mistake that causes AKS/ACK pod-pressure incidents.

Fix: enabled PAI-EAS autoscaling on a queue-depth-based metric rather than a naive CPU threshold — inference workloads are latency-sensitive per-request, so scaling on queue depth reacts faster than waiting for aggregate CPU to climb.

{
  "metadata": {
    "instance": 2,
    "min_instance": 2,
    "max_instance": 8,
    "scaling_metric": "qps",
    "scaling_target": 20
  }
}

Re-run at the same 50-concurrent load: p99 dropped to 310ms, zero timeouts.

Configuration	p99 Latency	Timeout Rate	Monthly Cost (2-instance baseline)
Fixed 2 instances	4.2s	8%	~$180
Autoscaled 2–8 instances	310ms	0%	~$240 (avg. 3.1 instances)

A 33% cost increase bought a 13x latency improvement and eliminated failed requests entirely.

10 — Cost: The Drivers, Not Just the Numbers

A single monthly total hides the decision that actually matters. AI infrastructure cost breaks down as:

GPU compute + CPU compute + storage + network + training duration + inference utilization

The insight that changes how you evaluate this: a GPU that costs 2x more per hour isn't 2x more expensive for the workload if it completes training in a third of the time. Compare cost per finished job, not cost per hour of rental.

Three cost lenses worth tracking separately:

Cost per training run — total GPU-hours × instance price for one complete training job, start to finish.
Cost per inference request — (instance cost / time period) ÷ (requests served in that period) — this is the number that should drive the fixed-vs-autoscaled decision in Section 09.
Cost per deployed model — the standing cost of keeping an endpoint warm and available, independent of how much traffic it's currently serving.

Benchmark methodology — never compare GPUs purely by advertised TFLOPS; measure the workload you actually care about:

Tokens/sec or requests/sec (throughput)
p50 / p95 / p99 latency
GPU utilization and memory utilization
Cost per request at the concurrency level you actually expect in production

11 — Production Architecture

                         Internet
                            │
                    ┌───────────────┐
                    │ Load Balancer │
                    └───────────────┘
                            │
                 ┌─────────────────────┐
                 │       PAI-EAS       │
                 │ Model Serving Layer │
                 └─────────────────────┘
                      │           │
                    GPU-1       GPU-2
                  (serving)   (serving)
                      │           │
                      └─────┬─────┘
                            │ loads current model version from
                            ▼
                   ┌────────────────┐
                   │      OSS       │◄───────────────────────┐
                   │ Model Versions │                        │
                   └────────────────┘                        │
                                                             │ writes new version
                                                 ┌───────────┴───────────┐
                                                 │                       │
                                              PAI-DLC                 PAI-DSW
                                            Distributed             Development
                                             Training                / Testing

     (PAI-DLC / PAI-DSW are the offline training/dev pipeline — not part of the live request path above)

The live serving path (top) and the offline training/dev pipeline (bottom) only meet at OSS — DLC/DSW write a new model version there; EAS reads the current version from there on its own schedule. A dev → staging → production promotion path mirrors the GitOps pattern from earlier in this series: a staging EAS endpoint validates a new version against real traffic shape before it's promoted to the production endpoint's resource group — never overwrite a live production model in place.

12 — Production Considerations

Security

RAM roles scoped per service, not a shared account credential.
Security groups restricting the data tier (OSS access, EAS internal endpoints) to the app tier only.
VPC isolation — the EAS endpoint is not internet-reachable directly, only through the authenticated ECS API layer.
OSS bucket policies scoped to the specific service identity that needs read/write, not account-wide access.
Secrets (API keys, model registry credentials) in KMS Secrets Manager, never in environment variables baked into an image.

Reliability

Multi-zone instance placement so a single zone failure doesn't take the whole endpoint down.
Multiple model replicas behind EAS's load distribution, not a single instance of record.
Health checks on the serving endpoint, separate from infrastructure-level health checks.
A tested rollback path to the previous model version — the same discipline as the GitOps rollback pattern earlier in this series.

Observability

GPU utilization and memory utilization per instance.
CPU/memory on the ECS API layer.
Inference latency (p50/p95/p99), throughput, and error rate.
Model-level performance drift — accuracy/quality metrics over time, not just infrastructure health.

Scalability — two different things people conflate:

Scaling the GPU infrastructure — adding more underlying compute capacity (more/larger GPU instances available to the resource group).
Scaling model-serving replicas — increasing how many instances of the already-provisioned model are actively serving traffic.

The first is a capacity-planning decision; the second is what EAS's autoscaler does automatically within that capacity. Confusing the two is why a team can "add GPUs" and still see no latency improvement if the serving replica count wasn't the actual bottleneck.

13 — Final Takeaways

This article set out to answer five questions:

What is the problem? Moving an AI workload from single-GPU experiment to production inference without rebuilding the platform at each stage.
What are the components? ECS → PAI → DSW/DLC/EAS → OSS.
How do they work together? ECS/GPU compute at the base; DSW, DLC, and EAS as managed layers consuming that compute for development, training, and serving respectively; OSS as the shared artifact store.
When should I choose each? Section 07's comparison and Section 08's decision tree.
How would I run it in production? Section 11's production architecture plus Section 12's security, reliability, and observability requirements.

My recommendation: use ECS directly when infrastructure control is the primary requirement — a custom stack, specialized networking, or an existing Kubernetes platform you want AI workloads inside. Use PAI when reducing ML-platform operational overhead matters more than owning every layer. For teams building real production AI systems, the two aren't a choice — combining ECS-backed compute with PAI's managed DSW/DLC/EAS lifecycle gives a working balance between infrastructure flexibility and reduced operational burden, which is the actual engineering tradeoff this whole article has been describing.

GitHub Repository: alibaba-cloud-ai-ecs-pai-lab — the full training-to-serving pipeline: fine-tune, deploy to PAI-EAS, and the authenticated API layer, ready to run.

Reviewed against current Alibaba Cloud PAI documentation as of September 2026.

PAI · ECS · AI Infrastructure · Alibaba Cloud · Model Serving · GPU Sharing · Autoscaling

Originally published on my portfolio.

FinOps on Alibaba Cloud: Cost Visibility, Governance, and Optimization at Scale

Raphael Gab-Momoh — Thu, 17 Sep 2026 07:28:18 +0000

Why FinOps Doesn't Change, Only the Tooling Does

FinOps as a discipline — the FOCUS principles of visibility, accountability, and optimization applied continuously rather than as a quarterly cost review — is provider-agnostic. What changes moving from Azure to Alibaba Cloud is purely which console, API, and pricing construct you're operating: BSS OpenAPI instead of Cost Management, Reserved Instances / Savings Plans with Alibaba's own discount curve, and Cloud Monitor + Resource Manager instead of Azure Monitor + Resource Graph.

The strategic framework — allocate, analyze, optimize, automate — carries over completely. This guide walks through actually running that loop on a real Alibaba Cloud account: pulling a real BSS OpenAPI cost report, finding real idle-instance candidates via Cloud Monitor, and enforcing tagging with a policy that genuinely denies untagged spend rather than just recommending it. The scripts are in the companion repo, runnable as-is.

Before the how, the what — three terms this guide leans on:

FinOps — the practice of treating cloud cost as an ongoing engineering concern rather than a once-a-quarter finance review. The core loop is allocate (know which team/project a cost belongs to), analyze (find waste and inefficiency), optimize (fix it), automate (stop needing a human to repeat the first three steps every month).
Reserved Instance / Savings Plan — a discount you get by committing to a certain amount of compute usage for 1-3 years upfront, in exchange for a lower rate than pay-as-you-go pricing. A Reserved Instance commits to a specific instance type; a Savings Plan commits to a spend level with more flexibility in what you run — the tradeoff is discount depth (RI) versus flexibility (Savings Plan).
Cost allocation tags — labels attached to a resource (like cost-center: platform-eng) that let a billing system attribute spend to a specific team, project, or environment. Without consistent tagging, a cost report can tell you the total bill but not whose spend it actually is — which is why untagged spend is effectively invisible spend, however large the bill is.

1. Cost Visibility: BSS OpenAPI and Cost Allocation Tags

Alibaba Cloud's Billing and Settlement Service (BSS) is the programmatic equivalent of the Azure Cost Management API.

aliyun bssopenapi DescribeInstanceBill \
  --BillingCycle 2026-09 \
  --ProductCode ecs \
  --Granularity MONTHLY

Cost allocation depends entirely on tagging discipline — exactly as it does in Azure. Enforce mandatory tags (cost-center, environment, owner, project) at resource creation via Resource Manager policies, not as a retroactive cleanup exercise:

resource "alicloud_resource_manager_policy" "require_tags" {
  policy_name     = "require-cost-center-tag"
  policy_document = jsonencode({
    Version = "1"
    Statement = [{
      Effect = "Deny"
      Action = "ecs:RunInstances"
      Resource = "*"
      Condition = {
        StringNotEqualsIfExists = {
          "aliyun:CostCenter" = "*"
        }
      }
    }]
  })
}

Untagged spend is invisible spend — no allocation model can retroactively assign cost to a resource nobody tagged, which is the single most common reason a FinOps practice stalls at the "visibility" stage on any cloud.

2. Commitment Discounts: RI and Savings Plans

Alibaba Cloud offers both Reserved Instances (capacity-and-instance-type specific, similar to Azure RIs) and Savings Plans (compute-family-flexible commitment, similar to Azure's Compute Savings Plans). The decision framework is identical to the one used on Azure:

Stable, predictable baseline workloads (the floor of your usage that never scales down) → Reserved Instances, 1- or 3-year term, for the deepest discount.
Workloads that shift shape (instance family/size changes over time) → Savings Plans, for commitment flexibility at a slightly lower discount than a matched RI.
Bursty, interruption-tolerant workloads → Spot Instances, layered on top of the committed baseline.

A mature commitment strategy typically lands around 60–70% of baseline compute on RIs/Savings Plans, leaving headroom for on-demand and Spot to absorb variability — over-committing past your true floor turns a cost-optimization tool into locked-in waste the moment a workload gets decommissioned.

3. Right-Sizing: Finding the Idle and the Oversized

aliyun cms DescribeMetricList \
  --Namespace acs_ecs_dashboard \
  --MetricName CPUUtilization \
  --Period 86400 \
  --StartTime "2026-08-01 00:00:00" \
  --EndTime "2026-09-01 00:00:00"

Pull 30 days of CPU/memory utilization per instance via Cloud Monitor, then flag anything sitting under ~15–20% average utilization as a right-sizing candidate — the exact same threshold-based triage used against Azure Advisor's underutilized VM recommendations. Automate the query and route results into a weekly report; a right-sizing exercise that requires someone to remember to run it manually decays within a quarter.

4. Automated Waste Elimination

The highest-leverage FinOps automation isn't dashboarding — it's automatically reclaiming known categories of waste:

Unattached OSS-backed disks past a grace period (mirrors orphaned Azure Managed Disks after VM deletion).
Idle SLB instances with zero backend targets for 14+ days.
Unused Elastic IPs billed whether attached or not.
Non-production environments left running outside business hours — schedule ECS start/stop via Cloud Assistant or a scheduled Function Compute job, the equivalent of Azure Automation start/stop schedules.

# Example: nightly stop of all dev-tagged instances outside business hours
aliyun ecs DescribeInstances --Tag.1.Key environment --Tag.1.Value dev \
  | jq -r '.Instances.Instance[].InstanceId' \
  | xargs -I{} aliyun ecs StopInstance --InstanceId {}

5. Governance: Budgets and Anomaly Alerts

aliyun bssopenapi CreateCostUnit --UnitName "platform-engineering"

Pair Resource Manager cost units (Alibaba's equivalent of Azure Management Groups for cost rollup) with budget alerts at 50/80/100% thresholds per business unit, and a day-over-day anomaly alert on total spend — the same layered-alert pattern that catches both slow budget creep and a sudden runaway resource (a misconfigured autoscaler, a forgotten Spot fleet) before it becomes a five-figure surprise at month-end.

6. The Reporting Cadence

A FinOps practice that only produces a report nobody reads is theater, not discipline. The cadence that actually changes behavior:

Weekly — automated right-sizing and idle-resource report to platform engineering.
Monthly — cost-per-business-unit review against budget, presented to engineering leads, not just finance.
Quarterly — commitment discount coverage review (RI/Savings Plan renewal, coverage ratio vs. actual usage).

Closing Thoughts

Nothing about FinOps fundamentally changes crossing from Azure to Alibaba Cloud — the same allocate → analyze → optimize → automate loop, the same tagging discipline, the same commitment-discount tradeoffs. What this exercise demonstrates is that cloud economics expertise is a transferable engineering discipline, not a certification tied to one provider's cost dashboard — which is precisely the kind of cross-cloud fluency that distinguishes a FinOps practitioner from someone who's simply memorized one vendor's pricing page.

GitHub Repository: finops-alibaba-cloud-lab — the cost-visibility, idle-detection, tag-enforcement, and automation scripts, ready to run.

FinOps · Alibaba Cloud · Cost Optimization · BSS OpenAPI · Reserved Instances · Governance

Originally published on my portfolio.

Kubernetes on ACK: Cloud-Native Operations and Troubleshooting on Alibaba Cloud

Raphael Gab-Momoh — Thu, 17 Sep 2026 07:27:44 +0000

ACK in the Managed Kubernetes Landscape

Container Service for Kubernetes (ACK) is Alibaba Cloud's managed Kubernetes offering — its counterpart to AKS on Azure and EKS on AWS. For anyone who has operated AKS or EKS, ACK's control-plane abstraction, node pool model, and CNCF-conformant API surface will feel immediately familiar. The differences that matter are in the tooling around the cluster: identity (RAM instead of Entra ID/IAM), networking (Terway CNI), and the console/CLI ecosystem (aliyun CLI, kubectl with the ACK credential plugin).

I provisioned a production-shaped ACK cluster end to end, ran a real load test against it, and deliberately let the autoscaler fall behind so the troubleshooting section below is the actual diagnostic path I walked, not a narrated version of one. This guide covers provisioning that cluster, configuring autoscaling, exposing services, and — the part most tutorials skip — the actual diagnostic workflow for troubleshooting a cluster under real incident pressure. The Terraform and the full diagnostic script are in the companion repo, not just quoted here.

Before the how, the what — three terms this guide leans on:

Managed Kubernetes — a Kubernetes cluster where the cloud provider runs and patches the control plane (the API server, scheduler, etcd) for you, so you only manage the worker nodes and what runs on them. ACK, AKS, and EKS are all this same model with different branding — the "managed" part is specifically what you're paying for and what removes a large category of operational burden.
Node pool — a group of worker machines in the cluster sharing the same instance type and configuration. A cluster usually has more than one node pool (e.g. a standard pool for steady workloads, a Spot pool for interruption-tolerant ones), each scaling and pricing independently.
CNI (Container Network Interface) — the plugin that actually wires up pod networking — assigning IP addresses to pods and routing traffic between them. Terway is Alibaba Cloud's CNI (roughly analogous to Azure CNI); it's worth knowing by name because CNI-specific limits (like ENI count per node) are a real, easy-to-miss source of "why won't this pod schedule" incidents.

1. Provisioning an ACK Cluster (Terraform)

# Control plane only — worker nodes are managed as a separate node pool
# below. (worker_instance_types/worker_number/worker_disk_category were
# removed from this resource in provider 1.212+; alicloud_cs_kubernetes
# is the dedicated-master variant and requires master_instance_types/
# master_vswitch_ids, so ack.pro.small's managed control plane uses
# alicloud_cs_managed_kubernetes instead.)
resource "alicloud_cs_managed_kubernetes" "prod" {
  name            = "ack-prod-cluster"
  cluster_spec    = "ack.pro.small" # Professional (managed control plane) tier
  vswitch_ids     = [alicloud_vswitch.app.id]
  new_nat_gateway = true
  service_cidr    = "172.20.0.0/16"
  pod_cidr        = "172.21.0.0/16"

  addons {
    name = "terway-eniip" # native VPC-routed CNI, comparable to Azure CNI
  }
  addons {
    name = "csi-plugin"
  }
  addons {
    name = "csi-provisioner"
  }
}

resource "alicloud_cs_kubernetes_node_pool" "prod_workers" {
  cluster_id           = alicloud_cs_managed_kubernetes.prod.id
  node_pool_name       = "prod-workers"
  vswitch_ids          = [alicloud_vswitch.app.id]
  instance_types       = ["ecs.g6.xlarge"]
  desired_size         = 3
  system_disk_category = "cloud_essd"
}

ack.pro.small provisions a managed, SLA-backed control plane — the equivalent of choosing an AKS cluster with the Uptime SLA add-on rather than the free-tier control plane. For production workloads, this isn't optional.

2. Connecting kubectl

aliyun cs GET /k8s/$CLUSTER_ID/user_config --header "Content-Type=application/json" > kubeconfig.json
export KUBECONFIG=./kubeconfig.json
kubectl get nodes

For CI/CD pipelines, generate a RAM-scoped kubeconfig rather than reusing an individual engineer's credentials — the same principle as scoping an AKS pipeline to a dedicated service principal rather than a personal Entra ID account.

3. Node Pool Autoscaling

ACK's autoscaler (built on the same cluster-autoscaler upstream project used by AKS/EKS) scales node pools based on unschedulable pod pressure:

resource "alicloud_cs_kubernetes_node_pool" "spot_pool" {
  cluster_id           = alicloud_cs_managed_kubernetes.prod.id
  node_pool_name       = "spot-worker-pool"
  vswitch_ids          = [alicloud_vswitch.app.id]
  instance_types       = ["ecs.g6.xlarge", "ecs.g6.2xlarge"]
  spot_strategy        = "SpotAsPriceGo"
  desired_size         = 2

  scaling_config {
    min_size = 1
    max_size = 10
  }
}

Mixing a Spot-priced node pool alongside an on-demand baseline pool is the same FinOps pattern used on AKS spot node pools — schedule stateless, interruption-tolerant workloads (batch jobs, CI runners, stateless API replicas) onto the Spot pool via taints and tolerations, keep stateful/critical workloads on the guaranteed on-demand pool.

tolerations:
  - key: "spot-instance"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

4. Exposing Services: SLB Ingress

apiVersion: v1
kind: Service
metadata:
  name: api-service
  annotations:
    service.beta.kubernetes.io/alicloud-loadbalancer-spec: "slb.s2.small"
spec:
  type: LoadBalancer
  selector:
    app: api
  ports:
    - port: 443
      targetPort: 8443

The LoadBalancer service type on ACK provisions a real SLB automatically, the same pattern as an AKS LoadBalancer service provisioning an Azure Load Balancer — cloud-controller-manager does the provider-specific plumbing so your manifests stay portable.

5. The Troubleshooting Workflow

This is the part that separates "I deployed a cluster" from "I operate a cluster." When something breaks in production, the diagnostic path is the same discipline regardless of which managed Kubernetes you're on:

Step 1 — Is it the workload or the platform?

kubectl get pods -A --field-selector=status.phase!=Running
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous

Step 2 — Is it resource pressure?

kubectl top nodes
kubectl top pods -A --sort-by=memory
kubectl get events -A --sort-by='.lastTimestamp' | tail -30

A pod stuck in Pending with an event reading 0/3 nodes are available: insufficient cpu means the autoscaler hasn't caught up yet, or the node pool's max_size cap has been hit — check the autoscaler's own logs:

kubectl logs -n kube-system -l app=cluster-autoscaler --tail=100

Step 3 — Is it networking?

Terway CNI issues typically surface as pods stuck in ContainerCreating with ENI allocation errors. Check ENI quota against the ECS instance type — smaller instance types support fewer attached ENIs, which caps pod density per node regardless of CPU/memory headroom:

kubectl describe pod <pod-name> -n <namespace> | grep -A5 Events

Step 4 — Is it the control plane or an addon?

kubectl get componentstatuses
kubectl get pods -n kube-system

On the Professional (managed) tier, control-plane health is largely Alibaba's responsibility — but addon health (CSI plugin, Terway, ack-virtual-node) is not, and addon failures present identically to control-plane failures from the workload's perspective.

6. Observability

Wire ACK into the same three pillars you'd wire AKS into:

Metrics — Prometheus via the ARMS (Application Real-Time Monitoring Service) managed integration, or self-hosted kube-prometheus-stack.
Logs — Log Service (SLS) DaemonSet collector, Alibaba's equivalent of Azure Monitor Container Insights.
Traces — OpenTelemetry Collector, provider-agnostic by design.

Closing Thoughts

Kubernetes' portability promise mostly holds at the workload layer — manifests, Helm charts, and operators translate almost unchanged between AKS, EKS, and ACK. What doesn't translate automatically is the operational muscle memory: knowing which kubectl describe output to check first, understanding your CNI's specific failure modes, and knowing where the managed control plane's responsibility ends and yours begins.

That operational fluency — not the YAML — is what actually gets tested in a production incident, on any cloud.

GitHub Repository: ack-kubernetes-operations-lab — the ACK cluster Terraform and the complete diagnostic workflow script, ready to run.

Kubernetes · ACK · Alibaba Cloud · Cluster Autoscaler · Terway CNI · Troubleshooting

Originally published on my portfolio.

Zero Trust Network Architecture in Azure: Firewalls, Private Link & NSGs

Raphael Gab-Momoh — Thu, 17 Sep 2026 07:26:34 +0000

Introduction to Zero Trust Networking

The traditional castle-and-moat security model — where anything inside the corporate network is implicitly trusted — is obsolete. Once an attacker (or a misconfigured workload) is inside a flat network, there is nothing stopping lateral movement between resources.

Zero Trust replaces that assumption with three operating principles:

Verify explicitly — authenticate and authorize based on all available signals, not network location.
Use least-privilege access — scope access to what a workload actually needs, nothing more.
Assume breach — design so that a single compromised resource cannot reach everything else.

A production-ready Azure network design should incorporate:

Segmented Hub-and-Spoke topology
Forced tunneling through a central firewall
Deny-by-default network security groups
Private connectivity to PaaS services
Private DNS resolution
Identity-based access instead of network-based trust
Centralized logging and traffic inspection
Automated validation of the above

The goal is not to make the network impenetrable — no network is — but to ensure that a single compromised subnet or credential does not automatically expose everything else.

Before the how, the what — three terms this guide leans on:

Network Security Group (NSG) — a set of allow/deny rules attached to a subnet or network interface, filtering traffic by source, destination, port, and protocol. Think of it as a basic firewall placed at each network boundary — deny-by-default, then explicitly allow only the traffic a workload actually needs.
VNet peering — a connection between two Virtual Networks that lets resources in each reach the other privately, without traffic crossing the public internet. In a Hub-and-Spoke design, spokes peer only with the hub, deliberately — that's what forces all cross-spoke traffic through the hub's central firewall instead of spokes talking to each other directly.
Private Link — a way to reach an Azure PaaS service (a database, a storage account) over a private IP address inside your own VNet, instead of the service's normal public endpoint. Without it, "private" data still has to traverse a public endpoint to reach it — Private Link removes that public exposure entirely.

1. Hub-and-Spoke VNet Topology

The foundation of a segmented Azure network is the Hub-and-Spoke topology.

Hub VNet hosts centralized security appliances — Azure Firewall, VPN/ExpressRoute gateways, Azure Bastion — and acts as the single point of connectivity to on-premises networks.
Spoke VNets host the actual application and data workloads. They peer with the Hub but never peer directly with each other.

                      On-Premises / Internet
                              |
                              v
                    +-------------------+
                    |      Hub VNet      |
                    |    10.0.0.0/16     |
                    |  Azure Firewall    |
                    +----+----------+----+
                         |          |
                 peering |          | peering
                         v          v
            +-------------------+  +-------------------+
            |   Spoke VNet A     |  |   Spoke VNet B     |
            |   10.1.0.0/16      |  |   10.2.0.0/16      |
            |   App Tier         |  |   Data Tier        |
            +-------------------+  +-------------------+

Because spokes never peer with each other, all inter-spoke ("east-west") traffic is forced through the Hub, giving you a single choke point to inspect and filter every flow.

Do not assume peering alone enforces this. VNet peering only establishes reachability — the routing and firewall rules described in the next section are what actually force traffic through the Hub.

2. Forced Tunneling with User Defined Routes

By default, a VM's outbound traffic goes straight to the internet. To force it through the Hub firewall instead, attach a User Defined Route (UDR) to every spoke subnet that sends a default route (0.0.0.0/0) to the firewall's private IP.

Spoke Subnet (10.1.1.0/24)
         |
         v
  Route Table: 0.0.0.0/0 -> Azure Firewall private IP
         |
         v
  Azure Firewall (Hub VNet, AzureFirewallSubnet)
         |
    Application Rules (allowed FQDNs only)
         |
         v
      Internet

This is often called forced tunneling. Without a route table attached to the subnet, workloads bypass the firewall entirely regardless of how well the firewall itself is configured.

3. Centralized Egress via Azure Firewall

Once traffic is forced through the Hub, Azure Firewall becomes the enforcement point for outbound access.

Configure Application Rules to allow outbound HTTPS traffic only to the fully qualified domain names (FQDNs) a workload actually needs — for example api.github.com or mcr.microsoft.com — and let everything else be dropped by the default deny.

Do not treat the firewall as a set-and-forget appliance. FQDN allow-lists need to be reviewed whenever an application takes on a new external dependency, or legitimate traffic will start failing instead of being logged as a security event.

Azure Firewall overview

4. Micro-Segmentation with NSGs and Application Security Groups

Network Security Groups (NSGs) act as virtual firewalls on subnets and network interfaces. A resilient NSG design follows two rules:

Deny all by default — the lowest-priority rule should deny all inbound and outbound traffic, so anything not explicitly allowed is blocked.
Group by role, not by IP — Application Security Groups (ASGs) let you write rules against roles like Web-ASG or DB-ASG instead of specific IP addresses that change as resources scale.

{
  "name": "Allow-Web-To-DB",
  "properties": {
    "priority": 200,
    "access": "Allow",
    "direction": "Inbound",
    "protocol": "Tcp",
    "sourcePortRange": "*",
    "destinationPortRange": "1433",
    "sourceApplicationSecurityGroups": [{ "id": ".../applicationSecurityGroups/Web-ASG" }],
    "destinationApplicationSecurityGroups": [{ "id": ".../applicationSecurityGroups/DB-ASG" }]
  }
}

The rule above only permits SQL traffic (port 1433) from members of Web-ASG to members of DB-ASG. Everything else hitting the data-tier subnet is caught by the deny-all rule at the bottom of the NSG.

5. Securing PaaS Services with Azure Private Link

By default, Azure PaaS services — Storage Accounts, Azure SQL, Key Vault — are reachable over the public internet, protected only by identity and firewall rules on the service itself. Azure Private Link removes the public attack surface entirely by projecting the service into your VNet with a private IP address.

The configuration flow:

Disable public network access on the PaaS resource (Azure SQL, Key Vault, etc.).
Create a Private Endpoint for that resource inside a dedicated subnet in the data-tier spoke.
Point a Private DNS Zone (for example privatelink.database.windows.net or privatelink.vaultcore.azure.net) at the private IP, so existing connection strings resolve privately without any application code changes.
Workloads in peered spokes can now reach the service without traffic ever leaving the Microsoft backbone.

Do not skip step 3. Disabling public access and creating the Private Endpoint without the matching Private DNS Zone is a common failure mode — the resource is technically private, but clients still try to resolve the public hostname and connections fail.

6. Identity as the Real Perimeter

Network segmentation limits where traffic can go. It does not decide who is allowed to send it. Zero Trust's "verify explicitly" principle means access decisions should be anchored in identity, not network location.

Application
    |
    v
Managed Identity
    |
    v
Microsoft Entra ID
    |
    v
Azure SQL / Key Vault (RBAC-scoped)

Use managed identities for workload-to-PaaS authentication instead of connection strings or shared keys, and scope Microsoft Entra role assignments to the minimum required — a VM being on the "trusted" subnet should never be treated as sufficient authorization on its own.

7. Validate the Architecture, Don't Assume It

A Zero Trust network is only as good as its last verified state. Configuration drift — a route table detached during troubleshooting, a public-access flag re-enabled by a script — silently reopens the perimeter you built.

Automate two checks continuously:

DNS resolution — confirm PaaS hostnames still resolve to private (RFC 1918) addresses, not public ones.
Port reachability — confirm only the ports permitted by your NSG rules are actually reachable from a given subnet.

Resolve hostname
      |
      v
  Private IP? --- No --> ALERT: public exposure
      |
     Yes
      |
      v
Test allowed ports
      |
      v
  Only expected ports open? --- No --> ALERT: NSG drift

Run this validation after every deployment, not just once at build time — drift happens after go-live, not during it.

8. Observability

A segmented network still needs centralized visibility. Route the following into Azure Monitor / Log Analytics:

Network Logs

NSG flow logs (allowed/denied flows per rule)
Azure Firewall logs (application, network, and DNS rule hits)
Private DNS Zone query logs

Identity Logs

Microsoft Entra sign-in logs
Managed identity token requests

Infrastructure Metrics

Firewall throughput and rule processing latency
Private Endpoint connection state

A useful operational summary might look like:

+------------------------------------------------+
|           ZERO TRUST NETWORK HEALTH             |
+------------------------------------------------+
| Firewall Denies/min | NSG Denies/min | Drift    |
|         14          |        3       |  None    |
+------------------------------------------------+
| Private Endpoints    | Public Exposure          |
|    HEALTHY (4/4)     |    None Detected         |
+------------------------------------------------+

9. Recommended Production Architecture

Putting the pieces together:

                         Internet / On-Prem
                                |
                                v
                       +------------------+
                       |    Hub VNet      |
                       |   10.0.0.0/16    |
                       |  Azure Firewall  |
                       +--------+---------+
                                |
                        Forced Tunneling (UDR)
                                |
                 +--------------+--------------+
                 |                             |
                 v                             v
        +-----------------+           +-----------------+
        |  Spoke A: App    |           |  Spoke B: Data   |
        |  10.1.0.0/16     |           |  10.2.0.0/16     |
        |  NSG + ASGs      |           |  Private Endpoints|
        +-----------------+           +--------+----------+
                                                |
                                        Private DNS Zones
                                                |
                                                v
                                     Azure SQL / Key Vault

        Identity     -> Microsoft Entra ID + Managed Identities
        Monitoring   -> Azure Monitor / NSG Flow Logs / Firewall Logs
        Validation   -> Automated DNS + port-reachability checks
        IaC          -> Bicep

Operational Checklist

Before considering a Zero Trust network deployment complete, verify:

[ ] Spoke VNets peer only with the Hub, never with each other.
[ ] Every spoke subnet has a route table forcing 0.0.0.0/0 to the firewall.
[ ] Azure Firewall Application Rules allow only required FQDNs.
[ ] Every NSG ends with an explicit deny-all rule.
[ ] NSG rules reference Application Security Groups, not hardcoded IPs, where possible.
[ ] Public network access is disabled on every PaaS resource that has a Private Endpoint.
[ ] Matching Private DNS Zones are linked to every VNet that needs to resolve a private endpoint.
[ ] Workload-to-PaaS authentication uses managed identities, not shared keys.
[ ] DNS resolution and port-reachability are validated automatically after every deployment.
[ ] NSG flow logs and Firewall logs are flowing into Azure Monitor.
[ ] IaC (Bicep) is the source of truth — no manual portal changes to routing or NSGs.

Final Takeaway

Zero Trust networking in Azure is not a single toggle switch — it is a layered defense strategy where each layer assumes the others might fail.

Segmentation → forced tunneling → firewall enforcement → micro-segmentation → private connectivity → identity → continuous validation → observability.

The strongest architecture assumes a subnet or credential will eventually be compromised, and is designed so that compromise stays contained.

Zero Trust = Segmentation + Identity + Private Connectivity + Continuous Validation

Zero Trust · Hub-and-Spoke · Azure Firewall · NSG · Private Link · Bicep

Originally published on my portfolio.

FinOps in Practice: Stop Wasting Money on Idle Azure Resources

Raphael Gab-Momoh — Thu, 17 Sep 2026 07:26:00 +0000

Introduction: Where Azure Cost Waste Hides

Cloud waste rarely comes from one obviously expensive resource. It often accumulates from dozens or hundreds of small decisions:

Disks left behind after VM deletion
Unused public IP addresses
Stopped or underutilized VMs
Oversized Azure SQL databases
Old snapshots
Unused storage
Overprovisioned Kubernetes nodes
Excessive log retention
Non-production resources running 24/7
Resources without owners or cost-center tags

The goal of FinOps is not simply to "spend less."

The goal is to ensure that every cloud resource has a business purpose and that the organization gets measurable value from its cloud investment.

Azure Resource Graph overview

A good FinOps workflow looks like:

Discover
   ↓
Measure
   ↓
Analyze
   ↓
Optimize
   ↓
Automate
   ↓
Monitor

Before the how, the what — three tools this guide relies on:

Azure Resource Graph — a query service that lets you search across every resource in a subscription (or many subscriptions) at once, using a SQL-like query language (KQL). Where the Azure Portal shows you resources one blade at a time, Resource Graph answers questions like "show me every unattached disk across the whole tenant" in a single query.
Azure Policy — a service that enforces rules about what's allowed to exist in a subscription, either by blocking non-compliant resources at creation or by continuously auditing existing ones against a rule. It's the mechanism that turns "we should tag everything" from a wish into something actually checked automatically.
Azure Advisor — a built-in recommendation engine that analyzes your actual usage patterns and suggests specific actions (resize this VM, delete that idle resource) based on Microsoft's own telemetry and best-practice rules, rather than requiring you to write the analysis query yourself.

1. Finding Orphaned Managed Disks

One of the easiest places to begin is with unattached Azure Managed Disks.

When a VM is deleted, its disks may remain depending on the deletion configuration and lifecycle policies.

These disks can continue generating storage charges even though they are no longer serving an active workload.

Azure Resource Graph Query

Run the following KQL query in Azure Resource Graph Explorer:

Resources
| where type =~ "microsoft.compute/disks"
| extend diskState = tostring(properties.diskState)
| where diskState =~ "Unattached"
| project
    name,
    resourceGroup,
    subscriptionId,
    location,
    diskState,
    sku = tostring(sku.name),
    diskSizeGB = toint(properties.diskSizeGB),
    resourceId = id
| order by diskSizeGB desc

This gives you a useful inventory of potentially orphaned disks.

Important

Do not automatically delete every unattached disk.

An unattached disk may still be:

A backup disk
A disaster-recovery resource
A migration artifact
A staging resource
A manually managed data disk
Required for an upcoming deployment

Use an approval workflow before deletion.

A safer process is:

Unattached Disk
      ↓
Identify Owner
      ↓
Check Tags
      ↓
Check Activity / Age
      ↓
Confirm Business Purpose
      ↓
Approval
      ↓
Delete

2. Finding Unattached Public IP Addresses

Public IP addresses can also become orphaned when resources are deleted.

Use Azure Resource Graph to identify Public IP resources that do not have an active IP configuration.

Resources
| where type =~ "microsoft.network/publicipaddresses"
| extend ipConfig = tostring(properties.ipConfiguration.id)
| where isempty(ipConfig)
| project
    name,
    resourceGroup,
    subscriptionId,
    location,
    sku = tostring(sku.name),
    resourceId = id
| order by name asc

Again, treat the result as a candidate cleanup list, not an automatic deletion list.

A public IP may have a legitimate purpose even if it is temporarily unattached.

3. Finding Unattached Network Interfaces

Network interfaces can also remain after VM deletion.

A useful Resource Graph query is:

Resources
| where type =~ "microsoft.network/networkinterfaces"
| extend vmId = tostring(properties.virtualMachine.id)
| where isempty(vmId)
| project
    name,
    resourceGroup,
    subscriptionId,
    location,
    resourceId = id
| order by name asc

This helps identify NICs that are no longer associated with a VM.

Before deleting them, check whether they are being referenced by another architecture component or reserved for future deployment.

4. Find Old Snapshots

Snapshots can accumulate over time, particularly in environments with frequent testing, migration, or backup activities.

Use:

Resources
| where type =~ "microsoft.compute/snapshots"
| project
    name,
    resourceGroup,
    subscriptionId,
    location,
    diskSizeGB = toint(properties.diskSizeGB),
    timeCreated = todatetime(properties.timeCreated),
    resourceId = id
| order by timeCreated asc

You can then identify snapshots that exceed your organization's retention period.

For example:

Snapshot age > 90 days
        ↓
Check retention policy
        ↓
Check owner
        ↓
Check backup requirements
        ↓
Delete if approved

5. Identify Underutilized Virtual Machines

Deleting unused resources is only one part of FinOps.

A potentially larger saving opportunity is right-sizing active resources.

For VMs, examine metrics such as:

CPU utilization
Memory utilization
Network throughput
Disk I/O
Application performance
Peak utilization

For example:

VM Size: Standard_D8s_v5

Average CPU: 8%
Peak CPU:    22%

If the workload consistently operates far below its allocated capacity, investigate whether the VM can be downsized.

However, do not right-size based solely on average CPU.

Consider:

Peak traffic
Memory requirements
Application latency
Disk performance
Network requirements
Business-critical workloads
Seasonal demand

A better approach is:

Measure → Analyze → Test → Right-size → Monitor

6. Azure SQL Right-Sizing

Azure SQL is another important FinOps optimization area.

Organizations frequently provision more compute than the workload actually requires.

Review:

CPU utilization
Data I/O
Log I/O
Storage utilization
Query performance
DTU/vCore utilization
Connection counts
Workload patterns

Analyze at least several weeks of production data where possible, rather than making decisions from a single day's metrics.

7. Azure SQL Serverless

For workloads with intermittent or unpredictable usage, Azure SQL Database serverless compute can be an option.

Serverless can dynamically scale compute based on workload and can automatically pause compute during periods of inactivity when configured and supported by the workload.

This can be particularly useful for:

Development environments
Test environments
Low-volume applications
Intermittent workloads

However, serverless is not automatically cheaper for every database.

Evaluate:

Workload pattern
Auto-pause requirements
Resume latency
Minimum compute
Storage requirements
Application behaviour

FinOps means choosing the right pricing model for the workload, not simply choosing the feature with the lowest advertised price.

8. Azure SQL Elastic Pools

If an organization operates many Azure SQL databases with inconsistent utilization, Elastic Pools can allow databases to share a pool of compute resources.

For example:

Before:

Database A → Dedicated Compute
Database B → Dedicated Compute
Database C → Dedicated Compute
Database D → Dedicated Compute

Potential alternative:

After:

          Azure SQL Elastic Pool
        +------------------------+
        |                        |
        | DB A  DB B  DB C  DB D |
        |                        |
        +------------------------+

This can improve resource utilization when database workloads have different usage patterns.

Always validate workload characteristics before migrating.

9. Azure Advisor and Cost Management

Do not rely exclusively on custom Resource Graph queries.

Use Azure's native FinOps capabilities as well.

Important services include:

Azure Cost Management

Use it to analyze:

Actual spend
Forecasted spend
Cost by subscription
Cost by resource group
Cost by service
Cost by tag
Cost trends

Azure Advisor

Use recommendations related to:

Right-sizing
Idle resources
Reserved capacity
Savings opportunities
Reliability
Security

The combination provides:

Azure Cost Management
        +
Azure Advisor
        +
Azure Resource Graph
        +
Azure Monitor
        ↓
FinOps Visibility

10. Enforce Resource Tagging

FinOps becomes significantly harder when resources do not have clear ownership.

Establish a mandatory tagging strategy.

For example:

Environment = Production
Owner       = Platform-Team
CostCenter  = Engineering
Application = Customer-API
Project     = SaaS-Platform
ManagedBy   = Terraform

Use Azure Policy to audit or enforce required tags.

For example:

Resource Created
       ↓
Azure Policy
       ↓
Required Tags?
    /         YES        NO
   |          |
 Allow     Deny/Audit

Be careful with strict Deny policies in production. Introduce tagging policies gradually and validate their impact before enforcement.

11. Automate Dev/Test Shutdown

Non-production environments are often left running overnight.

For example:

Developer VM
      |
      +---- Monday–Friday
      |       08:00–19:00
      |
      +---- Weekend
              OFF

For eligible workloads, implement automated schedules using Azure-native automation capabilities.

Possible approaches include:

Azure Automation
Azure Functions
Logic Apps
VM auto-shutdown capabilities
Scheduled automation workflows

The exact implementation should account for VM dependencies and development workflows.

12. Detect Resources Without Owners

A powerful FinOps governance strategy is to identify resources that cannot be attributed to a team or application.

Example Resource Graph query:

Resources
| extend owner = tostring(tags["Owner"])
| extend environment = tostring(tags["Environment"])
| extend costCenter = tostring(tags["CostCenter"])
| where isempty(owner)
    or isempty(environment)
    or isempty(costCenter)
| project
    name,
    type,
    resourceGroup,
    subscriptionId,
    location,
    owner,
    environment,
    costCenter,
    resourceId = id

This creates a governance backlog.

Instead of immediately deleting these resources:

Missing Metadata
      ↓
Identify Owner
      ↓
Assign Cost Center
      ↓
Classify Workload
      ↓
Optimize

13. Build an Automated FinOps Workflow

A mature FinOps process should move beyond manually running queries.

A practical workflow is:

Azure Resources
      |
      v
Azure Resource Graph
      |
      v
Identify Candidates
      |
      v
Azure Monitor / Cost Management
      |
      v
Validate Utilization
      |
      v
FinOps Rules
      |
      v
Approval
      |
      +--------+
      |        |
      v        v
 Optimize   Delete
      |        |
      +--------+
           |
           v
     Cost Reduction
           |
           v
       Reporting

Automation can be implemented using:

Azure Functions
Azure Automation
Logic Apps
GitHub Actions
Azure DevOps
PowerShell
Python

For destructive actions, use an approval or quarantine stage rather than immediately deleting resources.

14. Create a Resource Quarantine Strategy

Instead of:

Detect → Delete

use:

Detect
   ↓
Tag as Candidate
   ↓
Notify Owner
   ↓
Grace Period
   ↓
Final Validation
   ↓
Approval
   ↓
Delete

For example, a candidate resource could receive:

FinOpsStatus = CleanupCandidate
CleanupAfter = 2026-04-01

This gives owners an opportunity to recover resources that were incorrectly identified.

15. Measure FinOps Success

FinOps should produce measurable business outcomes.

Track metrics such as:

Cloud Waste

Potential Waste / Total Cloud Spend

Optimization Savings

Baseline Cost - Optimized Cost

Resource Ownership

Tagged Resources / Total Resources × 100

Idle Resource Reduction

Idle Resources Before
        ↓
Idle Resources After

Also measure:

Monthly cloud spend
Cost per application
Cost per customer
Cost per transaction
Cost per environment
Forecast accuracy
Optimization savings
Commitment utilization

Operational Checklist

Before implementing automated cleanup:

[ ] Enable Azure Cost Management reporting.
[ ] Review Azure Advisor recommendations.
[ ] Inventory resources with Azure Resource Graph.
[ ] Identify unattached managed disks.
[ ] Identify unattached NICs.
[ ] Identify unattached public IPs.
[ ] Review old snapshots.
[ ] Identify underutilized VMs.
[ ] Review Azure SQL utilization.
[ ] Evaluate serverless workloads.
[ ] Evaluate SQL Elastic Pools.
[ ] Establish mandatory tagging.
[ ] Configure Azure Policy.
[ ] Implement dev/test shutdown schedules.
[ ] Establish resource ownership.
[ ] Create a cleanup-candidate workflow.
[ ] Require approval before destructive actions.
[ ] Monitor savings after optimization.
[ ] Report FinOps KPIs to stakeholders.

Final Takeaway

FinOps is not a one-time cleanup exercise.

It is a continuous engineering discipline:

Discover → Measure → Optimize → Govern → Automate → Monitor

The objective is not to make Azure as cheap as possible.

The objective is to make Azure efficient, predictable, accountable, and aligned with business value.

A mature Azure FinOps practice combines:

Azure Cost Management + Azure Advisor + Azure Resource Graph + Azure Monitor + Azure Policy + Automation

That combination turns cloud cost management from a monthly finance exercise into an automated engineering practice.

FinOps · Azure Cost Optimization · Resource Graph · Azure Policy · Governance · Cloud Economics

Originally published on my portfolio.

Supabase vs. MongoDB vs. Firebase: A Real App, Three Backends

Raphael Gab-Momoh — Thu, 17 Sep 2026 07:25:26 +0000

Comparing Marketing Pages Is Not Comparing Databases

"Supabase vs. Firebase vs. MongoDB" comparisons are common and almost all of them compare feature lists, not running systems. This one builds the same small app — a notes tool with signup, create, and list — against all three, actually runs each one, and reports exactly what happened, including the parts that didn't go smoothly. The app is deliberately minimal so the comparison isolates what each backend does differently, not what the app happens to need.

Before Step 1, three terms this comparison leans on:

Row Level Security (RLS) — Postgres's own mechanism (which Supabase builds directly on top of) for enforcing per-row access control inside the database itself. A policy like "users can only see rows where user_id matches their own ID" is enforced by Postgres on every query, regardless of what client or API hits it.
Firestore Security Rules — a separate, non-SQL rules language that Firestore evaluates on every request, independent of the database engine underneath. Functionally similar goal to RLS, but a different language, different deployment mechanism, and — as this article found — a different operational model.
BaaS (Backend as a Service) — a category both Supabase and Firebase belong to: a hosted platform that provides authentication, a database, and an auto-generated or managed API, so an app team writes little or no backend code. MongoDB, by contrast, is just a database — everything above the data layer (auth, API, authorization) has to be built separately.

Step 1 — The app, minimal on purpose

One route to create a note, one to list them, gated behind authentication:

app.post("/notes", authMiddleware, async (req, res) => { /* create a note, owned by req.userId */ });
app.get("/notes", authMiddleware, async (req, res) => { /* list notes, scoped to req.userId */ });

Why this specific feature set: create-and-list-scoped-to-a-user exercises exactly the place these three backends differ most — authentication, and per-row authorization. A feature set any smaller wouldn't touch authorization at all; anything larger would start comparing app complexity instead of backend complexity.

Step 2 — Supabase: self-hosted, zero backend code

supabase init
supabase start

That second command pulled and started twelve containers — Postgres, GoTrue (auth), PostgREST (auto-generated REST API), Realtime, Kong (API gateway), Studio, and supporting services — on a single Standard_B2s Azure VM (3.8 GB RAM). All twelve came up healthy.

The entire "backend" for this app is one SQL migration:

create table if not exists notes (
  id uuid primary key default gen_random_uuid(),
  user_id uuid not null references auth.users(id),
  title text not null,
  body text,
  created_at timestamptz default now()
);

alter table notes enable row level security;

create policy "Users can manage their own notes" on notes
  for all using (auth.uid() = user_id) with check (auth.uid() = user_id);

No server, no route handlers, no auth middleware. PostgREST inspects this table and its RLS policy and immediately exposes a real, working REST API for it. Verified end to end against the real running stack:

curl -X POST http://localhost:54321/auth/v1/signup \
  -H "apikey: $ANON_KEY" -H "Content-Type: application/json" \
  -d '{"email":"test@notescompare.dev","password":"testpassword123"}'
# -> real JWT returned

curl -X POST http://localhost:54321/rest/v1/notes \
  -H "apikey: $ANON_KEY" -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -H "Prefer: return=representation" \
  -d '{"title":"First real note","body":"via PostgREST","user_id":"'"$USER_ID"'"}'
# -> [{"id":"4cdcbee7-...","user_id":"ee5a831c-...","title":"First real note", ...}]

Why this matters more than "Supabase has a nice dashboard": the RLS policy is enforced by Postgres itself, on every request, through an API that required writing zero lines of server code. The auth token, the row-scoping, and the REST endpoint all came from a fourteen-line SQL file.

Step 3 — MongoDB: same feature, real hand-written server required

MongoDB has no auto-generated API and no built-in per-document authorization. Everything Supabase gave for free had to be written:

function authMiddleware(req, res, next) {
  const token = (req.headers.authorization || "").replace("Bearer ", "");
  if (!token) return res.status(401).json({ error: "Missing token" });
  try {
    req.userId = jwt.verify(token, JWT_SECRET).sub;
    next();
  } catch {
    res.status(401).json({ error: "Invalid token" });
  }
}

app.post("/auth/signup", async (req, res) => {
  const { email, password } = req.body;
  const passwordHash = await bcrypt.hash(password, 10);
  const result = await users.insertOne({ email, passwordHash });
  const token = jwt.sign({ sub: result.insertedId.toString() }, JWT_SECRET, { expiresIn: "1h" });
  res.json({ access_token: token });
});

Run against a real self-hosted mongo:8 container on the same VM:

curl -X POST localhost:3001/auth/signup -H 'Content-Type: application/json' \
  -d '{"email":"test3@mongocompare.dev","password":"testpassword123"}'
# -> {"access_token":"eyJhbGciOi..."}

curl -X POST localhost:3001/notes -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"title":"First mongo note","body":"via Express"}'
# -> {"userId":"6aab8be9...","title":"First mongo note", ...}

It works, and it works well — MongoDB's own document model and query API are genuinely pleasant. But the honest comparison is the line count: Supabase's backend is fourteen lines of SQL; MongoDB's equivalent required a real password-hashing scheme, a JWT signing/verification layer, and manually filtering every query by userId in application code — and that authorization logic has to be correctly repeated on every single query by hand, with nothing at the database layer enforcing it if a future route forgets to filter.

Step 4 — Firebase: a real rejection, and a real tooling wall

Firebase Auth worked immediately — signup against the real Identity Toolkit REST API returned a genuine ID token with no setup beyond a project's public API key:

curl -X POST "https://identitytoolkit.googleapis.com/v1/accounts:signUp?key=$API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"email":"test@notescompare.dev","password":"testpassword123","returnSecureToken":true}'
# -> real idToken returned

Writing to Firestore with that token was correctly rejected:

{
  "error": {
    "code": 403,
    "message": "Missing or insufficient permissions.",
    "status": "PERMISSION_DENIED"
  }
}

That rejection is Firestore's security rules doing exactly their job — the project's existing production rules didn't recognize a brand-new collection this test tried to write to, and Firestore defaults to denying anything a rule doesn't explicitly allow. The correct fix is deploying an updated rule scoping that collection to its owner, the same shape as the Supabase RLS policy above:

match /notes_compare_lab/{noteId} {
  allow read, delete: if request.auth != null && request.auth.uid == resource.data.userId;
  allow create: if request.auth != null && request.auth.uid == request.resource.data.userId;
}

Deploying that rule non-interactively hit a real wall:

Error: Request to https://firebaserules.googleapis.com/v1/projects/eguari:test had HTTP Error: 403,
The caller does not have permission

The Firebase CLI's stored login session lacked the scope needed to test and deploy security rules, and re-authenticating with broader scope requires an interactive browser login (firebase login --reauth), which explicitly refuses to run in a non-interactive session: "Cannot run login in non-interactive mode." A fresh brand-new Firebase project hit an even earlier version of the same wall — serviceusage.services.use permission denied, minutes after the project's own creation, on the account that had just created it.

This is the actual finding, not a workaround-and-move-on: Supabase and MongoDB were both fully reproducible from a script, self-hosted, with no dependency on an interactive human session. Firebase's tooling, at least for rules deployment and fresh-project setup, assumes a human is present to click through a browser at least once. That's a real operational difference between the three — not a knock on Firestore's security model itself, which behaved correctly by rejecting an unauthorized write.

Step 5 — What actually differs, stated plainly

	Supabase (self-hosted)	MongoDB (self-hosted)	Firebase
Backend code needed for this app	None (SQL only)	~70 lines (auth + API)	None (client SDK only)
Authorization enforcement	Postgres RLS, database-level	Application code, must be repeated per query	Firestore Security Rules, database-level
Self-hostable / portable	Yes — full stack via `supabase start`	Yes — official Docker image	No — hosted only
Non-interactive automation	Fully scriptable	Fully scriptable	Assumes an interactive human session for rules/project setup
Query model	SQL (Postgres)	Document (BSON)	Document (protobuf-backed)
Realtime	Built in (Postgres logical replication)	Requires building it yourself (change streams)	Built in

Closing Thoughts

The honest answer to "which is better" is that it depends on which row of that table matters most to a given team. If self-hosting and full automation matter — CI pipelines, ephemeral test environments, air-gapped deployments — Supabase and MongoDB are both genuinely reproducible from a script; Firebase, at least as tested here, is not. If minimizing backend code matters most, Supabase's RLS-enforced auto-API and Firebase's client-SDK model both beat MongoDB's "write it yourself" reality — but only Supabase's version of that convenience turned out to be scriptable end to end in this test. None of these conclusions came from reading documentation; they came from actually running all three and writing down exactly what happened, rejection messages included.

GitHub Repository: notes-compare-lab — all three implementations, the exact commands run, and the real error messages this article quotes.

Reviewed against Supabase CLI, MongoDB 8, and the Firebase CLI as of September 2026.

Supabase · MongoDB · Firebase · PostgreSQL · Firestore · Backend as a Service

Originally published on my portfolio.

Docker Compose for Real Local Dev Parity

Raphael Gab-Momoh — Thu, 17 Sep 2026 07:24:51 +0000

The Compose File That Looks Right on Paper

Most Docker Compose tutorials show an app service and a database service, wire them together with depends_on, and call it done. The gap between that and a Compose setup that actually mirrors production is usually invisible until the app starts before the database is ready to accept connections, or a "restart" quietly wipes data nobody meant to lose.

This article builds a real two-service app — Express talking to Postgres, with an endpoint that actually writes and reads rows — and proves two specific things that a Compose file looks like it does but doesn't automatically do: that the app genuinely waits for the database to be ready, and that data survives a full teardown, not just a container restart.

Before Step 1, one term this walkthrough leans on:

depends_on with condition: service_healthy — plain depends_on: [db] only waits for the database container to start, not for Postgres inside it to actually be accepting connections — those are different moments, often seconds apart. condition: service_healthy makes Compose wait for the dependency's own healthcheck to pass first, which is the actual guarantee an app needs before its first query.

Step 1 — An app with a real reason to need a database

app.get("/visits", async (req, res) => {
  try {
    await pool.query(
      "CREATE TABLE IF NOT EXISTS visits (id SERIAL PRIMARY KEY, seen_at TIMESTAMPTZ DEFAULT now())"
    );
    await pool.query("INSERT INTO visits DEFAULT VALUES");
    const result = await pool.query("SELECT COUNT(*) FROM visits");
    res.json({ visits: Number(result.rows[0].count) });
  } catch (err) {
    res.status(500).json({ error: String(err) });
  }
});

Why a visit counter and not a health check stub: the whole point of this article is proving data persistence, which requires a route that actually writes something meaningful to Postgres and a way to observe that value changing and then surviving. A counter that increments on every request is the smallest thing that does both.

Step 2 — The Compose file, with the two guarantees made explicit

services:
  app:
    build: .
    ports:
      - "3000:3000"
    environment:
      DB_HOST: db
      DB_PORT: 5432
      DB_USER: appuser
      DB_PASSWORD: ${DB_PASSWORD:-devpassword}
      DB_NAME: compose_lab
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:17-alpine
    environment:
      POSTGRES_USER: appuser
      POSTGRES_PASSWORD: ${DB_PASSWORD:-devpassword}
      POSTGRES_DB: compose_lab
    volumes:
      - db-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U appuser -d compose_lab"]
      interval: 5s
      timeout: 5s
      retries: 5

volumes:
  db-data:

Why DB_HOST: db works at all: Compose creates a private network for the whole file automatically, and every service can reach every other service by its service name as a hostname — app connects to db, not to localhost or a hardcoded IP, because Compose's built-in DNS resolves db to whichever container is currently running that service.

Why the healthcheck runs pg_isready, not just a TCP check: Postgres's own port can accept a TCP connection slightly before the database has finished its startup sequence and is ready to run real queries. pg_isready is Postgres's own readiness probe — the same signal a production orchestrator would use — so condition: service_healthy is waiting on the same guarantee a real deployment would.

Why the volume is named, not a host bind mount: db-data:/var/lib/postgresql/data creates a Docker-managed named volume, independent of any specific container. When the db container is removed and recreated, Compose reattaches the same volume — the data was never inside the container's own writable layer to begin with.

Step 3 — Bringing it up for real (no local Docker, so this ran on a real Azure VM)

sudo docker compose up -d --build

 Container compose-app-db-1 Starting
 Container compose-app-db-1 Started
 Container compose-app-db-1 Waiting
 Container compose-app-db-1 Healthy
 Container compose-app-app-1 Starting
 Container compose-app-app-1 Started

That Waiting → Healthy step, appearing between the database starting and the app starting, is condition: service_healthy doing its actual job — visible in the log, not just configured and hoped for.

curl -s localhost:3000/visits
curl -s localhost:3000/visits
curl -s localhost:3000/visits

{"visits":1}
{"visits":2}
{"visits":3}

Step 4 — Proving persistence means tearing everything down, not restarting

A container restart is a weak test — the same container, same writable layer, same everything, just re-executed. Real proof means removing the containers entirely and recreating them from the Compose file, which is what docker compose down followed by docker compose up actually does:

sudo docker compose down
sudo docker compose up -d

 Container compose-app-app-1 Removed
 Container compose-app-db-1 Removed
 Network compose-app_default Removed
 Network compose-app_default Created
 Container compose-app-db-1 Created
 Container compose-app-app-1 Created
 Container compose-app-db-1 Healthy
 Container compose-app-app-1 Started

curl -s localhost:3000/visits

{"visits":5}

Both containers were fully removed and rebuilt from scratch — new container IDs, new network — and the counter picked up at 5, exactly where it left off, instead of resetting to 1. That's the named volume doing its job: the containers are disposable, the data isn't, and this is the one moment this article actually proves that distinction rather than describing it.

Closing Thoughts

A Compose file with depends_on and a volumes: block looks like it guarantees startup ordering and data persistence — most of the syntax exists specifically for that purpose. Whether it actually delivers on either guarantee is a different question, answered by watching the Waiting → Healthy transition happen in real output, and by tearing the whole stack down and confirming the data comes back rather than assuming a restart proved anything. Local dev parity with production isn't the YAML syntax — it's whether the failure modes production cares about (a service starting before its dependency is ready, a container recreation losing state) are things you've actually watched not happen.

GitHub Repository: docker-multistage-lab — the compose-app/ directory has the full Express + Postgres app, the Compose file, and a .env.example for the database password.

Reviewed against Docker Compose v5.5.1 as of September 2026.

Docker · Docker Compose · PostgreSQL · Local Development · Health Checks

Originally published on my portfolio.

Rootless Containers: Converting a Real Image and Fixing What Breaks

Raphael Gab-Momoh — Thu, 17 Sep 2026 07:24:18 +0000

Why Most Images Still Run as Root

Docker Bench for Security, CIS benchmarks, and every container security checklist say the same thing: don't run your container process as root. Almost every tutorial Dockerfile does it anyway, because the fix looks like adding one line — USER node — and stopping there works for the simplest possible app. It doesn't work the moment the app touches the filesystem, and most articles about rootless containers never show you that failure, because they never tried to reproduce it.

This article takes the multi-stage app from the first article in this series, adds one small feature that writes a file at startup, converts it to run as a non-root user, and lets it actually crash — with the real error, not a description of one — before showing the fix.

Before Step 1, one term this walkthrough leans on:

The node user — the official node Docker images ship a pre-created, non-root user literally named node (UID 1000), specifically so you don't have to create your own with useradd. USER node in a Dockerfile switches every subsequent instruction, and the final container process, to run as that user instead of root — but it does nothing about file ownership for anything already copied into the image before that line runs.

Step 1 — Giving the app a reason to fail

import express from "express";
import fs from "fs";
import path from "path";

const app = express();
const PORT = process.env.PORT || 3000;
const DATA_DIR = path.join(process.cwd(), "data");
const STARTUP_LOG = path.join(DATA_DIR, "started.txt");

app.get("/healthz", (req, res) => {
  res.json({ status: "ok" });
});

app.listen(PORT, () => {
  fs.mkdirSync(DATA_DIR, { recursive: true });
  fs.writeFileSync(STARTUP_LOG, `started at ${new Date().toISOString()}\n`, { flag: "a" });
  console.log(`listening on ${PORT}`);
});

Why this one change matters for the article: the previous two articles in this series used an app that never wrote to the filesystem at runtime, which is exactly why a naive USER line would have worked fine on them — and would have taught nothing. Real applications write logs, cache files, or uploaded content somewhere. fs.mkdirSync on startup is the smallest possible version of that.

Step 2 — The naive conversion

FROM node:22-alpine AS build
WORKDIR /app
COPY package.json ./
RUN npm install
COPY . .
RUN npm run build

FROM node:22-alpine AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY package.json ./
RUN npm install --omit=dev
COPY --from=build /app/dist ./dist
USER node
EXPOSE 3000
CMD ["node", "dist/server.js"]

Why this looks correct and isn't: every COPY and RUN instruction before USER node executes as root by default — that's Docker's own default, not something this Dockerfile opted into. WORKDIR /app creates /app owned by root. COPY --from=build /app/dist ./dist copies files in as root. The USER node line switches who runs the container process, but it has no effect on files and directories that already exist with root ownership from the layers built before it.

Step 3 — Running it for real and reading the actual crash

No local Docker daemon was available for this either, so the built image was run on Azure Container Instances instead — pull it, run it, read the logs, same idea as running it locally:

az acr build --registry $ACR_NAME --image rootless-lab:broken -f Dockerfile.rootless-broken .

TOKEN=$(az acr login --name $ACR_NAME --expose-token --output tsv --query accessToken)
az container create --resource-group $RG --name rootless-broken-test \
  --image $ACR_NAME.azurecr.io/rootless-lab:broken \
  --registry-login-server $ACR_NAME.azurecr.io \
  --registry-username 00000000-0000-0000-0000-000000000000 \
  --registry-password "$TOKEN" \
  --cpu 1 --memory 1 --restart-policy Never

az container logs --resource-group $RG --name rootless-broken-test

node:fs:1370
  const result = binding.mkdir(
                         ^

Error: EACCES: permission denied, mkdir '/app/data'
    at Object.mkdirSync (node:fs:1370:26)
    at Server.<anonymous> (file:///app/dist/server.js:12:8)
    ...
  errno: -13,
  code: 'EACCES',
  syscall: 'mkdir',
  path: '/app/data'
}

The container group's own state confirmed it: "state": "Failed". This is the exact failure mode that makes teams either give up on rootless containers or — worse — quietly add USER root back in a "temporary" fix that never gets reverted. Neither is necessary once the actual cause is visible.

Step 4 — The real fix: own the directory before switching users

FROM node:22-alpine AS build
WORKDIR /app
COPY package.json ./
RUN npm install
COPY . .
RUN npm run build

FROM node:22-alpine AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY package.json ./
RUN npm install --omit=dev
COPY --from=build /app/dist ./dist
RUN mkdir -p /app/data && chown -R node:node /app
USER node
EXPOSE 3000
CMD ["node", "dist/server.js"]

Why chown -R node:node /app has to happen before USER node, not after: chown itself requires root privileges to change ownership to a different user — running it as node would fail for the same reason the app's own mkdir failed. The ordering matters: root creates and pre-provisions the directory the app will need, hands ownership to node, and only then steps aside.

Step 5 — Verifying the fix, including who's actually running

az acr build --registry $ACR_NAME --image rootless-lab:fixed -f Dockerfile.rootless-fixed .
# ...create container the same way, pointing at :fixed...

az container logs --resource-group $RG --name rootless-fixed-test
# listening on 3000

curl http://<container-ip>:3000/healthz
# {"status":"ok"}

az container exec --resource-group $RG --name rootless-fixed-test --exec-command "id"
# uid=1000(node) gid=1000(node)

Three separate pieces of evidence, not one: the container's own logs show it started without crashing, curl proves the HTTP server actually answers, and az container exec ... id proves the process inside is genuinely running as UID 1000, not root — the actual property this whole exercise was for, confirmed rather than assumed from reading the Dockerfile.

Closing Thoughts

The gap between "add USER node" and "actually run rootless without breaking" is exactly one chown — but that gap is invisible until something inside the container tries to write to a directory it doesn't own, and most demo apps never do. The real lesson generalizes past this one fs.mkdirSync call: any container that logs to a file, caches to disk, or accepts uploads needs its writable directories explicitly handed to the non-root user before the app ever starts, and the only way to know you've done that correctly is to run it and watch it not crash — not to read the Dockerfile and assume it's fine.

GitHub Repository: docker-multistage-lab — includes Dockerfile.rootless-broken (reproduces the real EACCES crash) and Dockerfile.rootless-fixed, both runnable exactly as shown above.

Reviewed against current Azure CLI (az acr build, az container create, az container exec) as of September 2026.

Docker · Rootless Containers · Container Security · Azure Container Instances

Originally published on my portfolio.

Networking MYRIX: Hub-Spoke VNets, Bastion, and Load-Balancer Health on Azure

Raphael Gab-Momoh — Thu, 17 Sep 2026 07:23:43 +0000

Why This Article Isn't About the App

MYRIX is an AI-native competitive action-strategy game built with Flutter and the Flame engine — real, playable, running today on Firebase Hosting. But this article isn't about the game. It's about a question every previous article in this series has quietly skipped: how does traffic actually get to a VM, and how much of that path should be exposed to the internet at all?

Every prior Azure article in this series (Container Apps, App Service, a plain VM) put the compute resource directly behind a public IP or a managed platform's own ingress. This one does neither. MYRIX's container runs on a VM with no public IP whatsoever — not for the app, not for SSH. Every drop of traffic that reaches it, whether that's a player's browser or an engineer's terminal, passes through a piece of Azure networking built specifically to carry it there. That's the actual subject: VNets, subnets, peering, Bastion, and a Load Balancer health probe that does real monitoring work before Azure Monitor ever gets involved.

Before Step 1, the concepts this design leans on:

Hub-spoke topology — one central VNet (the hub) holds shared infrastructure (here, Azure Bastion); one or more spoke VNets hold workloads (here, the MYRIX VM). Spokes don't talk to each other directly; everything routes through, or is managed from, the hub. It's the same pattern a real Azure landing zone uses at much larger scale — this article builds the smallest version of it that's still structurally real.
VNet peering — a private, Microsoft-backbone connection between two VNets, as if they were one network, without any traffic touching the public internet. Peering is not automatically symmetric: each VNet needs its own peering resource pointing at the other.
Azure Bastion — a managed jump-host service. Instead of giving a VM a public IP so you can SSH into it, Bastion sits in the hub VNet with its own public IP, and tunnels an SSH session from your terminal, through Azure's backbone, to the VM's private IP. The VM's SSH port is never internet-facing.
Load Balancer health probe — a Standard Load Balancer doesn't just forward traffic; it repeatedly checks a URL path on each backend instance and only routes traffic to instances that are currently passing. This is networking-layer health monitoring, distinct from and earlier than anything Azure Monitor does.

Here's the full shape of it before any command runs:

                     INTERNET
                        |
                        v
        +----------------------------+
        |     Standard Load          |
        |     Balancer               |
        |  :80 -> /healthz probe     |
        +----------------------------+
                        |
                        v
        +----------------------------+
        |        SPOKE VNET          |
        |        10.1.0.0/16         |
        |                            |
        |    snet-web  10.1.0.0/24   |
        |         |                  |
        |         v                  |
        |    +------------------+    |
        |    |    vm-myrix      |    |
        |    |    private IP    |    |
        |    |    :80    :22    |    |
        |    +------------------+    |
        +----------------------------+
                        ^
                        |
                   VNet Peering
                        |
        +----------------------------+
        |         HUB VNET           |
        |        10.0.0.0/16         |
        |                            |
        |   AzureBastionSubnet       |
        |       10.0.0.0/26          |
        |            |               |
        |            v               |
        |       Azure Bastion        |
        +----------------------------+
                        ^
                        |
                  Engineer SSH
              (via Bastion tunnel)


                  MONITORING

     vm-myrix -> AMA -> DCR -> Log Analytics

Two paths reach this deployment, and they never cross: internet traffic comes in only through the Load Balancer on port 80; management access comes in only through Bastion, tunneled to port 22. Neither path ever touches a public IP that belongs to the VM itself — it doesn't have one.

Every command below was run against a live subscription and verified before being written here — including one real failure, caught and fixed during the container build, documented in Step 5 rather than smoothed over.

Step 1 — Resource group and two non-overlapping VNets

RG=myrix-net-rg
LOC=eastus
az group create --name $RG --location $LOC

az network vnet create --resource-group $RG --name vnet-hub --location $LOC \
  --address-prefixes 10.0.0.0/16 \
  --subnet-name AzureBastionSubnet --subnet-prefixes 10.0.0.0/26

az network vnet create --resource-group $RG --name vnet-spoke --location $LOC \
  --address-prefixes 10.1.0.0/16 \
  --subnet-name snet-web --subnet-prefixes 10.1.0.0/24

Why the subnet must be named exactly AzureBastionSubnet: Azure Bastion refuses to deploy into a subnet with any other name — this isn't a convention, it's a hard requirement enforced at deployment time.

Why the two VNets use non-overlapping address ranges (10.0.0.0/16 vs 10.1.0.0/16): peered VNets route traffic between each other based on IP address ranges. If both VNets used the same range, Azure would have no way to tell which network a given private IP actually belongs to, and peering would fail outright.

How to check that without doing binary math: the number after the slash (the CIDR prefix) says how many of the 32 bits in an IPv4 address are fixed as the "network" part — the rest are free to vary as host addresses. A smaller number after the slash means fewer fixed bits and more available addresses: /16 fixes the first 16 bits (the first two octets) and leaves 65,536 addresses free; /26, used for AzureBastionSubnet below, fixes 26 bits and leaves only 64 — plenty for a subnet that doesn't host application VMs.

For two /16 ranges, the easiest way to confirm they don't overlap is to just compare the second octet: 10.0.0.0/16 covers every address from 10.0.0.0 to 10.0.255.255; 10.1.0.0/16 covers 10.1.0.0 to 10.1.255.255. Those never share an address, because the second octet (0 vs 1) already differs — the last two octets don't even need checking. The general rule for planning a set of VNets this way: give each one a different value in whichever octet your prefix fixes. With /16s starting 10.x.0.0, that's up to 256 non-overlapping VNets (10.0 through 10.255) without ever needing a calculator — just confirm the fixed part of the address differs between any two ranges you're about to peer.

Step 2 — Peer the VNets in both directions

az network vnet peering create --resource-group $RG --name hub-to-spoke \
  --vnet-name vnet-hub --remote-vnet vnet-spoke --allow-vnet-access

az network vnet peering create --resource-group $RG --name spoke-to-hub \
  --vnet-name vnet-spoke --remote-vnet vnet-hub --allow-vnet-access

Why this is two commands, not one: peering is defined per-VNet, not per-pair. hub-to-spoke tells vnet-hub it can reach vnet-spoke; without the second command, that permission would only exist in one direction, and a resource in the spoke would have no route back to anything in the hub — including, critically, Bastion.

Step 3 — An NSG that only trusts Bastion for SSH

az network nsg create --resource-group $RG --name nsg-web

az network nsg rule create --resource-group $RG --nsg-name nsg-web -n Allow-HTTP-Inbound \
  --priority 100 --direction Inbound --access Allow --protocol Tcp \
  --source-address-prefixes Internet --destination-port-ranges 80

az network nsg rule create --resource-group $RG --nsg-name nsg-web -n Allow-SSH-From-Bastion \
  --priority 110 --direction Inbound --access Allow --protocol Tcp \
  --source-address-prefixes 10.0.0.0/26 --destination-port-ranges 22

az network vnet subnet update --resource-group $RG --vnet-name vnet-spoke --name snet-web \
  --network-security-group nsg-web

Why the SSH rule's source is 10.0.0.0/26 and not Internet: that CIDR is exactly the AzureBastionSubnet range from Step 1 — nothing else. Since the VM has no public IP at all (confirmed in Step 4), this rule is less about stopping internet attackers, who couldn't reach port 22 either way, and more about making the intended access path explicit and enforced at the network layer rather than relying on "well, there's no public IP" as the only control.

A second real bug, caught only after the Load Balancer was wired up in Step 6: curling the Load Balancer's public IP on /healthz timed out completely, even though the container was confirmed running and answering locally on the VM. The cause was a second NSG this article hadn't accounted for — az vm create in Step 4, run without an explicit --nsg flag, silently provisions its own default NSG directly on the VM's network interface, separate from nsg-web attached to the subnet here. That NIC-level NSG only had a default SSH rule; traffic on port 80 was allowed by nsg-web at the subnet but blocked by the NIC's own NSG, and Azure evaluates both — a request only gets through if every NSG in the path allows it. The fix was replacing the NIC's auto-created NSG with nsg-web itself, so the same rule set governs both layers:

az network nic update --resource-group $RG --name vm-myrixVMNic --network-security-group nsg-web

curl http://<lb-public-ip>/healthz returned ok immediately after. The lesson generalizes past this one command: a VM's effective inbound rules are the intersection of every NSG attached anywhere along the path — subnet and NIC both — not just the one you remembered attaching.

Step 4 — Azure Bastion, and a VM with no public IP

az network public-ip create --resource-group $RG --name pip-bastion --sku Standard --location $LOC

az network bastion create --resource-group $RG --name bastion-myrix --location $LOC \
  --vnet-name vnet-hub --public-ip-address pip-bastion

Bastion takes several minutes to provision — it's a managed PaaS resource, not a VM you can inspect mid-deploy.

az vm create --resource-group $RG --name vm-myrix \
  --image Ubuntu2204 --size Standard_B2s \
  --vnet-name vnet-spoke --subnet snet-web \
  --public-ip-address "" \
  --admin-username azureuser --generate-ssh-keys \
  --location $LOC

Why --public-ip-address "" matters more than it looks: az vm create allocates a public IP by default if you don't override it. Passing an explicit empty string is the only way to opt out — forgetting this flag would silently undo the entire point of putting Bastion in front of the VM.

To actually reach the VM for management, Bastion needs its native-client tunneling feature turned on (off by default) before a local terminal can tunnel through it:

az network bastion update --resource-group $RG --name bastion-myrix --enable-tunneling true

az network bastion tunnel --resource-group $RG --name bastion-myrix \
  --target-resource-id $(az vm show --resource-group $RG --name vm-myrix --query id -o tsv) \
  --resource-port 22 --port 2222

That second command blocks, holding open a local port 2222 that forwards through Bastion to the VM's SSH port. In a second terminal, ssh azureuser@localhost -p 2222 lands on the VM — having gone through Azure's backbone the entire way, never touching a public IP that belongs to the VM.

Step 5 — Building the image, and a real bug caught by the build itself

ACR_NAME=myrixacr$RANDOM
az acr create --resource-group $RG --name $ACR_NAME --sku Basic --admin-enabled false

az acr build --registry $ACR_NAME --image myrix:latest .

The Dockerfile is a two-stage build — a cirruslabs/flutter image runs flutter build web --release, and an nginx:alpine runtime stage serves the compiled output:

FROM ghcr.io/cirruslabs/flutter:3.38.4 AS build
WORKDIR /app
COPY pubspec.yaml pubspec.lock ./
RUN flutter pub get
COPY . .
RUN flutter build web --release

FROM nginx:1.27-alpine AS runtime
COPY --from=build /app/build/web /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80

The first az acr build run actually failed, with a real Dart compile error:

lib/components/gem_component.dart:143:63:
Error: 'Color' isn't a type.
lib/components/gem_component.dart:145:18:
Error: The method 'Path' isn't defined for the type 'GemComponent'.

That's confusing on its face — gem_component.dart clearly imports package:flutter/material.dart, which exports both Color and Path, and the exact same file compiled cleanly on a local flutter build web --release run before this. The difference was the build context sent to ACR: there was no .dockerignore, so COPY . . copied the local machine's own .dart_tool/package_config.json — a file flutter pub get had already generated fresh inside the container one step earlier — clobbering it with a version full of absolute paths to this Mac's local Flutter SDK and pub cache. Inside the container, those paths don't exist, package resolution silently breaks, and the compiler reports material.dart's own exports as undefined types. The fix was a .dockerignore excluding .dart_tool, build, and .git:

.dart_tool
build
.git
.idea
*.iml

Rebuilding after that fix succeeded. This is the same class of lesson as the ORIN article's import.meta bug — a failure that only exists because two build environments (local machine, remote container) silently diverged, and the fix is making sure they can't.

Step 6 — A Standard Load Balancer whose health probe is the first monitor

az network public-ip create --resource-group $RG --name pip-lb --sku Standard --location $LOC

az network lb create --resource-group $RG --name lb-myrix --location $LOC --sku Standard \
  --public-ip-address pip-lb --frontend-ip-name feip-myrix --backend-pool-name beap-myrix

az network lb probe create --resource-group $RG --lb-name lb-myrix -n probe-healthz \
  --protocol Http --port 80 --path /healthz

az network lb rule create --resource-group $RG --lb-name lb-myrix -n rule-http \
  --protocol Tcp --frontend-port 80 --backend-port 80 \
  --frontend-ip-name feip-myrix --backend-pool-name beap-myrix --probe-name probe-healthz

NIC_NAME=$(basename $(az vm show --resource-group $RG --name vm-myrix --query "networkProfile.networkInterfaces[0].id" -o tsv))
az network nic ip-config address-pool add --resource-group $RG --nic-name $NIC_NAME \
  --ip-config-name ipconfig-vm-myrix --lb-name lb-myrix --address-pool beap-myrix

The nginx.conf shipped in the container defines the probe's target explicitly:

location /healthz {
    access_log off;
    return 200 "ok\n";
}

Why this is a networking concept and not just a monitoring one: the Load Balancer polls GET /healthz on the VM every few seconds by default, entirely on its own, with no Azure Monitor alert or human involved. An instance that stops answering gets pulled out of rotation automatically — this is the first layer of health handling, and it happens at the load-balancing layer before any metric, log, or alert ever fires. This is also the only path that exposes anything about MYRIX to the public internet at all: port 80, and only port 80, on the Load Balancer's public IP, forwarded to the VM's private IP inside the spoke VNet.

Step 7 — VM-level monitoring: Log Analytics, the Azure Monitor Agent, and a Data Collection Rule

az monitor log-analytics workspace create --resource-group $RG --workspace-name law-myrix

az vm extension set --resource-group $RG --vm-name vm-myrix \
  --name AzureMonitorLinuxAgent --publisher Microsoft.Azure.Monitor

Installing the extension alone doesn't send anything anywhere. The Azure Monitor Agent needs to be told what to collect and where to send it — that's a separate resource, a Data Collection Rule (DCR), which has to exist and be explicitly associated with the VM before any telemetry actually reaches the workspace:

cat > dcr-myrix.json <<EOF
{
  "location": "$LOC",
  "properties": {
    "dataSources": {
      "performanceCounters": [
        {
          "name": "perfCounterDataSource",
          "streams": ["Microsoft-Perf"],
          "samplingFrequencyInSeconds": 60,
          "counterSpecifiers": [
            "\\Processor(_Total)\\% Processor Time",
            "\\Memory(*)\\% Used Memory"
          ]
        }
      ]
    },
    "destinations": {
      "logAnalytics": [
        { "workspaceResourceId": "$(az monitor log-analytics workspace show --resource-group $RG --workspace-name law-myrix --query id -o tsv)", "name": "law-destination" }
      ]
    },
    "dataFlows": [
      { "streams": ["Microsoft-Perf"], "destinations": ["law-destination"] }
    ]
  }
}
EOF

az monitor data-collection rule create --resource-group $RG --name dcr-myrix --rule-file dcr-myrix.json

az monitor data-collection rule association create --name dcr-assoc-myrix \
  --rule-id $(az monitor data-collection rule show --resource-group $RG --name dcr-myrix --query id -o tsv) \
  --resource $(az vm show --resource-group $RG --name vm-myrix --query id -o tsv)

The actual chain is VM → Azure Monitor Agent → Data Collection Rule → Log Analytics workspace, not "install the agent, data appears." The DCR here defines one stream (Microsoft-Perf, carrying CPU and memory counters) and one destination (law-myrix); a production setup would typically add Syslog or additional counter streams to the same rule rather than creating a second one.

Why this is a second, separate layer from Step 6's probe: the Load Balancer's probe answers one binary question — "is port 80 responding right now?" It has no idea whether the VM is at 95% memory, whether disk space is running low, or what the CPU trend looked like over the last six hours. This DCR-routed telemetry is what turns "up or down" into a queryable history in Log Analytics.

Step 8 — Alerts on both signals

VM_ID=$(az vm show --resource-group $RG --name vm-myrix --query id -o tsv)
az monitor metrics alert create --resource-group $RG --name alert-vm-cpu-high \
  --scopes $VM_ID --condition "avg Percentage CPU > 85" \
  --description "MYRIX VM sustained high CPU" --window-size 5m --evaluation-frequency 5m

LB_ID=$(az network lb show --resource-group $RG --name lb-myrix --query id -o tsv)
az monitor metrics alert create --resource-group $RG --name alert-lb-unhealthy \
  --scopes $LB_ID --condition "avg DipAvailability < 100" \
  --description "MYRIX backend failing /healthz probe" --window-size 5m --evaluation-frequency 1m

Why DipAvailability is the more urgent alert of the two: it's the Load Balancer's own metric for "percentage of backend instances currently passing their health probe." Alerting on it dropping below 100% catches the exact moment /healthz stops answering — independent of whether CPU, memory, or any other VM-level metric still looks perfectly normal. A container can hang or deadlock without ever spiking CPU; this alert catches that failure mode, the VM-level one doesn't.

Closing Thoughts

Every other article in this series exposed a compute resource to the internet more or less directly — a Container App's own ingress, an App Service's default hostname, a VM with a public IP and an open port. This one deliberately didn't. The only thing reachable from the internet at all is port 80 on a Load Balancer's public IP; the VM itself, and the SSH access needed to manage it, sit entirely inside a private network that only Bastion can reach. That's not extra complexity for its own sake — it's what "networking concepts" actually buys an application: a much smaller, more deliberate surface area than "give everything a public IP and lock it down with firewall rules after the fact."

GitHub Repository: myrix — the real Flutter/Flame game repository, now with the Dockerfile, nginx health endpoint, and this article's complete networking setup documented in docs/AZURE_NETWORKING.md.

Reviewed against current Azure CLI (az network vnet, az network bastion, az network lb, az monitor) as of September 2026.

Azure Networking · VNet Peering · Azure Bastion · Load Balancer · Azure Monitor · Hub-Spoke Topology

Originally published on my portfolio.