DEV Community

Cover image for Kubernetes on ACK: Cloud-Native Operations and Troubleshooting on Alibaba Cloud
Raphael Gab-Momoh
Raphael Gab-Momoh

Posted on Originally published at raphaelgmomoh.pages.dev

Kubernetes on ACK: Cloud-Native Operations and Troubleshooting on Alibaba Cloud

#kubernetes #ack #alibabacloud #architecture

ACK in the Managed Kubernetes Landscape

Container Service for Kubernetes (ACK) is Alibaba Cloud's managed Kubernetes offering — its counterpart to AKS on Azure and EKS on AWS. For anyone who has operated AKS or EKS, ACK's control-plane abstraction, node pool model, and CNCF-conformant API surface will feel immediately familiar. The differences that matter are in the tooling around the cluster: identity (RAM instead of Entra ID/IAM), networking (Terway CNI), and the console/CLI ecosystem (aliyun CLI, kubectl with the ACK credential plugin).

I provisioned a production-shaped ACK cluster end to end, ran a real load test against it, and deliberately let the autoscaler fall behind so the troubleshooting section below is the actual diagnostic path I walked, not a narrated version of one. This guide covers provisioning that cluster, configuring autoscaling, exposing services, and — the part most tutorials skip — the actual diagnostic workflow for troubleshooting a cluster under real incident pressure. The Terraform and the full diagnostic script are in the companion repo, not just quoted here.

Before the how, the what — three terms this guide leans on:

  • Managed Kubernetes — a Kubernetes cluster where the cloud provider runs and patches the control plane (the API server, scheduler, etcd) for you, so you only manage the worker nodes and what runs on them. ACK, AKS, and EKS are all this same model with different branding — the "managed" part is specifically what you're paying for and what removes a large category of operational burden.
  • Node pool — a group of worker machines in the cluster sharing the same instance type and configuration. A cluster usually has more than one node pool (e.g. a standard pool for steady workloads, a Spot pool for interruption-tolerant ones), each scaling and pricing independently.
  • CNI (Container Network Interface) — the plugin that actually wires up pod networking — assigning IP addresses to pods and routing traffic between them. Terway is Alibaba Cloud's CNI (roughly analogous to Azure CNI); it's worth knowing by name because CNI-specific limits (like ENI count per node) are a real, easy-to-miss source of "why won't this pod schedule" incidents.

1. Provisioning an ACK Cluster (Terraform) 

# Control plane only — worker nodes are managed as a separate node pool
# below. (worker_instance_types/worker_number/worker_disk_category were
# removed from this resource in provider 1.212+; alicloud_cs_kubernetes
# is the dedicated-master variant and requires master_instance_types/
# master_vswitch_ids, so ack.pro.small's managed control plane uses
# alicloud_cs_managed_kubernetes instead.)
resource "alicloud_cs_managed_kubernetes" "prod" {
  name            = "ack-prod-cluster"
  cluster_spec    = "ack.pro.small" # Professional (managed control plane) tier
  vswitch_ids     = [alicloud_vswitch.app.id]
  new_nat_gateway = true
  service_cidr    = "172.20.0.0/16"
  pod_cidr        = "172.21.0.0/16"

  addons {
    name = "terway-eniip" # native VPC-routed CNI, comparable to Azure CNI
  }
  addons {
    name = "csi-plugin"
  }
  addons {
    name = "csi-provisioner"
  }
}

resource "alicloud_cs_kubernetes_node_pool" "prod_workers" {
  cluster_id           = alicloud_cs_managed_kubernetes.prod.id
  node_pool_name       = "prod-workers"
  vswitch_ids          = [alicloud_vswitch.app.id]
  instance_types       = ["ecs.g6.xlarge"]
  desired_size         = 3
  system_disk_category = "cloud_essd"
}
Enter fullscreen mode Exit fullscreen mode

ack.pro.small provisions a managed, SLA-backed control plane — the equivalent of choosing an AKS cluster with the Uptime SLA add-on rather than the free-tier control plane. For production workloads, this isn't optional.

2. Connecting kubectl 

aliyun cs GET /k8s/$CLUSTER_ID/user_config --header "Content-Type=application/json" > kubeconfig.json
export KUBECONFIG=./kubeconfig.json
kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

For CI/CD pipelines, generate a RAM-scoped kubeconfig rather than reusing an individual engineer's credentials — the same principle as scoping an AKS pipeline to a dedicated service principal rather than a personal Entra ID account.

3. Node Pool Autoscaling

ACK's autoscaler (built on the same cluster-autoscaler upstream project used by AKS/EKS) scales node pools based on unschedulable pod pressure:

resource "alicloud_cs_kubernetes_node_pool" "spot_pool" {
  cluster_id           = alicloud_cs_managed_kubernetes.prod.id
  node_pool_name       = "spot-worker-pool"
  vswitch_ids          = [alicloud_vswitch.app.id]
  instance_types       = ["ecs.g6.xlarge", "ecs.g6.2xlarge"]
  spot_strategy        = "SpotAsPriceGo"
  desired_size         = 2

  scaling_config {
    min_size = 1
    max_size = 10
  }
}
Enter fullscreen mode Exit fullscreen mode

Mixing a Spot-priced node pool alongside an on-demand baseline pool is the same FinOps pattern used on AKS spot node pools — schedule stateless, interruption-tolerant workloads (batch jobs, CI runners, stateless API replicas) onto the Spot pool via taints and tolerations, keep stateful/critical workloads on the guaranteed on-demand pool.

tolerations:
  - key: "spot-instance"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"
Enter fullscreen mode Exit fullscreen mode

4. Exposing Services: SLB Ingress 

apiVersion: v1
kind: Service
metadata:
  name: api-service
  annotations:
    service.beta.kubernetes.io/alicloud-loadbalancer-spec: "slb.s2.small"
spec:
  type: LoadBalancer
  selector:
    app: api
  ports:
    - port: 443
      targetPort: 8443
Enter fullscreen mode Exit fullscreen mode

The LoadBalancer service type on ACK provisions a real SLB automatically, the same pattern as an AKS LoadBalancer service provisioning an Azure Load Balancer — cloud-controller-manager does the provider-specific plumbing so your manifests stay portable.

5. The Troubleshooting Workflow

This is the part that separates "I deployed a cluster" from "I operate a cluster." When something breaks in production, the diagnostic path is the same discipline regardless of which managed Kubernetes you're on:

Step 1 — Is it the workload or the platform?

kubectl get pods -A --field-selector=status.phase!=Running
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous
Enter fullscreen mode Exit fullscreen mode

Step 2 — Is it resource pressure?

kubectl top nodes
kubectl top pods -A --sort-by=memory
kubectl get events -A --sort-by='.lastTimestamp' | tail -30
Enter fullscreen mode Exit fullscreen mode

A pod stuck in Pending with an event reading 0/3 nodes are available: insufficient cpu means the autoscaler hasn't caught up yet, or the node pool's max_size cap has been hit — check the autoscaler's own logs:

kubectl logs -n kube-system -l app=cluster-autoscaler --tail=100
Enter fullscreen mode Exit fullscreen mode

Step 3 — Is it networking?

Terway CNI issues typically surface as pods stuck in ContainerCreating with ENI allocation errors. Check ENI quota against the ECS instance type — smaller instance types support fewer attached ENIs, which caps pod density per node regardless of CPU/memory headroom:

kubectl describe pod <pod-name> -n <namespace> | grep -A5 Events
Enter fullscreen mode Exit fullscreen mode

Step 4 — Is it the control plane or an addon?

kubectl get componentstatuses
kubectl get pods -n kube-system
Enter fullscreen mode Exit fullscreen mode

On the Professional (managed) tier, control-plane health is largely Alibaba's responsibility — but addon health (CSI plugin, Terway, ack-virtual-node) is not, and addon failures present identically to control-plane failures from the workload's perspective.

6. Observability

Wire ACK into the same three pillars you'd wire AKS into:

  • Metrics — Prometheus via the ARMS (Application Real-Time Monitoring Service) managed integration, or self-hosted kube-prometheus-stack.
  • Logs — Log Service (SLS) DaemonSet collector, Alibaba's equivalent of Azure Monitor Container Insights.
  • Traces — OpenTelemetry Collector, provider-agnostic by design.

Closing Thoughts

Kubernetes' portability promise mostly holds at the workload layer — manifests, Helm charts, and operators translate almost unchanged between AKS, EKS, and ACK. What doesn't translate automatically is the operational muscle memory: knowing which kubectl describe output to check first, understanding your CNI's specific failure modes, and knowing where the managed control plane's responsibility ends and yours begins.

That operational fluency — not the YAML — is what actually gets tested in a production incident, on any cloud.

GitHub Repository: ack-kubernetes-operations-lab — the ACK cluster Terraform and the complete diagnostic workflow script, ready to run.

Kubernetes · ACK · Alibaba Cloud · Cluster Autoscaler · Terway CNI · Troubleshooting

Originally published on my portfolio.

Top comments (0)