DEV Community

Cover image for Kubernetes on AKS: From Zero to Production Workloads - FinOps: Measure, Price and Cut Cluster Costs (Part 8/9)
Rahimah Sulayman
Rahimah Sulayman

Posted on

Kubernetes on AKS: From Zero to Production Workloads - FinOps: Measure, Price and Cut Cluster Costs (Part 8/9)

#finops #azure #monitoring #cloudcostoptimization

Introduction

Cloud-native infrastructure is not production-ready simply because its workloads are running. It must also be measurable, cost-aware, and configured to avoid paying for unnecessary capacity.

In this mission, I evaluated the resource consumption of my Azure Kubernetes Service (AKS) cluster, examined Kubernetes resource requests and limits, queried the Azure Retail Prices API to estimate compute costs, and implemented workload and node-pool scaling controls. I also removed an externally exposed LoadBalancer Service, tested the AKS stop/start lifecycle, and documented the findings and optimization actions in COST-REPORT.md.

The objective was to connect Kubernetes resource telemetry with Azure pricing and practical cost-control measures, while distinguishing observed utilization from requested capacity, configured limits, and estimated charges.

What This Part Covers

  • Measuring node and Pod resource usage.
  • Inspecting CPU and memory requests and limits.
  • Installing jq to process Azure pricing data.
  • Querying the Azure Retail Prices API for the VM size used by the cluster.
  • Estimating the monthly compute cost.
  • Configuring workload autoscaling with a Horizontal Pod Autoscaler (HPA).
  • Enabling cluster autoscaling for the AKS node pool.
  • Removing the public LoadBalancer Service.
  • Testing AKS stop/start operations.
  • Creating and committing a cost report.

1. Measure Actual Resource Utilization

I started with the node-level metrics:

kubectl top nodes
Enter fullscreen mode Exit fullscreen mode

The output showed:

NAME                                CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
aks-nodepool1-33590657-vmss000000   242m         12%      1479Mi          25%
Enter fullscreen mode Exit fullscreen mode

top

At the time of measurement, the node was using approximately 12% of its reported CPU capacity and 25% of its memory capacity.

These figures provide a snapshot of actual resource consumption. They are useful for identifying whether the node is under pressure or operating with substantial unused capacity, although a single observation does not establish the workload's peak or long-term requirements.

Inspect the Pods consuming CPU

I then inspected the Pods across namespaces, sorted by CPU usage:

kubectl top pods -A --sort-by=cpu | head -15
Enter fullscreen mode Exit fullscreen mode

kubectl top pods

The output included system components in kube-system and application workloads in k8s-lab, including:

  • csi-azurefile-node
  • azure-cns
  • metrics-server
  • konnectivity-agent
  • coredns
  • kube-proxy
  • web Deployment Pods
  • node-agent
  • store-0 and store-1

The displayed CPU consumption of the application Pods was low at the time of measurement. This helped establish that the cluster's workloads were not consuming a substantial share of the node's available CPU during the observed interval.

The distinction matters because a Kubernetes cluster can have a relatively low current workload while still reserving significant capacity through its configured resource requests.

2. Compare Resource Requests, Limits and Actual Usage

To understand how much capacity the workloads had requested or been permitted to use, I inspected the node's allocated resources:

kubectl describe nodes | grep -A 8 "Allocated resources"
Enter fullscreen mode Exit fullscreen mode

kubectl describe nodes

The output showed:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests       Limits
  --------           --------       ------
  cpu                1187m (62%)    10692m (562%)
  memory             1330Mi (22%)   13443360Ki (226%)
Enter fullscreen mode Exit fullscreen mode

The figures reveal three different aspects of resource management.

Metric CPU Memory
Observed node utilization 12% 25%
Allocated requests 62% 22%
Allocated limits 562% 226%

Actual utilization represents observed consumption at the time of measurement.

Requests represent the capacity declared by workloads for scheduling purposes. CPU requests of 62% indicate that workloads had reserved a significant proportion of the node's allocatable CPU capacity, even though observed CPU utilization was only 12%.

Limits define upper resource boundaries for individual containers. Their aggregated percentages can exceed 100% because Kubernetes permits resource overcommitment. The reported CPU limit of 562% therefore does not mean that the node was actually using more than five times its CPU capacity.

These measurements should not be interpreted interchangeably. Low current utilization can indicate an opportunity to review requests, but reducing them requires an understanding of the workload's peak behavior, performance requirements, and reliability objectives.

Verify the Deployment's resource settings

I inspected the container configuration for the web Deployment:

kubectl get deployment web -n k8s-lab -o json \
  | jq '.spec.template.spec.containers[] |
        {name: .name, requests: .resources.requests, limits: .resources.limits}'
Enter fullscreen mode Exit fullscreen mode

The result showed:

{
  "name": "nginx",
  "requests": {
    "cpu": "50m",
    "memory": "64Mi"
  },
  "limits": {
    "cpu": "200m",
    "memory": "128Mi"
  }
}
Enter fullscreen mode Exit fullscreen mode

Each NGINX container requested 50m CPU and 64Mi memory, with limits of 200m CPU and 128Mi memory.

With three replicas, those declared requests contribute to the Deployment's aggregate resource reservations. Repeating this comparison across workloads helps identify where requests may be substantially higher than typical consumption or where capacity may be constrained.

The goal is not to minimize requests indiscriminately. It is to align resource allocation with actual workload requirements while preserving sufficient headroom for bursts and normal operating variation.

3. Install jq for Pricing Data Analysis

To process the JSON returned by Azure's pricing endpoint, I installed jq in the Linux environment:

sudo apt-get install -y jq
Enter fullscreen mode Exit fullscreen mode

install

The package manager installed jq and its required dependencies successfully.

This utility made it easier to filter the Azure Retail Prices API response and extract fields such as the product name, meter, price, unit of measure, region, and price type.

4. Retrieve VM Pricing from the Azure Retail Prices API

The cluster configuration identified the following environment values:

RESOURCE_GROUP=rg-aks-lab
CLUSTER_NAME=aks-lab
LOCATION=westeurope
NODE_SIZE=Standard_D2s_v3
Enter fullscreen mode Exit fullscreen mode

I queried the Azure Retail Prices API for virtual machine pricing matching the region and VM size:

curl -sG "https://prices.azure.com/api/retail/prices" \
  --data-urlencode "\$filter=serviceName eq 'Virtual Machines' and armRegionName eq '$LOCATION' and armSkuName eq '$NODE_SIZE' and priceType eq 'Consumption'" \
  | jq -r '.Items[]
    | select(.meterName | test("Spot|Low Priority"; "i") | not)
    | [.productName, .skuName, .meterName, .retailPrice, .unitOfMeasure, .armRegionName, .priceType]
    | @tsv'
Enter fullscreen mode Exit fullscreen mode

cost

The response contained multiple pricing records for the requested VM SKU. The available records included different operating-system products and pricing types, so the SKU alone was not sufficient to select the correct rate.

I compared the returned fields to distinguish the applicable consumption rate from Windows-specific, DevTest, Spot, and Low Priority entries.

For the cost report, I used an observed public retail consumption rate of $0.12 per hour for the Standard_D2s_v3 VM in westeurope.

Estimate the monthly VM compute cost

Using a simplified month of 730 hours, the calculation was:

[
\text{Estimated monthly cost} =
\text{Hourly rate} \times 730
]

[
\$0.12 \times 730 = \$87.60
]

The resulting estimate was:

Item Estimate
VM size Standard_D2s_v3
Region westeurope
Hourly rate used $0.12
Assumed monthly hours 730
Estimated monthly VM compute cost $87.60

This is a price-based estimate, not a confirmed Azure invoice amount. The calculation covers the VM rate used in the report and excludes additional AKS and Azure resource charges. Actual billing can also depend on the applicable pricing record, subscription or purchasing terms, usage duration, and associated resources.

For that reason, the estimate provides a useful baseline for cost planning, but it should not be presented as the complete operating cost of the cluster.

5. Identify and Remove Unnecessary Public Exposure

I reviewed the Services across namespaces:

kubectl get services -A
Enter fullscreen mode Exit fullscreen mode

get services

The output showed the internal web-svc ClusterIP Service, the headless store Service, and the web-public LoadBalancer Service.

The public Service was exposing NGINX using this external IP:

20.61.154.190
Enter fullscreen mode Exit fullscreen mode

I inspected its configuration and endpoints:

kubectl describe service web-public -n k8s-lab
Enter fullscreen mode Exit fullscreen mode

The Service used the selector app=web, exposed port 80, and routed traffic to the NGINX Pods. Its type was LoadBalancer, and the external IP had been assigned.

The lab no longer required this public endpoint for its remaining work, so I removed the Service:

kubectl delete service web-public -n k8s-lab
Enter fullscreen mode Exit fullscreen mode

Kubernetes confirmed:

service "web-public" deleted from k8s-lab namespace
Enter fullscreen mode Exit fullscreen mode

service web-public

Removing the Service eliminated that Kubernetes-managed public exposure. The internal Service remained available for cluster-internal access.

This action also removed the LoadBalancer Service configuration that was no longer needed by the lab. Deleting the Service should not be treated as proof that every associated Azure charge disappears immediately; the resulting Azure resources and billing records still need to be considered when validating actual savings.

6. Configure Horizontal Pod Autoscaling

The web Deployment had been configured with three replicas. Rather than permanently fixing it at that count, I created a Horizontal Pod Autoscaler (HPA) to allow Kubernetes to adjust the number of Pods based on CPU utilization.

I configured the HPA as follows:

kubectl autoscale deployment web -n k8s-lab \
  --cpu-percent=70 \
  --min=1 \
  --max=5
Enter fullscreen mode Exit fullscreen mode

Kubernetes returned:

horizontalpodautoscaler.autoscaling/web autoscaled
Enter fullscreen mode Exit fullscreen mode

The command displayed a deprecation notice for --cpu-percent, recommending the newer --cpu option with a percentage or resource quantity format.

I verified the resulting HPA:

kubectl get hpa web -n k8s-lab
Enter fullscreen mode Exit fullscreen mode

The output showed:

NAME   REFERENCE      TARGETS   MINPODS   MAXPODS   REPLICAS
web    Deployment/web cpu: 2%/70%  1      5         3
Enter fullscreen mode Exit fullscreen mode

autoscaling

At the time of the check, the observed CPU utilization was approximately 2% against a 70% target. The HPA was configured to maintain between one and five replicas, with three replicas currently reported.

This configuration allows the replica count to respond to CPU utilization rather than requiring manual scaling for every workload fluctuation.

The HPA does not guarantee that replicas immediately scale down whenever usage is low, nor does it remove the need for appropriate resource requests and metrics. It evaluates workload metrics against its configuration and adjusts the desired replica count as conditions warrant.

7. Enable AKS Node-Pool Autoscaling

Pod autoscaling changes the number of application replicas, but it does not automatically increase the number of virtual machines in the cluster unless cluster-level scaling is configured.

I enabled the cluster autoscaler for nodepool1, allowing the node count to vary between one and three:

az aks nodepool update \
  --resource-group "$RESOURCE_GROUP" \
  --cluster-name "$CLUSTER_NAME" \
  --name nodepool1 \
  --enable-cluster-autoscaler \
  --min-count 1 \
  --max-count 3
Enter fullscreen mode Exit fullscreen mode

nodepool update

The command updated the node-pool configuration. Azure CLI also displayed a notice that the command behavior had been altered by the aks-preview extension.

I then queried the effective settings:

az aks nodepool show \
  --resource-group "$RESOURCE_GROUP" \
  --cluster-name "$CLUSTER_NAME" \
  --name nodepool1 \
  --query "{Pool:name,Count:count,Autoscaling:enableAutoScaling,Min:minCount,Max:maxCount}" \
  --output table
Enter fullscreen mode Exit fullscreen mode

nodepool show

The output showed:

Pool       Count   Autoscaling   Min   Max
nodepool1  1       True          1     3
Enter fullscreen mode Exit fullscreen mode

This verified that cluster autoscaling was enabled and that the configured minimum and maximum node counts were one and three.

The node pool was currently at one node. With autoscaling enabled, Kubernetes can request additional node capacity when scheduling demand requires it, subject to the configured limits and Azure's ability to provision the capacity.

HPA and cluster autoscaling serve different purposes

These settings control separate layers of the environment:

  • Horizontal Pod Autoscaler: adjusts the number of application Pods.
  • Cluster autoscaler: adjusts the number of AKS nodes when cluster capacity needs to change.

The HPA was configured for one to five web replicas, while the node pool was configured for one to three nodes. The two mechanisms work at different levels and should be configured together with appropriate resource requests, scheduling requirements, and capacity limits.

The maximum node count is a limit, not a target. Enabling autoscaling does not mean the cluster will remain at three nodes.

8. Test the AKS Stop/Start Lifecycle

I also exercised the AKS cluster's stop/start commands.

To stop the cluster, I ran:

az aks stop \
  --resource-group "$RESOURCE_GROUP" \
  --name "$CLUSTER_NAME"
Enter fullscreen mode Exit fullscreen mode

I then started it again:

az aks start \
  --resource-group "$RESOURCE_GROUP" \
  --name "$CLUSTER_NAME"
Enter fullscreen mode Exit fullscreen mode

aks stop and start

Finally, I checked its reported power state:

az aks show \
  --resource-group "$RESOURCE_GROUP" \
  --name "$CLUSTER_NAME" \
  --query powerState.code \
  --output tsv
Enter fullscreen mode Exit fullscreen mode

The resulting state was:

Running
Enter fullscreen mode Exit fullscreen mode

az aks show

This confirmed that the cluster was running after the stop/start sequence.

For a lab or development environment that is not needed continuously, stopping the cluster during idle periods can be part of a cost-control strategy. However, the cluster must first be checked for active workloads, scheduled operations, dependencies, and service availability requirements.

The stop/start exercise demonstrates the control operation; it is not, on its own, a measured comparison of the before-and-after Azure bill. Actual savings should be verified against the applicable billing records and the resources that continue to incur charges.

9. Document the Findings in COST-REPORT.md

To capture the analysis and decisions, I created COST-REPORT.md in the project repository:

cat << 'EOF' > COST-REPORT.md
# AKS FinOps Cost Report

## Cluster Configuration
- Resource Group: rg-aks-lab
- Cluster: aks-lab
- Region: westeurope
- Node Pool: nodepool1
- VM Size: Standard_D2s_v3
- Node Count: 1

## Resource Utilization
- Node CPU usage: 12%
- Node memory usage: 25%
- CPU requests: 62%
- Memory requests: 22%

## Cost Estimate
- Estimated VM rate: $0.12/hour
- Estimated monthly VM cost (730 hours): $87.60
- Estimate excludes additional AKS and Azure resource charges.

## Cost Optimization Actions
- Configured web deployment autoscaling between 1 and 5 replicas.
- Enabled cluster autoscaling for nodepool1 between 1 and 3 nodes.
- Reduced the node pool to 1 node.
- Removed the web-public LoadBalancer service.
- Stopped the AKS cluster when not in use.

## Notes
The VM price is a public retail-price estimate, not a confirmed invoice amount. Actual costs depend on billing rates and associated Azure resources.
EOF
Enter fullscreen mode Exit fullscreen mode

cost report

The report consolidated the cluster configuration, observed utilization, VM price estimate, and optimization actions into a version-controlled artifact.

A concise cost report is useful because it distinguishes the measurements from the actions taken and documents what remains outside the estimate. In this case, it explicitly records that the $87.60 figure excludes additional AKS and Azure resource charges.

10. Commit the Cost Report

I checked the repository state:

git status
Enter fullscreen mode Exit fullscreen mode

git status

The report appeared as an untracked file. I staged it:

git add COST-REPORT.md
Enter fullscreen mode Exit fullscreen mode

Then I committed it:

git commit -m "docs: AKS cost review and optimisation"
Enter fullscreen mode Exit fullscreen mode

git status and commit

Git confirmed the commit:

[main 2725349] docs: AKS cost review and optimisation
1 file changed, 30 insertions(+)
create mode 100644 COST-REPORT.md
Enter fullscreen mode Exit fullscreen mode

This committed the cost report to the main branch, preserving the analysis and optimization decisions alongside the lab's other project documentation.

11. Summary of Cost Optimization Decisions

The main findings and actions were:

Area Finding or action Operational purpose
Node utilization 12% CPU and 25% memory usage observed Establish a snapshot of actual resource consumption
Resource requests CPU requests at 62%; memory requests at 22% Identify declared capacity reservations for further review
Resource limits Aggregated CPU and memory limits above 100% Understand configured overcommitment rather than actual usage
VM pricing $0.12/hour used for the estimate Establish a public retail-price baseline
Monthly estimate $87.60 for 730 hours of VM compute Quantify an illustrative compute cost
Workload scaling HPA configured for one to five replicas Match the web replica count to CPU demand
Node scaling Cluster autoscaler configured for one to three nodes Allow node capacity to adjust to scheduling demand
Public networking web-public Service deleted Remove an external endpoint no longer required by the lab
Cluster lifecycle Stop/start operations tested Establish an option for managing idle lab capacity
Documentation COST-REPORT.md committed Preserve the findings and optimization actions

These actions address different aspects of cost management. Autoscaling controls capacity, removing an unnecessary Service eliminates unneeded exposure, resource measurements inform future tuning, and stopping an idle cluster provides a lifecycle control. The retail-price estimate supports planning but does not replace actual cost analysis.

Conclusion

This FinOps review combined Kubernetes telemetry with Azure pricing data to establish a practical baseline for the AKS lab.

The observed node utilization was 12% CPU and 25% memory, while resource requests and limits revealed how declared capacity differed from current consumption. A public VM consumption rate of $0.12 per hour was used to estimate $87.60 for 730 hours of VM compute, excluding additional AKS and Azure resource charges.

I configured the web HPA to scale between one and five replicas, enabled node-pool autoscaling between one and three nodes, removed the public web-public Service, tested the AKS stop/start workflow, and committed the findings in COST-REPORT.md.

The key principle is that cost optimization should be based on measured usage, explicit assumptions, and deliberate trade-offs. Reducing infrastructure cost is valuable only when the resulting capacity and availability remain appropriate for the workload.

In Part 9, the series will conclude with the final mission, bringing the lab's implementation, documentation, and cleanup together.

This article is Part 8 of the nine-part series, “Kubernetes on AKS: From Zero to Production Workloads.”

Top comments (1)

Collapse
 
rahimah_dev profile image
Rahimah Sulayman •

Cloud cost optimization is not simply about spending less, it is about understanding what resources are actually being used, identifying waste, and making informed trade-offs without compromising reliability. This mission connects Kubernetes metrics, Azure retail pricing, workload autoscaling, and node-pool scaling to a practical FinOps approach. Documenting the findings in COST-REPORT.md also makes the decisions transparent, measurable, and easier to review. Efficient infrastructure starts with visibility.