DEV Community

Desmond Goldsmith
Desmond Goldsmith

Posted on

Building a Production-Style CI/CD Pipeline with Azure DevOps, ACR, Helm, AKS and PostgreSQL

#cicd #azure #aks #devops

Introduction

After learning Docker, Kubernetes, Helm, Azure and Azure DevOps separately, I wanted to put everything together into one practical project.

The goal was not to build a complicated application.

The goal was to understand what happens when application code moves from source control into a real deployment environment.

The final flow I wanted to understand was:

Developer
    |
    v
GitHub Repository
    |
    v
Azure DevOps
    |
    v
CI/CD Pipeline
    |
    +----------------------+
    |                      |
    v                      v
Build + Test          Docker Build
                           |
                           v
                  Azure Container Registry
                           |
                           v
                         Helm
                           |
                           v
                          AKS
                           |
                           v
                         DEV
                           |
                           v
                    Running Application
Enter fullscreen mode Exit fullscreen mode

The application itself was intentionally simple:

  • Frontend
  • Backend
  • PostgreSQL database

The infrastructure around the application was where most of the learning happened.

I wanted to understand:

  • Azure Resource Groups
  • Azure Container Registry
  • Azure Kubernetes Service
  • Azure networking
  • Private networking
  • PostgreSQL Flexible Server
  • Private DNS
  • Docker
  • Kubernetes namespaces
  • Deployments
  • Services
  • ConfigMaps
  • Secrets
  • Helm
  • Azure DevOps
  • Azure service connections
  • Workload Identity Federation
  • Azure RBAC
  • Kubernetes RBAC
  • CI/CD
  • Troubleshooting

For this project I focused on DEV first.

UAT, PROD and approval gates will be added later.

1. The Final Architecture

The DEV environment ended up looking roughly like this:

                         GitHub
                           |
                           v
                    Azure DevOps
                           |
                           v
                  Build + Test + Docker
                           |
                           v
                   Azure Container
                      Registry
                           |
                           v
                         Helm
                           |
                           v
                  Azure Kubernetes
                       Service
                           |
              +------------+------------+
              |                         |
              v                         v
        frontend namespace        backend namespace
              |                         |
              v                         v
       Frontend Deployment       Backend Deployment
              |                         |
              v                         v
       LoadBalancer Service       ClusterIP Service
              |                         |
              v                         |
        Public IP / Browser          |
                                      v
                            PostgreSQL Flexible
                                 Server
                                      |
                                      v
                              Private Network
                                      |
                                      v
                              Private DNS Zone
Enter fullscreen mode Exit fullscreen mode

The important design decision was that PostgreSQL was not installed inside AKS.

Instead, I used Azure Database for PostgreSQL Flexible Server.

That means AKS is responsible for running application workloads while Azure manages the database service.

2. The Project

The project repository is:

k8app-azure-devops
Enter fullscreen mode Exit fullscreen mode

Repository:

https://github.com/Desmondgoldsmith/k8app-azure-devops
Enter fullscreen mode Exit fullscreen mode

The repository contains the frontend, backend and Kubernetes/Helm configuration.

A simplified structure is:

k8app-azure-devops/
│
├── backend/
│   ├── app/
│   └── backend.dockerfile
│
├── frontend/
│   └── frontend.dockerfile
│
├── helm/
│   └── k8app/
│       ├── Chart.yaml
│       ├── values.yaml
│       └── templates/
│           ├── frontend-deployment.yaml
│           ├── frontend-service.yaml
│           ├── backend-deployment.yaml
│           ├── backend-service.yaml
│           ├── backend-configmap.yaml
│           └── backend-secret.yaml
│
└── azure-pipelines.yml
Enter fullscreen mode Exit fullscreen mode

The exact application is less important than the infrastructure around it.

3. Azure Subscription and Region

I used an Azure for Students subscription.

The region I used was:

South Africa North
Enter fullscreen mode Exit fullscreen mode

I chose the region partly because my student subscription had restrictions on which Azure regions I could use.

For example, some regions were unavailable under the subscription's policy.

Before creating resources, it is worth checking which regions your subscription allows.

You can do that from the Azure Portal when creating a resource, or with:

az account list-locations -o table
Enter fullscreen mode Exit fullscreen mode

4. Create the Azure Resource Group

The first Azure resource I created was the Resource Group.

I used:

Resource Group:
devops-aks-lab-rg-sa

Region:
South Africa North
Enter fullscreen mode Exit fullscreen mode

Why create a Resource Group?

A Resource Group is a logical container for related Azure resources.

Instead of having the project's resources scattered across Azure, I could put the main resources into one place:

devops-aks-lab-rg-sa
    |
    +-- ACR
    +-- AKS
    +-- PostgreSQL
    +-- Networking resources
Enter fullscreen mode Exit fullscreen mode

This also makes permissions, management and cleanup easier.

Azure Portal

Go to:

Azure Portal
    ↓
Resource groups
    ↓
Create
Enter fullscreen mode Exit fullscreen mode

Then select:

Subscription:
Azure for Students

Resource group:
devops-aks-lab-rg-sa

Region:
South Africa North
Enter fullscreen mode Exit fullscreen mode

Click:

Review + create
    ↓
Create
Enter fullscreen mode Exit fullscreen mode

Azure CLI

I could also create it using:

az login
Enter fullscreen mode Exit fullscreen mode

Then:

az group create \
  --name devops-aks-lab-rg-sa \
  --location southafricanorth
Enter fullscreen mode Exit fullscreen mode

Verify:

az group show \
  --name devops-aks-lab-rg-sa \
  -o table
Enter fullscreen mode Exit fullscreen mode

5. Create Azure Container Registry

The next component was Azure Container Registry.

I created:

ACR name:
dessydevopsacr
Enter fullscreen mode Exit fullscreen mode

The registry login server is:

dessydevopsacr.azurecr.io
Enter fullscreen mode Exit fullscreen mode

Why do we need ACR?

The CI pipeline builds Docker images.

Those images need somewhere to live before Kubernetes can deploy them.

So ACR becomes the central image repository:

Azure DevOps
     |
     | Docker image
     v
Azure Container Registry
     |
     | Docker image
     v
AKS
Enter fullscreen mode Exit fullscreen mode

For this project I had two images:

dessydevopsacr.azurecr.io/k8app-backend
dessydevopsacr.azurecr.io/k8app-frontend
Enter fullscreen mode Exit fullscreen mode

Azure Portal

Go to:

Azure Portal
    ↓
Create a resource
    ↓
Search:
Container Registry
    ↓
Create
Enter fullscreen mode Exit fullscreen mode

Under Basics, select:

Subscription:
Azure for Students

Resource group:
devops-aks-lab-rg-sa

Registry name:
dessydevopsacr

Location:
South Africa North

SKU:
Basic
Enter fullscreen mode Exit fullscreen mode

For this project I kept the ACR admin account disabled.

Then:

Review + create
    ↓
Create
Enter fullscreen mode Exit fullscreen mode

Azure CLI

Equivalent CLI:

az acr create \
  --resource-group devops-aks-lab-rg-sa \
  --name dessydevopsacr \
  --location southafricanorth \
  --sku Basic \
  --admin-enabled false
Enter fullscreen mode Exit fullscreen mode

Verify:

az acr show \
  --name dessydevopsacr \
  --resource-group devops-aks-lab-rg-sa \
  --query "{name:name,loginServer:loginServer,sku:sku.name}" \
  -o table
Enter fullscreen mode Exit fullscreen mode

Expected login server:

dessydevopsacr.azurecr.io
Enter fullscreen mode Exit fullscreen mode

Why disable the ACR admin account?

ACR can provide a static username and password through its admin account.

I did not want the CI/CD pipeline to depend on that type of long-lived credential.

Instead, Azure DevOps uses a service connection.

That gives us a cleaner authentication model:

Azure DevOps
     |
     v
Service Connection
     |
     v
Azure / ACR
Enter fullscreen mode Exit fullscreen mode

6. Create the Virtual Network

Because PostgreSQL was going to use private networking, networking needed to be considered before creating the database.

The basic idea was to have separate network areas for different workloads.

Conceptually:

Virtual Network
|
+-- AKS subnet
|
+-- PostgreSQL subnet
Enter fullscreen mode Exit fullscreen mode

The PostgreSQL subnet we used was:

postgres-subnet
10.0.2.0/24
Enter fullscreen mode Exit fullscreen mode

A /24 gives a defined block of private addresses for the database subnet.

The important reason for separating the database into its own subnet is that the database has a different purpose and security boundary from the application workloads.

Important note about the IP plan

The project definitely used:

postgres-subnet:
10.0.2.0/24
Enter fullscreen mode Exit fullscreen mode

The AKS cluster also used:

Pod CIDR:
10.244.0.0/16

Service CIDR:
10.240.0.0/16
Enter fullscreen mode Exit fullscreen mode

These are not the same thing as the Azure VNet subnet.

The exact Azure VNet/AKS subnet CIDR is not preserved in my project notes, so I am intentionally not inventing it here.

When reproducing the project, the network should be designed so the Azure VNet contains non-overlapping subnet ranges, for example:

VNet
|
+-- AKS subnet
|
+-- PostgreSQL subnet
       |
       +-- 10.0.2.0/24
Enter fullscreen mode Exit fullscreen mode

The important rule is:

Do not create overlapping CIDR ranges.

For example, your VNet cannot have two different subnets that both contain the same addresses.

7. Create the PostgreSQL Subnet

In the Azure Portal, I created the subnet from the Virtual Network.

Go to:

Azure Portal
    ↓
Virtual networks
    ↓
Select your project VNet
    ↓
Subnets
    ↓
+ Subnet
Enter fullscreen mode Exit fullscreen mode

Create:

Subnet name:
postgres-subnet

Subnet address range:
10.0.2.0/24
Enter fullscreen mode Exit fullscreen mode

Save the subnet.

The purpose is to reserve a dedicated private network range for PostgreSQL.

8. Why PostgreSQL Was Private

I deliberately did not expose PostgreSQL to the public internet.

Instead:

AKS
 |
 | Private network
 v
PostgreSQL
Enter fullscreen mode Exit fullscreen mode

rather than:

Internet
   |
   v
Public PostgreSQL endpoint
Enter fullscreen mode Exit fullscreen mode

This is closer to how a production architecture would normally be designed.

The application workloads communicate with the database through private networking.

9. Create the Private DNS Zone

This was one of the concepts that took some time to understand.

Creating a private IP for PostgreSQL is not enough.

Applications normally connect to PostgreSQL using a hostname such as:

dessy-k8app-postgres.postgres.database.azure.com
Enter fullscreen mode Exit fullscreen mode

The hostname needs to resolve to the private IP address.

That is where Private DNS comes in.

I created:

private.postgres.database.azure.com
Enter fullscreen mode Exit fullscreen mode

What does Private DNS do?

Think of DNS as a phonebook.

The application knows:

dessy-k8app-postgres.postgres.database.azure.com
Enter fullscreen mode Exit fullscreen mode

but the network ultimately needs an IP address.

Private DNS provides the mapping:

Hostname
    |
    v
Private DNS
    |
    v
Private IP
Enter fullscreen mode Exit fullscreen mode

In this project, the PostgreSQL private IP resolved to:

10.0.2.4
Enter fullscreen mode Exit fullscreen mode

So the flow became:

AKS Pod
   |
   | asks DNS:
   | "Where is
   | dessy-k8app-postgres.postgres.database.azure.com?"
   |
   v
Private DNS Zone
   |
   v
10.0.2.4
   |
   v
PostgreSQL
Enter fullscreen mode Exit fullscreen mode

Without DNS resolution, an application could know the hostname but would not know where to send the network traffic.

10. Create the Private DNS Zone in Azure Portal

Go to:

Azure Portal
    ↓
Create a resource
    ↓
Search:
Private DNS zones
    ↓
Create
Enter fullscreen mode Exit fullscreen mode

Set:

Resource group:
devops-aks-lab-rg-sa

Name:
private.postgres.database.azure.com
Enter fullscreen mode Exit fullscreen mode

Click:

Review + create
    ↓
Create
Enter fullscreen mode Exit fullscreen mode

After creating it, the DNS zone needs to be linked to the VNet.

Go to:

Private DNS zone
    ↓
Virtual network links
    ↓
+ Add
Enter fullscreen mode Exit fullscreen mode

Select the VNet used by the application/database.

This allows resources in the VNet to use that private DNS zone.

11. Create Azure Database for PostgreSQL Flexible Server

I created:

PostgreSQL server:
dessy-k8app-postgres

Region:
South Africa North

PostgreSQL:
16

SKU:
Standard_B1ms

Database:
app
Enter fullscreen mode Exit fullscreen mode

The hostname is:

dessy-k8app-postgres.postgres.database.azure.com
Enter fullscreen mode Exit fullscreen mode

The server uses private networking.

Why use PostgreSQL Flexible Server?

Instead of running PostgreSQL inside Kubernetes:

AKS
 |
 +-- PostgreSQL Pod
Enter fullscreen mode Exit fullscreen mode

I used:

AKS
 |
 | private network
 v
Azure PostgreSQL Flexible Server
Enter fullscreen mode Exit fullscreen mode

This separates application compute from database infrastructure.

AKS can concentrate on running:

frontend
backend
Enter fullscreen mode Exit fullscreen mode

while Azure manages the PostgreSQL service.

12. Create PostgreSQL in the Azure Portal

Go to:

Azure Portal
    ↓
Create a resource
    ↓
Search:
Azure Database for PostgreSQL Flexible Server
    ↓
Create
Enter fullscreen mode Exit fullscreen mode

Under Basics, configure:

Subscription:
Azure for Students

Resource group:
devops-aks-lab-rg-sa

Server name:
dessy-k8app-postgres

Region:
South Africa North

PostgreSQL version:
16

Workload type:
Development / appropriate low-cost option

Compute:
Standard_B1ms
Enter fullscreen mode Exit fullscreen mode

Configure the administrator:

Administrator username:
k8appadmin

Administrator password:
<your password>
Enter fullscreen mode Exit fullscreen mode

Do not publish the password.

13. Configure PostgreSQL Networking

During PostgreSQL creation, select private access rather than public access.

The database needs to use the VNet and the dedicated subnet:

Virtual network:
<project VNet>

Subnet:
postgres-subnet
Enter fullscreen mode Exit fullscreen mode

The private DNS configuration should use:

private.postgres.database.azure.com
Enter fullscreen mode Exit fullscreen mode

The important relationship is:

PostgreSQL
    |
    v
postgres-subnet
10.0.2.0/24
    |
    v
Private DNS
private.postgres.database.azure.com
Enter fullscreen mode Exit fullscreen mode

Azure then creates the private networking relationship.

14. Create the Application Database

After creating the server, I created the database:

app
Enter fullscreen mode Exit fullscreen mode

From the PostgreSQL resource, go to:

PostgreSQL Flexible Server
    ↓
Databases
    ↓
+ Add
Enter fullscreen mode Exit fullscreen mode

Create:

Database name:
app
Enter fullscreen mode Exit fullscreen mode

The application will later connect using:

Server:
dessy-k8app-postgres.postgres.database.azure.com

Database:
app

User:
k8appadmin
Enter fullscreen mode Exit fullscreen mode

The password is kept secret.

15. Verify PostgreSQL Networking From AKS

Creating the database is not enough.

I wanted to answer a more important question:

Can a workload running inside AKS actually reach PostgreSQL?

I tested this from inside Kubernetes.

First I checked DNS resolution.

kubectl run psql-test \
  --rm -it \
  --image=postgres:16 \
  --restart=Never \
  -- \
  getent hosts dessy-k8app-postgres.postgres.database.azure.com
Enter fullscreen mode Exit fullscreen mode

The hostname resolved to the private PostgreSQL IP.

In our case:

10.0.2.4
Enter fullscreen mode Exit fullscreen mode

This proved that the private DNS path was working.

16. Test the PostgreSQL Connection

I then used a temporary PostgreSQL client pod.

kubectl run psql-test \
  --rm -it \
  --image=postgres:16 \
  --restart=Never \
  -- \
  psql "host=dessy-k8app-postgres.postgres.database.azure.com port=5432 dbname=app user=k8appadmin sslmode=require"
Enter fullscreen mode Exit fullscreen mode

When prompted, I entered the database password.

The connection succeeded over TLS.

That proved this path was working:

AKS Pod
   |
   v
Private DNS
   |
   v
Private IP
   |
   v
PostgreSQL Flexible Server
Enter fullscreen mode Exit fullscreen mode

This was an important test because it separated database/networking problems from application problems.

17. Create the AKS Cluster

The next major component was Azure Kubernetes Service.

I created:

AKS:
dessy-aks-cluster

Resource Group:
devops-aks-lab-rg-sa

Region:
South Africa North
Enter fullscreen mode Exit fullscreen mode

The cluster used:

Azure CNI Overlay

Pod CIDR:
10.244.0.0/16

Service CIDR:
10.240.0.0/16

Load Balancer:
Standard
Enter fullscreen mode Exit fullscreen mode

I also enabled Microsoft Entra integration and Kubernetes RBAC.

18. Why Azure CNI Overlay?

AKS networking can be configured in different ways.

For this project I used Azure CNI Overlay.

One important distinction is that these CIDRs:

10.244.0.0/16
10.240.0.0/16
Enter fullscreen mode Exit fullscreen mode

are Kubernetes networking ranges.

They are different from the Azure VNet subnet ranges.

I found this distinction important when learning AKS networking because it is easy to look at an IP such as:

10.244.x.x
Enter fullscreen mode Exit fullscreen mode

and assume it is an Azure VNet IP.

It is not.

The simplified model is:

Azure VNet
    |
    +-- Azure subnet
          |
          +-- AKS nodes
                |
                +-- Kubernetes pod networking
                |
                +-- Kubernetes service networking
Enter fullscreen mode Exit fullscreen mode

19. Create AKS in Azure Portal

Go to:

Azure Portal
    ↓
Create a resource
    ↓
Search:
Kubernetes Service
    ↓
Create
Enter fullscreen mode Exit fullscreen mode

Under Basics, configure:

Subscription:
Azure for Students

Resource group:
devops-aks-lab-rg-sa

Cluster preset:
appropriate development option

Kubernetes cluster name:
dessy-aks-cluster

Region:
South Africa North
Enter fullscreen mode Exit fullscreen mode

Configure the node pool with one node for this learning environment.

20. Configure AKS Identity

Under the identity/security configuration, use:

Managed identity
Enter fullscreen mode Exit fullscreen mode

rather than manually managing long-lived credentials.

This allows Azure resources to work with managed identities.

21. Configure Microsoft Entra ID and Kubernetes RBAC

I enabled Microsoft Entra integration.

The cluster uses Microsoft Entra ID for identity and Kubernetes RBAC for authorization.

This became an important lesson later.

Authentication and authorization are not the same thing.

Think about them as:

Microsoft Entra ID
       |
       | Who are you?
       v
Authentication
       |
       v
Kubernetes RBAC
       |
       | What are you allowed to do?
       v
Authorization
Enter fullscreen mode Exit fullscreen mode

22. Configure AKS Networking

For the cluster networking, I used:

Network plugin:
Azure CNI

Network plugin mode:
Overlay

Pod CIDR:
10.244.0.0/16

Service CIDR:
10.240.0.0/16
Enter fullscreen mode Exit fullscreen mode

I also used a Standard Load Balancer.

This is important because the frontend Kubernetes Service was later configured as:

type: LoadBalancer
Enter fullscreen mode Exit fullscreen mode

Azure then provisions an Azure Load Balancer/public IP for that service.

23. Enable OIDC and Workload Identity

I enabled:

OIDC issuer
Workload identity
Enter fullscreen mode Exit fullscreen mode

These capabilities are useful for modern identity-based authentication between Kubernetes workloads and Azure services.

They also make it possible to avoid embedding long-lived Azure credentials inside applications.

24. Connect kubectl to AKS

After the cluster was created, I retrieved its credentials:

az aks get-credentials \
  --resource-group devops-aks-lab-rg-sa \
  --name dessy-aks-cluster \
  --overwrite-existing
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

I expected the node to show:

Ready
Enter fullscreen mode Exit fullscreen mode

This confirmed that my local machine could communicate with the cluster.

25. AKS Identity and ACR

There are several identities involved in this project.

This was one of the most important things I learned.

The AKS cluster has a kubelet managed identity.

I checked it with:

az aks show \
  --resource-group devops-aks-lab-rg-sa \
  --name dessy-aks-cluster \
  --query identityProfile.kubeletidentity.clientId \
  -o tsv
Enter fullscreen mode Exit fullscreen mode

The kubelet identity needs permission to pull images from ACR.

The permission required is:

AcrPull
Enter fullscreen mode Exit fullscreen mode

The flow is:

AKS Kubelet Identity
        |
        | AcrPull
        v
Azure Container Registry
        |
        v
Docker Image
Enter fullscreen mode Exit fullscreen mode

26. Give AKS AcrPull Permission

In Azure Portal, go to:

Container Registry
    ↓
dessydevopsacr
    ↓
Access control (IAM)
    ↓
Role assignments
Enter fullscreen mode Exit fullscreen mode

Add a role assignment.

Select:

Role:
AcrPull
Enter fullscreen mode Exit fullscreen mode

Assign it to the AKS kubelet managed identity.

After this:

AKS
 |
 | AcrPull
 v
ACR
 |
 v
k8app-backend image
k8app-frontend image
Enter fullscreen mode Exit fullscreen mode

The important distinction is:

Azure DevOps pushes the image. AKS pulls the image.

These are two different operations and can use two different identities.

27. Understanding the Three Identity Layers

This project involved three different concepts.

OIDC / Workload Identity

This answers:

Who are you?
Enter fullscreen mode Exit fullscreen mode

For Azure DevOps:

Azure DevOps
      |
      v
OIDC / Workload Identity Federation
      |
      v
Microsoft Entra ID
Enter fullscreen mode Exit fullscreen mode

Azure RBAC

This answers:

What can you do in Azure?
Enter fullscreen mode Exit fullscreen mode

For example:

Azure DevOps identity
       |
       v
Contributor
       |
       v
Resource Group
Enter fullscreen mode Exit fullscreen mode

Kubernetes RBAC

This answers:

What can you do inside Kubernetes?
Enter fullscreen mode Exit fullscreen mode

For example:

Azure DevOps identity
       |
       v
Kubernetes RoleBinding
       |
       v
edit
       |
       +-- frontend namespace
       |
       +-- backend namespace
Enter fullscreen mode Exit fullscreen mode

The simplified mental model is:

OIDC
 ↓
Who are you?

Azure RBAC
 ↓
What can you do in Azure?

Kubernetes RBAC
 ↓
What can you do in Kubernetes?
Enter fullscreen mode Exit fullscreen mode

28. Create the Azure DevOps Project

I created an Azure DevOps project:

DessyTest-1
Enter fullscreen mode Exit fullscreen mode

Go to:

Azure DevOps
    ↓
New Project
Enter fullscreen mode Exit fullscreen mode

Create:

Project name:
DessyTest-1
Enter fullscreen mode Exit fullscreen mode

The GitHub repository remains the source repository.

Azure DevOps is being used for the CI/CD pipeline.

The conceptual relationship is:

GitHub
   |
   v
Azure DevOps
   |
   v
Pipeline
Enter fullscreen mode Exit fullscreen mode

29. Connect GitHub to Azure DevOps

Inside Azure DevOps, create/configure the GitHub connection.

Go to:

Azure DevOps
    ↓
Project Settings
    ↓
Service connections
Enter fullscreen mode Exit fullscreen mode

Create the GitHub connection and authorize the repository.

The purpose is to allow Azure DevOps to retrieve the repository source code.

30. Create the ACR Service Connection

Azure DevOps needs a way to authenticate to ACR.

Go to:

Azure DevOps
    ↓
Project Settings
    ↓
Service connections
    ↓
New service connection
Enter fullscreen mode Exit fullscreen mode

Select:

Docker Registry
Enter fullscreen mode Exit fullscreen mode

Then configure it for Azure Container Registry.

I created the service connection:

dessydevopsacr
Enter fullscreen mode Exit fullscreen mode

This connection is used by the Docker task in the pipeline.

Conceptually:

Azure DevOps
     |
     v
ACR Service Connection
     |
     v
Azure Container Registry
Enter fullscreen mode Exit fullscreen mode

31. Create the AKS Azure Service Connection

The deployment stage needs Azure access so it can obtain AKS credentials.

I created:

dessy-aks-deploy
Enter fullscreen mode Exit fullscreen mode

Go to:

Azure DevOps
    ↓
Project Settings
    ↓
Service connections
    ↓
New service connection
Enter fullscreen mode Exit fullscreen mode

Select the Azure Resource Manager connection type.

Use:

Workload Identity Federation
Enter fullscreen mode Exit fullscreen mode

rather than a long-lived client secret.

This gives:

Azure DevOps
      |
      v
OIDC / Workload Identity Federation
      |
      v
Microsoft Entra ID
      |
      v
Azure
Enter fullscreen mode Exit fullscreen mode

32. Give the Azure DevOps Identity Azure Permissions

The service principal used by the deployment service connection needs permission to work with the Azure resources.

For this learning environment, I assigned:

Contributor
Enter fullscreen mode Exit fullscreen mode

at the resource group scope:

devops-aks-lab-rg-sa
Enter fullscreen mode Exit fullscreen mode

In Azure Portal:

Resource Group
    ↓
devops-aks-lab-rg-sa
    ↓
Access control (IAM)
    ↓
Add role assignment
Enter fullscreen mode Exit fullscreen mode

Select:

Contributor
Enter fullscreen mode Exit fullscreen mode

Then select the service principal used by:

dessy-aks-deploy
Enter fullscreen mode Exit fullscreen mode

This was intentionally broad for the learning environment.

A production environment should use more restrictive permissions where practical.

33. Kubernetes RBAC for Azure DevOps

Getting Azure permissions was not enough.

At one point the pipeline could authenticate to Azure but Kubernetes still rejected some operations.

That taught me an important lesson:

Azure authentication and Kubernetes authorization are separate problems.

The Azure DevOps service principal was:

<AZURE_DEVOPS_SERVICE_PRINCIPAL_OBJECT_ID>
Enter fullscreen mode Exit fullscreen mode

I created a RoleBinding in the:

frontend
Enter fullscreen mode Exit fullscreen mode

namespace and another in:

backend
Enter fullscreen mode Exit fullscreen mode

The binding gives the identity the Kubernetes:

edit
Enter fullscreen mode Exit fullscreen mode

ClusterRole.

Example:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: azure-devops-deployer
  namespace: backend
subjects:
  - kind: User
    name: <AZURE_DEVOPS_SERVICE_PRINCIPAL_OBJECT_ID>
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: rbac.authorization.k8s.io
Enter fullscreen mode Exit fullscreen mode

The frontend binding is the same except for:

namespace: frontend
Enter fullscreen mode Exit fullscreen mode

34. Test Kubernetes RBAC

I tested whether the Azure DevOps identity could create deployments.

For backend:

kubectl auth can-i create deployments \
  --as=<AZURE_DEVOPS_SERVICE_PRINCIPAL_OBJECT_ID> \
  -n backend
Enter fullscreen mode Exit fullscreen mode

For frontend:

kubectl auth can-i create deployments \
  --as=<AZURE_DEVOPS_SERVICE_PRINCIPAL_OBJECT_ID> \
  -n frontend
Enter fullscreen mode Exit fullscreen mode

Both returned:

yes
Enter fullscreen mode Exit fullscreen mode

I deliberately did not give the pipeline:

cluster-admin
Enter fullscreen mode Exit fullscreen mode

because that would give it unnecessarily broad access to the whole cluster.

35. Create the Kubernetes Namespaces

I separated the frontend and backend workloads.

The namespaces were:

frontend
backend
Enter fullscreen mode Exit fullscreen mode

Create them:

kubectl create namespace frontend
kubectl create namespace backend
Enter fullscreen mode Exit fullscreen mode

Verify:

kubectl get namespaces
Enter fullscreen mode Exit fullscreen mode

The reason for separate namespaces is organization and isolation.

Instead of:

default
 |
 +-- frontend
 +-- backend
 +-- everything else
Enter fullscreen mode Exit fullscreen mode

we have:

frontend namespace
 |
 +-- frontend Deployment
 +-- frontend Service

backend namespace
 |
 +-- backend Deployment
 +-- backend Service
 +-- ConfigMap
 +-- Secret
Enter fullscreen mode Exit fullscreen mode

It also gives us a clean foundation for namespace-level RBAC.

36. Why Kubernetes Deployment?

A Deployment describes how Kubernetes should run an application.

For example:

kind: Deployment
Enter fullscreen mode Exit fullscreen mode

tells Kubernetes:

I want this application running.
Enter fullscreen mode Exit fullscreen mode

It controls things such as:

  • Number of replicas
  • Container image
  • Container ports
  • Pod labels
  • Updates

For example:

replicas: 1
Enter fullscreen mode Exit fullscreen mode

means we currently want one pod.

37. Why Kubernetes Service?

Pods are temporary.

Their IP addresses can change.

A Kubernetes Service gives applications a stable way to communicate with a group of Pods.

The model is:

Pod
 |
 | temporary IP
 v
Service
 |
 | stable name/IP
 v
Other application
Enter fullscreen mode Exit fullscreen mode

There are two different Services in this project.

Frontend:

LoadBalancer
Enter fullscreen mode Exit fullscreen mode

Backend:

ClusterIP
Enter fullscreen mode Exit fullscreen mode

38. Frontend Service

The frontend Service is:

apiVersion: v1

kind: Service
metadata:
  name: k8app-frontend
  namespace: frontend
spec:
  type: LoadBalancer
  selector:
    app: k8app-frontend
  ports:
    - port: 80
      targetPort: 80
Enter fullscreen mode Exit fullscreen mode

The important part is:

type: LoadBalancer
Enter fullscreen mode Exit fullscreen mode

This tells AKS that the service should be externally reachable.

The flow is:

Browser
   |
   v
Azure Load Balancer
   |
   v
k8app-frontend Service
   |
   v
Frontend Pod
Enter fullscreen mode Exit fullscreen mode

39. Backend Service

The backend Service is:

apiVersion: v1
kind: Service
metadata:
  name: k8app-backend
  namespace: backend
spec:
  type: ClusterIP
  selector:
    app: k8app-backend
  ports:
    - port: 80
      targetPort: 80
Enter fullscreen mode Exit fullscreen mode

The important part is:

type: ClusterIP
Enter fullscreen mode Exit fullscreen mode

ClusterIP means the service is intended to be reachable internally within Kubernetes.

Conceptually:

Frontend / internal caller
        |
        v
Backend ClusterIP
        |
        v
Backend Pod
Enter fullscreen mode Exit fullscreen mode

40. Create the Helm Chart

Instead of maintaining raw Kubernetes YAML for every environment, I used Helm.

The chart is:

helm/k8app
Enter fullscreen mode Exit fullscreen mode

Create the directory structure:

helm/
└── k8app/
    ├── Chart.yaml
    ├── values.yaml
    └── templates/
Enter fullscreen mode Exit fullscreen mode

41. Create Chart.yaml

The chart definition is:

apiVersion: v2
name: k8app
description: Helm chart for the K8App frontend and backend
type: application
version: 0.1.0
appVersion: "1.0.0"
Enter fullscreen mode Exit fullscreen mode

The important concept is that Helm packages our Kubernetes configuration into one deployable unit.

Instead of manually doing:

kubectl apply -f ...
kubectl apply -f ...
kubectl apply -f ...
Enter fullscreen mode Exit fullscreen mode

we can do:

helm upgrade --install ...
Enter fullscreen mode Exit fullscreen mode

42. Create values.yaml

The values file contains environment-specific values.

frontend:
  replicas: 1

  image:
    repository: dessydevopsacr.azurecr.io/k8app-frontend
    tag: ""

backend:
  replicas: 1

  image:
    repository: dessydevopsacr.io/k8app-backend
    tag: ""

  config:
    projectName: "k8app"
    postgresServer: "dessy-k8app-postgres.postgres.database.azure.com"
    postgresDb: "app"

  secret:
    postgresUser: "k8appadmin"
    postgresPassword: ""
Enter fullscreen mode Exit fullscreen mode

The correct backend registry is:

dessydevopsacr.azurecr.io
Enter fullscreen mode Exit fullscreen mode

so the backend repository should be:

repository: dessydevopsacr.azurecr.io/k8app-backend
Enter fullscreen mode Exit fullscreen mode

The final version is therefore:

frontend:
  replicas: 1

  image:
    repository: dessydevopsacr.azurecr.io/k8app-frontend
    tag: ""

backend:
  replicas: 1

  image:
    repository: dessydevopsacr.azurecr.io/k8app-backend
    tag: ""

  config:
    projectName: "k8app"
    postgresServer: "dessy-k8app-postgres.postgres.database.azure.com"
    postgresDb: "app"

  secret:
    postgresUser: "k8appadmin"
    postgresPassword: ""
Enter fullscreen mode Exit fullscreen mode

Notice that the password is blank.

That is intentional.

The pipeline supplies it securely during deployment.

43. Why Helm Values Exist

The values file lets us separate:

Application template
Enter fullscreen mode Exit fullscreen mode

from:

Environment values
Enter fullscreen mode Exit fullscreen mode

For example:

image:
  repository: ...
  tag: ""
Enter fullscreen mode Exit fullscreen mode

The chart doesn't need to know which pipeline build is being deployed.

The pipeline can provide:

--set backend.image.tag=$(Build.BuildId)
Enter fullscreen mode Exit fullscreen mode

and:

--set frontend.image.tag=$(Build.BuildId)
Enter fullscreen mode Exit fullscreen mode

So:

Pipeline Build ID
       |
       v
Docker image tag
       |
       v
Helm values
       |
       v
Kubernetes Deployment
Enter fullscreen mode Exit fullscreen mode

This creates a traceable relationship between a pipeline run and the deployed image.

44. Frontend Deployment

The frontend Deployment is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8app-frontend
  namespace: frontend
spec:
  replicas: {{ .Values.frontend.replicas }}
  selector:
    matchLabels:
      app: k8app-frontend
  template:
    metadata:
      labels:
        app: k8app-frontend
    spec:
      containers:
        - name: frontend
          image: "{{ .Values.frontend.image.repository }}:{{ .Values.frontend.image.tag }}"
          ports:
            - containerPort: 80
Enter fullscreen mode Exit fullscreen mode

The important Helm expression is:

{{ .Values.frontend.image.tag }}
Enter fullscreen mode Exit fullscreen mode

Helm replaces this with the actual value during deployment.

For example, if the pipeline passes:

--set frontend.image.tag=21
Enter fullscreen mode Exit fullscreen mode

the resulting Kubernetes image becomes:

dessydevopsacr.azurecr.io/k8app-frontend:21
Enter fullscreen mode Exit fullscreen mode

45. Backend Deployment

The backend Deployment contains the container and environment configuration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8app-backend
  namespace: backend
spec:
  replicas: {{ .Values.backend.replicas }}
  selector:
    matchLabels:
      app: k8app-backend
  template:
    metadata:
      labels:
        app: k8app-backend
    spec:
      containers:
        - name: backend
          image: "{{ .Values.backend.image.repository }}:{{ .Values.backend.image.tag }}"
          ports:
            - containerPort: 80
          env:
            - name: PROJECT_NAME
              valueFrom:
                configMapKeyRef:
                  name: k8app-backend-config
                  key: PROJECT_NAME

            - name: POSTGRES_SERVER
              valueFrom:
                configMapKeyRef:
                  name: k8app-backend-config
                  key: POSTGRES_SERVER

            - name: POSTGRES_DB
              valueFrom:
                configMapKeyRef:
                  name: k8app-backend-config
                  key: POSTGRES_DB

            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: k8app-backend-secret
                  key: POSTGRES_USER

            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: k8app-backend-secret
                  key: POSTGRES_PASSWORD
Enter fullscreen mode Exit fullscreen mode

This is where ConfigMaps and Secrets become important.

46. What Is a ConfigMap?

A ConfigMap stores non-sensitive configuration.

For this application, things such as:

PROJECT_NAME
POSTGRES_SERVER
POSTGRES_DB
Enter fullscreen mode Exit fullscreen mode

do not need to be treated as passwords.

So I created:

k8app-backend-config
Enter fullscreen mode Exit fullscreen mode

47. Create the ConfigMap

The Helm template is:

apiVersion: v1
kind: ConfigMap
metadata:
  name: k8app-backend-config
  namespace: backend
data:
  PROJECT_NAME: {{ .Values.backend.config.projectName | quote }}
  POSTGRES_SERVER: {{ .Values.backend.config.postgresServer | quote }}
  POSTGRES_DB: {{ .Values.backend.config.postgresDb | quote }}
Enter fullscreen mode Exit fullscreen mode

The resulting Kubernetes object looks conceptually like:

k8app-backend-config
|
+-- PROJECT_NAME=k8app
+-- POSTGRES_SERVER=dessy-k8app-postgres.postgres.database.azure.com
+-- POSTGRES_DB=app
Enter fullscreen mode Exit fullscreen mode

You can inspect it with:

kubectl get configmap k8app-backend-config -n backend
Enter fullscreen mode Exit fullscreen mode

Or:

kubectl describe configmap k8app-backend-config -n backend
Enter fullscreen mode Exit fullscreen mode

48. What Is a Kubernetes Secret?

A Secret is intended for sensitive configuration.

In our case:

POSTGRES_USER
POSTGRES_PASSWORD
Enter fullscreen mode Exit fullscreen mode

The password should not be stored directly in:

Deployment YAML
values.yaml
Git
Enter fullscreen mode Exit fullscreen mode

Instead, I created:

k8app-backend-secret
Enter fullscreen mode Exit fullscreen mode

49. Create the Kubernetes Secret

The Helm template is:

apiVersion: v1
kind: Secret
metadata:
  name: k8app-backend-secret
  namespace: backend
type: Opaque
stringData:
  POSTGRES_USER: {{ .Values.backend.secret.postgresUser | quote }}
  POSTGRES_PASSWORD: {{ .Values.backend.secret.postgresPassword | quote }}
Enter fullscreen mode Exit fullscreen mode

The password is supplied at deployment time.

I did not commit the actual password to Git.

50. How configMapKeyRef Works

This part initially looked confusing:

valueFrom:
  configMapKeyRef:
    name: k8app-backend-config
    key: POSTGRES_SERVER
Enter fullscreen mode Exit fullscreen mode

But it is actually simple.

It means:

Go to the ConfigMap named k8app-backend-config, find the key called POSTGRES_SERVER, and inject its value into the container as an environment variable named POSTGRES_SERVER.

So:

ConfigMap
k8app-backend-config
       |
       +-- POSTGRES_SERVER
                |
                v
        Backend container
                |
                v
        POSTGRES_SERVER
Enter fullscreen mode Exit fullscreen mode

The same thing happens for:

PROJECT_NAME
POSTGRES_DB
Enter fullscreen mode Exit fullscreen mode

51. How secretKeyRef Works

This:

valueFrom:
  secretKeyRef:
    name: k8app-backend-secret
    key: POSTGRES_PASSWORD
Enter fullscreen mode Exit fullscreen mode

means:

Go to the Secret named k8app-backend-secret, find the POSTGRES_PASSWORD key, and inject its value into the container as the POSTGRES_PASSWORD environment variable.

So:

Kubernetes Secret
       |
       +-- POSTGRES_PASSWORD
                |
                v
        Backend container
                |
                v
        POSTGRES_PASSWORD
Enter fullscreen mode Exit fullscreen mode

This keeps sensitive configuration separate from normal configuration.

52. Why the Backend Initially Failed

Before the ConfigMap and Secret were configured correctly, the backend container crashed.

The backend expected environment variables such as:

PROJECT_NAME
POSTGRES_SERVER
POSTGRES_USER
POSTGRES_PASSWORD
POSTGRES_DB
Enter fullscreen mode Exit fullscreen mode

The application's configuration code used environment-based configuration.

Without those variables, the application could not construct its database configuration.

This is an example of why logs are important.

Instead of guessing, I checked:

kubectl logs <backend-pod> -n backend
Enter fullscreen mode Exit fullscreen mode

The logs showed that required configuration values were missing.

That led to:

Backend Crash
     |
     v
Check Pod logs
     |
     v
Missing environment variables
     |
     v
Create ConfigMap + Secret
     |
     v
Reference them from Deployment
     |
     v
Backend starts
Enter fullscreen mode Exit fullscreen mode

53. Verify the Backend Configuration

Check the ConfigMap:

kubectl get configmap -n backend
Enter fullscreen mode Exit fullscreen mode

Check the Secret:

kubectl get secret -n backend
Enter fullscreen mode Exit fullscreen mode

Check the Deployment:

kubectl get deployment k8app-backend -n backend
Enter fullscreen mode Exit fullscreen mode

Check the Pod:

kubectl get pods -n backend
Enter fullscreen mode Exit fullscreen mode

If the pod is running:

1/1 Running
Enter fullscreen mode Exit fullscreen mode

then the application container has successfully started.

54. Verify the Backend Logs

Run:

kubectl logs deployment/k8app-backend -n backend
Enter fullscreen mode Exit fullscreen mode

If something fails, I normally investigate in this order:

kubectl get pods
        |
        v
kubectl describe pod
        |
        v
kubectl logs
        |
        v
Check ConfigMap / Secret
        |
        v
Check Service
        |
        v
Check database/network
Enter fullscreen mode Exit fullscreen mode

This is much better than randomly changing YAML.

55. Verify the Database From the Backend Namespace

Because the backend needs PostgreSQL, I verified that the backend environment contained the correct configuration.

The database host is:

dessy-k8app-postgres.postgres.database.azure.com
Enter fullscreen mode Exit fullscreen mode

The database is:

app
Enter fullscreen mode Exit fullscreen mode

The username is:

k8appadmin
Enter fullscreen mode Exit fullscreen mode

The password remains secret.

56. Validate the Helm Chart Before Deploying

One of the most useful Helm commands is:

helm template test-release ./helm/k8app
Enter fullscreen mode Exit fullscreen mode

This does not deploy anything.

It renders the Helm templates into Kubernetes YAML.

This allows us to catch template problems before sending anything to AKS.

For example:

helm template test-release ./helm/k8app
Enter fullscreen mode Exit fullscreen mode

If there is a problem such as:

nil pointer evaluating interface {}.create
Enter fullscreen mode Exit fullscreen mode

it usually means the template is trying to access a value that doesn't exist in values.yaml.

This is one reason I use:

helm template
Enter fullscreen mode Exit fullscreen mode

before:

helm upgrade
Enter fullscreen mode Exit fullscreen mode

57. Deploy With Helm

The main deployment command is:

helm upgrade --install k8app helm/k8app \
  --namespace backend \
  --set backend.image.tag=21 \
  --set frontend.image.tag=21 \
  --wait \
  --timeout 5m
Enter fullscreen mode Exit fullscreen mode

The number:

21
Enter fullscreen mode Exit fullscreen mode

was the image tag for that pipeline/build.

In the real pipeline, we don't manually type this.

Azure DevOps provides:

$(Build.BuildId)
Enter fullscreen mode Exit fullscreen mode

So the pipeline can automatically deploy the image produced by that build.

58. Why Helm Release Is in the Backend Namespace

One slightly unusual part of this design is that the Helm release is installed using:

--namespace backend
Enter fullscreen mode Exit fullscreen mode

while the chart itself contains resources in both:

frontend
backend
Enter fullscreen mode Exit fullscreen mode

Therefore:

helm list -n backend
Enter fullscreen mode Exit fullscreen mode

shows:

k8app
Enter fullscreen mode Exit fullscreen mode

while:

helm list -n frontend
Enter fullscreen mode Exit fullscreen mode

does not.

That does not mean the frontend wasn't deployed.

It means the Helm release itself is stored/tracked in the backend namespace while the chart creates resources in both namespaces.

59. Verify the Helm Release

Run:

helm list -n backend
Enter fullscreen mode Exit fullscreen mode

Expected:

NAME    NAMESPACE  REVISION  STATUS    CHART
k8app   backend    3         deployed  k8app-0.1.0
Enter fullscreen mode Exit fullscreen mode

You can also inspect:

helm status k8app -n backend
Enter fullscreen mode Exit fullscreen mode

60. Verify Kubernetes Deployments

Check frontend:

kubectl get deployments -n frontend
Enter fullscreen mode Exit fullscreen mode

Check backend:

kubectl get deployments -n backend
Enter fullscreen mode Exit fullscreen mode

Expected conceptually:

frontend
k8app-frontend

backend
k8app-backend
Enter fullscreen mode Exit fullscreen mode

61. Verify Pods

Run:

kubectl get pods -n frontend
Enter fullscreen mode Exit fullscreen mode

and:

kubectl get pods -n backend
Enter fullscreen mode Exit fullscreen mode

Healthy pods should show:

1/1 Running
Enter fullscreen mode Exit fullscreen mode

For example:

k8app-frontend-xxxxxxxxxx   1/1   Running
Enter fullscreen mode Exit fullscreen mode

and:

k8app-backend-xxxxxxxxxx    1/1   Running
Enter fullscreen mode Exit fullscreen mode

62. Verify Services

Frontend:

kubectl get service -n frontend
Enter fullscreen mode Exit fullscreen mode

Backend:

kubectl get service -n backend
Enter fullscreen mode Exit fullscreen mode

The frontend should be:

LoadBalancer
Enter fullscreen mode Exit fullscreen mode

while the backend should be:

ClusterIP
Enter fullscreen mode Exit fullscreen mode

63. The LoadBalancer Problem

Initially, the frontend Service showed:

EXTERNAL-IP   <pending>
Enter fullscreen mode Exit fullscreen mode

At first it was tempting to think Kubernetes was broken.

Instead, I used:

kubectl describe service k8app-frontend -n frontend
Enter fullscreen mode Exit fullscreen mode

The important event was:

PublicIPCountLimitReached
Enter fullscreen mode Exit fullscreen mode

The subscription had reached the regional public IP limit.

The message indicated that the subscription could not create more than three public IP addresses in that region.

This was a good example of why:

kubectl describe
Enter fullscreen mode Exit fullscreen mode

is useful.

kubectl get tells us what is happening.

kubectl describe often helps explain why.

64. Troubleshooting the Public IP Limit

I checked existing AKS resources.

One older AKS cluster was:

rest-go-aks
Enter fullscreen mode Exit fullscreen mode

I checked its state:

az aks show \
  --resource-group devops-aks-lab-rg-sa \
  --name rest-go-aks \
  --query "{name:name,powerState:powerState.code}" \
  -o table
Enter fullscreen mode Exit fullscreen mode

It showed:

Name         PowerState
------------  ----------
rest-go-aks  Stopped
Enter fullscreen mode Exit fullscreen mode

The old cluster was no longer needed for this project.

I removed it:

az aks delete \
  --resource-group devops-aks-lab-rg-sa \
  --name rest-go-aks \
  --yes \
  --no-wait
Enter fullscreen mode Exit fullscreen mode

Then I checked the public IP resources:

az network public-ip list \
  --query "[?location=='southafricanorth'].{Name:name,ResourceGroup:resourceGroup,IP:ipAddress}" \
  -o table
Enter fullscreen mode Exit fullscreen mode

The old resources were gone.

The frontend LoadBalancer was then able to obtain a public IP.

65. Final Frontend Public IP

The frontend Service eventually received:

4.253.76.190
Enter fullscreen mode Exit fullscreen mode

The browser application became accessible through:

http://4.253.76.190
Enter fullscreen mode Exit fullscreen mode

The final flow was therefore:

Browser
   |
   v
4.253.76.190
   |
   v
Azure Load Balancer
   |
   v
k8app-frontend Service
   |
   v
Frontend Pod
Enter fullscreen mode Exit fullscreen mode

66. Important Lesson From the LoadBalancer Failure

The Kubernetes Service itself was not necessarily the problem.

The chain was:

Service
   |
   v
Azure Load Balancer
   |
   v
Azure Public IP
Enter fullscreen mode Exit fullscreen mode

The Azure subscription had reached its public IP quota.

So the correct troubleshooting process was:

EXTERNAL-IP pending
        |
        v
kubectl describe service
        |
        v
PublicIPCountLimitReached
        |
        v
Check Azure resources
        |
        v
Find unused AKS cluster
        |
        v
Delete unused cluster
        |
        v
LoadBalancer gets public IP
Enter fullscreen mode Exit fullscreen mode

This was one of the most useful troubleshooting lessons in the project.

67. Azure DevOps Variable Group

The database password should not be placed directly in the pipeline YAML.

So I created a variable group:

k8app-dev-secrets
Enter fullscreen mode Exit fullscreen mode

Inside it:

POSTGRES_PASSWORD
Enter fullscreen mode Exit fullscreen mode

The variable was marked as secret.

68. Create the Variable Group in Azure DevOps

Go to:

Azure DevOps
    ↓
Pipelines
    ↓
Library
    ↓
+ Variable group
Enter fullscreen mode Exit fullscreen mode

Create:

Variable group name:
k8app-dev-secrets
Enter fullscreen mode Exit fullscreen mode

Add:

Name:
POSTGRES_PASSWORD

Value:
<your PostgreSQL password>
Enter fullscreen mode Exit fullscreen mode

Enable:

Keep this value secret
Enter fullscreen mode Exit fullscreen mode

Save the variable group.

69. Why the Pipeline Uses - group

The pipeline contains:

variables:
- group: k8app-dev-secrets
Enter fullscreen mode Exit fullscreen mode

This means:

Load the variables stored inside the Azure DevOps variable group named k8app-dev-secrets into this pipeline.

Later, the pipeline can reference:

$(POSTGRES_PASSWORD)
Enter fullscreen mode Exit fullscreen mode

So the flow is:

Azure DevOps Library
       |
       v
k8app-dev-secrets
       |
       +-- POSTGRES_PASSWORD
       |
       v
Azure DevOps Pipeline
       |
       v
Helm
       |
       v
Kubernetes Secret
       |
       v
Backend Pod
Enter fullscreen mode Exit fullscreen mode

The actual password should never be committed to Git.

70. The Azure DevOps Pipeline

The pipeline builds both images and then deploys them to DEV.

The important variables are:

variables:
- group: k8app-dev-secrets

- name: dockerRegistryServiceConnection
  value: 'a41b9e21-7348-4152-b6ca-249e8c78aae9'

- name: containerRegistry
  value: 'dessydevopsacr.azurecr.io'

- name: vmImageName
  value: 'ubuntu-latest'

- name: tag
  value: '$(Build.BuildId)'
Enter fullscreen mode Exit fullscreen mode

The important part is:

tag: '$(Build.BuildId)'
Enter fullscreen mode Exit fullscreen mode

This means every pipeline run can produce a unique image tag.

For example:

Build ID = 21
Enter fullscreen mode Exit fullscreen mode

produces:

dessydevopsacr.azurecr.io/k8app-backend:21
dessydevopsacr.io/k8app-frontend:21
Enter fullscreen mode Exit fullscreen mode

The correct registry hostname is:

dessydevopsacr.azurecr.io
Enter fullscreen mode Exit fullscreen mode

71. Build Backend Image

The backend job uses Docker@2.

Conceptually:

- task: Docker@2
  inputs:
    command: buildAndPush
    containerRegistry: $(dockerRegistryServiceConnection)
    repository: k8app-backend
    dockerfile: backend/backend.dockerfile
    buildContext: backend
    tags: |
      $(tag)
Enter fullscreen mode Exit fullscreen mode

The important pieces are:

Dockerfile:
backend/backend.dockerfile

Build context:
backend

Repository:
k8app-backend

Tag:
Build.BuildId
Enter fullscreen mode Exit fullscreen mode

The result goes to:

dessydevopsacr.azurecr.io/k8app-backend:<BuildID>
Enter fullscreen mode Exit fullscreen mode

72. Build Frontend Image

The frontend job does the same thing.

- task: Docker@2
  inputs:
    command: buildAndPush
    containerRegistry: $(dockerRegistryServiceConnection)
    repository: k8app-frontend
    dockerfile: frontend/frontend.dockerfile
    buildContext: frontend
    tags: |
      $(tag)
Enter fullscreen mode Exit fullscreen mode

The result is:

dessydevopsacr.azurecr.io/k8app-frontend:<BuildID>
Enter fullscreen mode Exit fullscreen mode

73. Deploy Stage

After the Build stage succeeds, the pipeline runs:

DeployDEV
Enter fullscreen mode Exit fullscreen mode

The deployment uses:

dessy-aks-deploy
Enter fullscreen mode Exit fullscreen mode

as its Azure service connection.

The pipeline obtains AKS credentials:

az aks get-credentials \
  --resource-group devops-aks-lab-rg-sa \
  --name dessy-aks-cluster \
  --overwrite-existing
Enter fullscreen mode Exit fullscreen mode

Then Helm performs the deployment.

74. Helm Deployment From Azure DevOps

The deployment command is:

helm upgrade --install k8app helm/k8app \
  --namespace backend \
  --set frontend.image.tag=$(Build.BuildId) \
  --set backend.image.tag=$(Build.BuildId) \
  --set backend.secret.postgresPassword="$(POSTGRES_PASSWORD)" \
  --wait \
  --timeout 5m
Enter fullscreen mode Exit fullscreen mode

Let's break this down.

helm upgrade --install 

helm upgrade --install
Enter fullscreen mode Exit fullscreen mode

means:

If the release exists, upgrade it. If it doesn't exist, install it.

This makes the pipeline reusable.

Release name 

k8app
Enter fullscreen mode Exit fullscreen mode

This is the Helm release name.

Chart 

helm/k8app
Enter fullscreen mode Exit fullscreen mode

This tells Helm where the chart is.

Namespace 

--namespace backend
Enter fullscreen mode Exit fullscreen mode

This is where the Helm release is tracked.

Frontend image 

--set frontend.image.tag=$(Build.BuildId)
Enter fullscreen mode Exit fullscreen mode

The pipeline automatically passes the current build ID into Helm.

Backend image 

--set backend.image.tag=$(Build.BuildId)
Enter fullscreen mode Exit fullscreen mode

The backend receives the same build ID.

That means one pipeline run corresponds to the same application version:

Build 21
 |
 +-- frontend:21
 |
 +-- backend:21
Enter fullscreen mode Exit fullscreen mode

PostgreSQL password 

--set backend.secret.postgresPassword="$(POSTGRES_PASSWORD)"
Enter fullscreen mode Exit fullscreen mode

The password comes from the Azure DevOps variable group.

It is then passed into the Helm chart and rendered into the Kubernetes Secret.

75. The Complete Image Flow

This is the relationship I wanted to understand:

Azure DevOps Build
       |
       | Build.BuildId = 21
       |
       +--------------------+
       |                    |
       v                    v
frontend:21            backend:21
       |                    |
       +---------+----------+
                 |
                 v
               ACR
                 |
                 v
               Helm
                 |
                 v
                AKS
Enter fullscreen mode Exit fullscreen mode

This gives us traceability between:

Pipeline run
      ↓
Docker image
      ↓
Helm release
      ↓
Kubernetes deployment
      ↓
Running application
Enter fullscreen mode Exit fullscreen mode

76. Final Pipeline Structure

The pipeline has two main stages:

stages:

- stage: Build
  displayName: Build and Push Images

- stage: DeployDEV
  displayName: Deploy to DEV
  dependsOn: Build
Enter fullscreen mode Exit fullscreen mode

The relationship is:

Build
  |
  | success
  v
DeployDEV
Enter fullscreen mode Exit fullscreen mode

The deployment cannot start until the build stage has succeeded.

77. Verify the Deployment From the Pipeline

After Helm deployment, I verify:

kubectl get deployments -n frontend
kubectl get deployments -n backend

kubectl get services -n frontend
kubectl get services -n backend
Enter fullscreen mode Exit fullscreen mode

I also use:

kubectl get pods -n frontend
kubectl get pods -n backend
Enter fullscreen mode Exit fullscreen mode

The objective is:

Frontend Pod     Running
Backend Pod      Running
Frontend Service LoadBalancer
Backend Service  ClusterIP
Enter fullscreen mode Exit fullscreen mode

78. Useful Kubernetes Debugging Commands

When a deployment fails, these are some of the commands I use most often.

List Pods 

kubectl get pods -n frontend
Enter fullscreen mode Exit fullscreen mode 
kubectl get pods -n backend
Enter fullscreen mode Exit fullscreen mode

Check Pod Details 

kubectl describe pod <pod-name> -n backend
Enter fullscreen mode Exit fullscreen mode

This is useful for:

  • Scheduling problems
  • Image pull errors
  • Environment problems
  • Mount problems
  • Events

Check Logs 

kubectl logs <pod-name> -n backend
Enter fullscreen mode Exit fullscreen mode

For the previous crashed container:

kubectl logs <pod-name> -n backend --previous
Enter fullscreen mode Exit fullscreen mode

Check Deployment 

kubectl describe deployment k8app-backend -n backend
Enter fullscreen mode Exit fullscreen mode

Check Services 

kubectl get services -n backend
Enter fullscreen mode Exit fullscreen mode 
kubectl describe service k8app-frontend -n frontend
Enter fullscreen mode Exit fullscreen mode

Check ConfigMaps 

kubectl get configmaps -n backend
Enter fullscreen mode Exit fullscreen mode 
kubectl describe configmap k8app-backend-config -n backend
Enter fullscreen mode Exit fullscreen mode

Check Secrets 

kubectl get secrets -n backend
Enter fullscreen mode Exit fullscreen mode

Do not casually print secret values into logs or screenshots.

79. Useful Helm Debugging Commands

Render the chart:

helm template test-release helm/k8app
Enter fullscreen mode Exit fullscreen mode

Render with debug:

helm template test-release helm/k8app --debug
Enter fullscreen mode Exit fullscreen mode

List releases:

helm list -n backend
Enter fullscreen mode Exit fullscreen mode

Check release:

helm status k8app -n backend
Enter fullscreen mode Exit fullscreen mode

Check release history:

helm history k8app -n backend
Enter fullscreen mode Exit fullscreen mode

Upgrade:

helm upgrade k8app helm/k8app \
  --namespace backend \
  --set backend.image.tag=21 \
  --set frontend.image.tag=21 \
  --wait \
  --timeout 5m
Enter fullscreen mode Exit fullscreen mode

80. Why helm template Is So Useful

One of the biggest Helm lessons for me was that Helm is essentially a templating and release-management layer around Kubernetes manifests.

For example:

image: "{{ .Values.backend.image.repository }}:{{ .Values.backend.image.tag }}"
Enter fullscreen mode Exit fullscreen mode

is not valid final Kubernetes configuration until Helm renders it.

Running:

helm template
Enter fullscreen mode Exit fullscreen mode

lets me see what Kubernetes will actually receive.

The debugging process becomes:

Helm template
     |
     v
Rendered YAML
     |
     v
Check values
     |
     v
Fix template
     |
     v
Deploy
Enter fullscreen mode Exit fullscreen mode

81. Verify the ACR Images

From Azure CLI:

az acr repository list \
  --name dessydevopsacr \
  -o table
Enter fullscreen mode Exit fullscreen mode

You should see repositories such as:

k8app-backend
k8app-frontend
Enter fullscreen mode Exit fullscreen mode

List tags for the backend:

az acr repository show-tags \
  --name dessydevopsacr \
  --repository k8app-backend \
  -o table
Enter fullscreen mode Exit fullscreen mode

Frontend:

az acr repository show-tags \
  --name dessydevopsacr \
  --repository k8app-frontend \
  -o table
Enter fullscreen mode Exit fullscreen mode

This helps verify that the CI stage actually pushed the images.

82. Verify Which Image Kubernetes Is Running

You can inspect the Deployment:

kubectl get deployment k8app-backend \
  -n backend \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
Enter fullscreen mode Exit fullscreen mode

And frontend:

kubectl get deployment k8app-frontend \
  -n frontend \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
Enter fullscreen mode Exit fullscreen mode

This should show the expected Build ID.

For example:

dessydevopsacr.azurecr.io/k8app-backend:21
Enter fullscreen mode Exit fullscreen mode

This is useful when troubleshooting:

Did the pipeline build the correct image, or is AKS still running an older version?

83. Verify the Application

After the deployment succeeds:

kubectl get service k8app-frontend -n frontend
Enter fullscreen mode Exit fullscreen mode

Get the external IP.

Then open the IP in a browser.

For this deployment, the frontend became accessible through:

http://4.253.76.190
Enter fullscreen mode Exit fullscreen mode

The application loaded successfully.

That gave me the final checkpoint:

GitHub
   ↓
Azure DevOps
   ↓
Build
   ↓
Docker
   ↓
ACR
   ↓
Helm
   ↓
AKS
   ↓
Frontend
   ↓
Running Application
Enter fullscreen mode Exit fullscreen mode

84. The Complete DEV Environment

At the end, the important Azure resources were:

Resource Group
devops-aks-lab-rg-sa
Enter fullscreen mode Exit fullscreen mode 
ACR
dessydevopsacr
Enter fullscreen mode Exit fullscreen mode 
AKS
dessy-aks-cluster
Enter fullscreen mode Exit fullscreen mode 
PostgreSQL
dessy-k8app-postgres
Enter fullscreen mode Exit fullscreen mode 
PostgreSQL subnet
postgres-subnet
10.0.2.0/24
Enter fullscreen mode Exit fullscreen mode 
Private DNS
private.postgres.database.azure.com
Enter fullscreen mode Exit fullscreen mode

The Kubernetes namespaces were:

frontend
backend
Enter fullscreen mode Exit fullscreen mode

The Helm release was:

k8app
Enter fullscreen mode Exit fullscreen mode

The container images were:

dessydevopsacr.azurecr.io/k8app-frontend:<BuildID>

dessydevopsacr.azurecr.io/k8app-backend:<BuildID>
Enter fullscreen mode Exit fullscreen mode

85. A Simple Way to Think About the Whole System

After building the project, this is the mental model I use:

SOURCE CODE
    |
    v
Azure DevOps
    |
    | Build
    v
Docker Image
    |
    v
ACR
    |
    | Pull
    v
AKS
    |
    +--------------------+
    |                    |
    v                    v
Frontend              Backend
    |                    |
LoadBalancer          ClusterIP
                         |
                         v
                PostgreSQL Flexible
                      Server
                         |
                         v
                  Private Network
                         |
                         v
                   Private DNS
Enter fullscreen mode Exit fullscreen mode

Each component has one main responsibility.

Component Responsibility
GitHub Source code
Azure DevOps CI/CD orchestration
Docker Package applications
ACR Store container images
Helm Package/deploy Kubernetes resources
AKS Run containers
Kubernetes Service Provide stable networking
ConfigMap Non-secret configuration
Secret Sensitive configuration
PostgreSQL Flexible Server Managed database
Private DNS Resolve private database hostname
Azure RBAC Azure authorization
Kubernetes RBAC Kubernetes authorization

86. Important Lessons I Learned

Lesson 1 — Authentication and authorization are different

I initially thought:

If Azure DevOps can authenticate to Azure,
it should automatically be able to deploy to Kubernetes.
Enter fullscreen mode Exit fullscreen mode

That is not necessarily true.

We had:

OIDC
 ↓
Azure authentication
Enter fullscreen mode Exit fullscreen mode

then:

Azure RBAC
 ↓
Azure permissions
Enter fullscreen mode Exit fullscreen mode

and separately:

Kubernetes RBAC
 ↓
Kubernetes permissions
Enter fullscreen mode Exit fullscreen mode

Lesson 2 — ACR push and AKS pull are different operations

Azure DevOps needs permission to:

PUSH
Enter fullscreen mode Exit fullscreen mode

images into ACR.

AKS needs permission to:

PULL
Enter fullscreen mode Exit fullscreen mode

images from ACR.

Therefore:

Azure DevOps
     |
     | Push
     v
ACR
     |
     | Pull
     v
AKS
Enter fullscreen mode Exit fullscreen mode

Lesson 3 — Private networking requires DNS

It is not enough to have:

Private IP
Enter fullscreen mode Exit fullscreen mode

The application normally uses:

hostname
Enter fullscreen mode Exit fullscreen mode

Therefore:

Hostname
   ↓
Private DNS
   ↓
Private IP
   ↓
Database
Enter fullscreen mode Exit fullscreen mode

The Private DNS zone was:

private.postgres.database.azure.com
Enter fullscreen mode Exit fullscreen mode

Lesson 4 — ConfigMap and Secret solve different problems

ConfigMap:

Normal configuration
Enter fullscreen mode Exit fullscreen mode

Secret:

Sensitive configuration
Enter fullscreen mode Exit fullscreen mode

So:

ConfigMap
 |
 +-- PROJECT_NAME
 +-- POSTGRES_SERVER
 +-- POSTGRES_DB
Enter fullscreen mode Exit fullscreen mode

while:

Secret
 |
 +-- POSTGRES_USER
 +-- POSTGRES_PASSWORD
Enter fullscreen mode Exit fullscreen mode

Lesson 5 — Kubernetes Pods are not permanent endpoints

Pods can be recreated.

Therefore applications should normally communicate through Services rather than hardcoding Pod IPs.

Pod
 ↓
Service
 ↓
Stable endpoint
Enter fullscreen mode Exit fullscreen mode

Lesson 6 — kubectl describe is extremely useful

When something is:

Pending
Enter fullscreen mode Exit fullscreen mode

or:

Failed
Enter fullscreen mode Exit fullscreen mode

don't immediately change configuration.

First investigate.

For example:

kubectl describe service k8app-frontend -n frontend
Enter fullscreen mode Exit fullscreen mode

revealed:

PublicIPCountLimitReached
Enter fullscreen mode Exit fullscreen mode

That saved a lot of unnecessary Kubernetes troubleshooting.

87. Troubleshooting Cheat Sheet

Pod is CrashLoopBackOff

Start with:

kubectl logs <pod> -n <namespace>
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl describe pod <pod> -n <namespace>
Enter fullscreen mode Exit fullscreen mode

Look for:

missing environment variables
bad configuration
application startup failure
database connection failure
Enter fullscreen mode Exit fullscreen mode

Pod is Pending

Run:

kubectl describe pod <pod> -n <namespace>
Enter fullscreen mode Exit fullscreen mode

Look at:

Events
Enter fullscreen mode Exit fullscreen mode

Possible causes include:

insufficient resources
scheduling problem
node problem
Enter fullscreen mode Exit fullscreen mode

ImagePullBackOff

Check:

kubectl describe pod <pod> -n <namespace>
Enter fullscreen mode Exit fullscreen mode

Verify:

Image name
Image tag
ACR repository
ACR permissions
AKS AcrPull role
Enter fullscreen mode Exit fullscreen mode

Service EXTERNAL-IP is Pending

Run:

kubectl describe service <service> -n <namespace>
Enter fullscreen mode Exit fullscreen mode

Check the events.

Don't assume the Kubernetes Service itself is broken.

The problem could be Azure infrastructure or quota.

Helm deployment fails

First run:

helm template test-release helm/k8app
Enter fullscreen mode Exit fullscreen mode

Then:

helm upgrade --install ...
Enter fullscreen mode Exit fullscreen mode

If necessary:

helm template test-release helm/k8app --debug
Enter fullscreen mode Exit fullscreen mode

Backend cannot connect to PostgreSQL

Check:

1. DNS
2. Private IP
3. Network connectivity
4. PostgreSQL server
5. Database name
6. Username
7. Password
8. TLS configuration
Enter fullscreen mode Exit fullscreen mode

For DNS:

kubectl run psql-test \
  --rm -it \
  --image=postgres:16 \
  --restart=Never \
  -- \
  getent hosts dessy-k8app-postgres.postgres.database.azure.com
Enter fullscreen mode Exit fullscreen mode

88. Security Notes

There are a few things I would improve before calling this a production-ready implementation.

First, never commit:

POSTGRES_PASSWORD
Enter fullscreen mode Exit fullscreen mode

to Git.

Never put it in:

values.yaml
Enter fullscreen mode Exit fullscreen mode

or:

Deployment YAML
Enter fullscreen mode Exit fullscreen mode

The real password should also never appear in:

screenshots
Dev.to article
GitHub repository
pipeline logs
Enter fullscreen mode Exit fullscreen mode

For a production implementation, I would also consider:

Azure Key Vault
External Secrets
Workload Identity
More restrictive RBAC
Private ACR
Private AKS networking
Network policies
TLS
Ingress
Health probes
Database migrations
Enter fullscreen mode Exit fullscreen mode

The current project is a learning environment designed to understand the complete flow.

89. What I Would Improve Next

The current DEV environment works, but there are several areas I want to improve.

1. Database migrations

The PostgreSQL database currently has no application tables.

The next step is to introduce migrations so the deployment process can create/update the schema safely.

2. Better secret management

The Azure DevOps variable group works for the learning environment.

A more production-oriented design would use:

Azure Key Vault
       |
       v
Workload Identity / Secret integration
       |
       v
AKS
Enter fullscreen mode Exit fullscreen mode

3. Ingress

The frontend currently uses:

LoadBalancer
Enter fullscreen mode Exit fullscreen mode

A production environment could instead use an ingress architecture:

Internet
   |
   v
WAF / Application Gateway
   |
   v
Ingress
   |
   +---- frontend
   |
   +---- backend
Enter fullscreen mode Exit fullscreen mode

This also provides a cleaner way to route traffic between frontend and backend.

4. Health probes

The Deployments should eventually have:

livenessProbe
readinessProbe
Enter fullscreen mode Exit fullscreen mode

This allows Kubernetes to understand whether an application is:

alive
Enter fullscreen mode Exit fullscreen mode

and whether it is:

ready to receive traffic
Enter fullscreen mode Exit fullscreen mode

5. Pipeline error handling

The deployment script should explicitly fail when an important command fails.

For example, I would eventually add:

set -euo pipefail
Enter fullscreen mode Exit fullscreen mode

This prevents a later command from accidentally hiding an earlier deployment failure.

6. Environment-specific Helm values

Eventually I want:

values-dev.yaml
values-uat.yaml
values-prod.yaml
Enter fullscreen mode Exit fullscreen mode

Then:

DEV
 ↓
UAT
 ↓
Approval
 ↓
PROD
Enter fullscreen mode Exit fullscreen mode

90. What Comes Next: UAT and PROD

For now, I deliberately stopped at DEV.

The next architecture will be:

             Build
               |
               v
              DEV
               |
               v
              UAT
               |
               v
        Approval Gate
               |
               v
             PROD
Enter fullscreen mode Exit fullscreen mode

The idea is that the same application artifact should move through environments rather than rebuilding different artifacts for each environment.

That gives us:

Build once
    |
    v
Promote the same version
    |
    +---- DEV
    |
    +---- UAT
    |
    +---- PROD
Enter fullscreen mode Exit fullscreen mode

This is the direction I want to take the Azure DevOps pipeline next.

91. Final DEV Checklist

Before considering the DEV environment complete, I verify:

[✓] Resource Group created

[✓] Azure Container Registry created

[✓] AKS created

[✓] PostgreSQL Flexible Server created

[✓] PostgreSQL private subnet created

[✓] Private DNS configured

[✓] AKS can resolve PostgreSQL hostname

[✓] AKS can connect to PostgreSQL

[✓] ACR contains frontend image

[✓] ACR contains backend image

[✓] AKS has AcrPull permission

[✓] frontend namespace created

[✓] backend namespace created

[✓] ConfigMap created

[✓] Kubernetes Secret created

[✓] Backend Deployment running

[✓] Frontend Deployment running

[✓] Backend Service is ClusterIP

[✓] Frontend Service is LoadBalancer

[✓] Helm chart renders successfully

[✓] Helm release deployed

[✓] Azure DevOps build succeeds

[✓] Azure DevOps deploys to AKS

[✓] Frontend receives public IP

[✓] Application accessible from browser
Enter fullscreen mode Exit fullscreen mode

Conclusion

This project started as a way for me to practice Azure DevOps and Kubernetes.

It became much more useful once I stopped looking at the individual commands and started understanding how the components fit together.

The final DEV deployment looks like:

                    GitHub
                       |
                       v
                Azure DevOps
                       |
                       v
              Build + Test
                       |
                       v
                Docker Build
                       |
                       v
                     ACR
                       |
                       v
                    Helm
                       |
                       v
                     AKS
                  /       \
                 /         \
                v           v
          Frontend       Backend
                |           |
          LoadBalancer   ClusterIP
                            |
                            v
                     Private Network
                            |
                            v
                  PostgreSQL Flexible
                         Server
Enter fullscreen mode Exit fullscreen mode

The most important thing I learned was that DevOps is not simply about writing YAML.

When something fails, I need to understand:

What failed?
     |
     v
Which component owns the failure?
     |
     v
How does that component communicate
with the other components?
     |
     v
What identity is being used?
     |
     v
What permission is required?
     |
     v
What evidence can I collect?
     |
     v
What should I change?
Enter fullscreen mode Exit fullscreen mode

For example:

Backend failed
      |
      v
Check logs
      |
      v
Missing environment variables
      |
      v
ConfigMap + Secret
      |
      v
Deployment references them
      |
      v
Backend starts
Enter fullscreen mode Exit fullscreen mode

And:

Frontend LoadBalancer pending
      |
      v
kubectl describe service
      |
      v
PublicIPCountLimitReached
      |
      v
Check Azure resources
      |
      v
Find unused AKS cluster
      |
      v
Remove unused resource
      |
      v
LoadBalancer receives public IP
Enter fullscreen mode Exit fullscreen mode

Those troubleshooting experiences were more valuable than simply getting a green pipeline on the first attempt.

The final goal is not just:

"I know the commands."

The goal is:

I understand what each component does, why it exists, how the components communicate, how identities and permissions work, and how to troubleshoot the environment when something breaks.

That is the foundation I now have for moving from DEV to UAT, approval gates and PROD.

Top comments (0)