Advanced Kubernetes Deployment with GitOps: A Hands-On Guide To Terraform, Ansible, ArgoCD And Observability Tools

Table of Contents

1. Introduction
2. Overview
3. Infrastructure Provisioning
4. Application Management
5. Logging and Monitoring
6. Challenges and Lessons Learned
7. Conclusion and Final Thoughts

Introduction

GitOps is a modern approach to managing software applications and the infrastructure they run on, leveraging Git as the single source of truth for version control. It integrates seamlessly with practices like Continuous Integration/Continuous Deployment (CI/CD) and Infrastructure as Code (IaC) to ensure consistency, automation, and reliability. The core workflow involves developers committing code or configuration changes to a Git repository, which triggers automated pipelines to build, test, and deploy the updates to the target environment.

GitOps is often used to manage Kubernetes clusters and cloud-native app development.
Kubernetes itself is an orchestration system used to deploy and manage containerized applications.

I recently completed a project where I followed GitOps practices to deploy a microservices based applications managed with kubernetes.
This project is a demonstration of modern DevOps practices through the implementation of GitOps to deploy a Kubernetes-managed microservices application. By using Git as the single source of truth for both infrastructure and application state, the project highlights the power of version-controlled, automated, and auditable deployments. GitOps, being Kubernetes-native, aligns seamlessly with container orchestration and microservices architecture, showcasing scalability, resilience, and automation in cloud-native systems. Additionally, it embraces the "shift-left" approach by empowering developers to contribute to operations through simple Git workflows, fostering collaboration and reducing deployment errors. With its focus on observability, traceability, and the ability to scale across diverse environments, this project underscores key principles of modern DevOps, making it highly relevant in today's software delivery landscape.

Overview

The aim of this project is to create a fully automated setup for deploying a microservices based application using kubernetes (helm) and also deploy observability tools on the provisioned kubernetes cluster to both monitor the state of the cluster and retreive logs.
The following tools and services used in the project.

• Terraform
• Ansible
• Azure Kubernetes Cluster
• Docker/Dockerhub
• Trivy
• Helm
• Kube-Prometheus Stack (monitoring)
• EFK Stack (logging)

This diagram shows the full flow of the project:

A fully complete DevOps setup typically involves multiple integrated components. This project utilized three Git repositories to manage distinct aspects of the workflow:

• Infrastructure Repository: Hosts Terraform and Ansible configurations for provisioning and configuring the Kubernetes cluster.

• Application Repository: Contains the microservices application source code, including the Dockerfile for building container images.

• Kubernetes Manifests Repository: Stores Kubernetes manifests and Helm charts for application deployment.

The project workflow involves running terraform apply to provision or update an Azure Kubernetes Service (AKS) cluster and generate an Ansible hosts.ini file with cluster details. Once the cluster is ready, Terraform triggers Ansible to execute additional tasks, including:

• Deploying the Kube-Prometheus stack for monitoring
• Setting up the EFK stack for centralized logging
• Installing ArgoCD using Helm
• Configuring an ArgoCD application to sync with the Kubernetes Manifests repository, ensuring that the cluster state matches the repository's desired state

This setup streamlines the process of deploying applications and observability tools. With a single terraform apply command, the entire infrastructure, monitoring, logging, and GitOps deployment configuration is provisioned and operational.

Infrastructure Provisioning

Azure Kubernetes Cluster is a managed kubernetes cluster which is used as the main infrastructure for deploying our microservices applications. Provisioning of the cluster is managed using terraform, an infrastructure as code tool.

resource "azurerm_kubernetes_cluster" "app-cluster" {
  name                = "app-cluster"
  location            = data.azurerm_resource_group.default.location
  resource_group_name = data.azurerm_resource_group.default.name
  dns_prefix          = "app-k8s"
  kubernetes_version  = "1.30.6"

  default_node_pool {
    name            = "app"
    node_count      = 2
    vm_size         = "Standard_DS2_v2"
    os_disk_size_gb = 50
  }
}

This is a basic code structure that create an AKS cluster.

After cluster creation, terraform triggers ansible to run:

  provisioner "local-exec" {
    command = <<-EOT
      sleep 60 # wait for cluster to be ready
      KUBECONFIG=../ansible/kubeconfig \
      ansible-playbook \
        -i ../ansible/inventory/hosts.ini \
        ../ansible/playbook.yml \
        -e "kubernetes_cluster_name=${azurerm_kubernetes_cluster.app-cluster.name}" \
        -e "kubernetes_resource_group=${data.azurerm_resource_group.default.name}"
    EOT

    working_dir = "../ansible"
  }
}

Ansible setup has roles that use helm in installing ArgoCD, Kube-Prometheus stack and EFK stack. It also creates an ArgoCD appplication.

# playbook.yml

- name: Deploy tools on AKS
  hosts: localhost
  gather_facts: no
  vars:
    ansible_python_interpreter: /Library/Frameworks/Python.framework/Versions/3.12/bin/python3

  roles:
    - role: argocd_install
      vars:
        kubeconfig: "{{ kubeconfig_path }}"
      when: not node_status.failed

    - role: logging_stack
      vars:
        kubeconfig: "{{ kubeconfig_path }}"
      when: not node_status.failed

    - role: monitoring_stack
      vars:
        kubeconfig: "{{ kubeconfig_path }}"
      when: not node_status.failed

The roles configurations also contain the ArgoCd application manifest files. Complete configuration code can be found here.

Application Management

The application to be deployed is designed using microservices based architecture. The source codes and dockerfile for each microservice is being saved in a an application repo. A continous integration pipeline is also setup to handle code changes to the application code and dockerfile. The pipeline has jobs that:

• Detect the services that has had changes
• Build a new docker image and push it to dockerhub
• Scan the docker images for vulnerability using Trivy
• Update the manifest files in the manifest repository

The configuration codes for the workflow can be found here.

Kubernetes Manifests

Each services contains kubernetes manifest files for Deployment, Services and Service account. There is also an Ingress configuration file available. The manifest are also managed by a custom helm chart in the repository.

The Helm chart is being updated with current tags by the application repo workflow. It is the single source of truth used with ArgoCD in the kubernetes cluster.

GitOps Workflow

Argo CD is a continuous delivery (CD) tool for Kubernetes that uses GitOps to manage and deliver applications
It is synced to the manifests repo, and it ensures that the current state of the manifest repo is the deployed state in the kubernetes cluster.

ArgoCD is installed in the kubernetes cluster using Ansible, and an application that handles the manifest files is also created using ansible.

# infra-repo/ansible/roles/argocd_install/templates/application.yml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: {{ app_name }}
  namespace: {{ argocd_namespace }}
spec:
  project: default
  source:
    repoURL: {{ app_repo_url }}
    targetRevision: {{ app_target_revision }}
    path: {{ app_path }}
    helm:
      releaseName: {{ app_name }}
  destination:
    server: https://kubernetes.default.svc
    namespace: {{ app_namespace }}
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

---
# infra-repo/ansible/roles/argocd_install/templates/application.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: {{ argocd_namespace }}
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: {{ argocd_host }}
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: {{ argocd_release_name }}-server
            port:
              number: 80

Logging and Monitoring

Metrcis, Logs and Alerts are pillars of observability. Deploying tools that simplify logging and monitoring our performance metrics is even more crucial in a microservices deployment where a single kubect logs on multiple pods and services might not suffice if you are trying to troubleshoot. This is where efficient and centralized metrics and log management tools like Kube-Prometheus (Prometheus, Grafana, Alert Manager) and EFK (Elasticsearch, Fluent Bit and Kibana) come in.

In this project setup, after terraform apply runs successfully, the Kube-prometheus stack and EFK stack are also deployed in the AKS cluster in the monitoring and logging namespaces respectively and with default configurations.
The configuration files that handles this can be accessed here.

Once the services are up, Kibana and Grafana UI can be accessed using their services via ingress or port-forwarding.

screenshot of deployed applications

screenshot of dashboards from kube-prometheus stack

screenshot of dashboard view with data on grafana

Challenges and Lessons Learned

Challenges:

• Managing Repository Synchronization:
  Ensuring consistent updates across the Infra, Application, and K8 Manifests repositories was complex, especially with multiple dependencies.

• Tooling Compatibility:
  Configuring Terraform and Ansible to work seamlessly required troubleshooting, particularly when passing dynamic values

• ArgoCD Configuration:
  Setting up ArgoCD applications to handle multiple microservices and ensuring proper synchronization posed a learning curve.

• Docker Image Management:
  Scanning and pushing Docker images while managing triggers for CI pipelines was challenging to automate effectively.

• Debugging Deployment Failures:
  Diagnosing deployment issues, such as misconfigured Kubernetes manifests or failed CI/CD pipelines, consumed significant time.

Lessons Learned:

• Importance of GitOps:
  GitOps principles simplify complex workflows by centralizing configurations in version-controlled repositories.

• Toolchain Proficiency:
  Cemented my understanding of tools like Terraform, Ansible, and ArgoCD significantly reduces setup time.

• Iterative Approach:
  Breaking down tasks into smaller, testable units (e.g., individual Ansible roles) improves development and debugging efficiency.

• Documentation and Automation:
  Comprehensive documentation and automation reduce the risk of misconfigurations during handoffs or scaling.

Conclusion

This project was a in-depth exploration of GitOps principles applied to a microservices based application deployment. By integrating Terraform, Ansible, and ArgoCD, I was able to streamline Kubernetes deployments while leveraging CI/CD pipelines with Github Actions for application management. Setting up logging and monitoring stacks ensured operational visibility while overcoming challenges deepened our understanding of Kubernetes and GitOps workflows.

This experience underscores the value of automation and iterative problem-solving in managing modern cloud-native architectures. Focusing more on the overall workflow and how the many different parts come together has been the biggest success of the project for me. As a next step, enhancing scalability, enhancing security, improving the overall continous integegration pipelines and exploring advanced GitOps practices, such as automated rollbacks and canary deployments, could further refine the workflow.