warming up
To Developers, DevOps seems like "black magic" that no one wants to be a part of.
I generally agree. Who wants to take part in this automatic headache?
Then a big question arises- As to how much I want to enrich the knowledge of the developers in ops tools - should I give the "keys to the spaceship" ?
On the one hand, it optimizes and directs the DevOps work to architecture and future tasks, but at the same time leaves a large margin for human error and bugs on the dev team. Guess who in the end will still be called to the flag when error occur.
These are some of the day-to-day requirements of a development team:
- deploy a new service easily.
- add an environment variable to all services in product x in environment y.
- change the db name in all services in the environment.
- drift a product from a DEV environment to a QA environment and keep the same QA resource allocation.
- change the default resource allocation of all services in the organization.
When you have a stack of 5-10 services these can be simple tasks that a developer can handle with a bit of brainstorming. But what happens in a large development team, high-scale of services (100+) divided into products, and in a large amount of environments?
We will have to plan a little more in depth for each requirement of the development team in order to both maintain the micro capabilities of each service separately and also to carry out broad changes in the environment.
That’s why I chose the helm library + helm umbrella stack deployed using ArgoCD Application set.
GitOps and ArgoCD
GitOps is a software engineering practice that uses a Git repository as its single source of truth. Teams commit declarative configurations into Git, and these configurations are used to create environments needed for the continuous delivery process.
One of the common tools applying GitOps method is ArgoCD , a Kubernetes-native continuous deployment (CD) tool implemented as kubernetes controller , responsible for continuously running and monitoring applications by comparing the live state to the desired state stored in the target git repository.
Enables developers to manage application configuration by code in a git repository and deploy it automatically or manually to kubernetes.
Provides automatic sync of application state to the current version of declarative configuration.
Managed by web interface and command-line interface.
Sneak peak
In the original project, I implemented the solution on several environments. Each branch in the repo represents the state of each ArgoCD in the dedicated cluster. The state is a declarative configuration of both infrastructure and products implemented by Helm charts and ArgoCD applications. For the purpose of this article, I will present the repository structure of one of the environments:
├── Chart.lock
├── Chart.yaml
├── README.md
├── apps
│ └── cms
│ └── application-env.properties
├── charts
│ ├── Tikal-main-chart-0.0.5.tgz
│ └── auth-service-1.0.0.tgz
├── envs
│ ├── auto3
│ │ └── services-values.yaml
│ ├── qa-euw1
│ │ └── services-values.yaml
│ └── qa-usw2
│ └── services-values.yaml
├── services
│ ├── auth-service
│ │ ├── Chart.yaml
│ │ ├── templates
│ │ │ ├── deployment.yaml
│ │ │ ├── hpa.yaml
│ │ │ ├── ingress.yaml
│ │ │ ├── service.yaml
│ │ │ └── serviceAccount.yaml
│ │ └── values.yaml
│ └── report-service
│ ├── Chart.yaml
│ ├── templates
│ │ ├── deployment.yaml
│ │ ├── hpa.yaml
│ │ ├── ingress.yaml
│ │ ├── service.yaml
│ │ └── serviceAccount.yaml
│ └── values.yaml
└── templates
└── cm.yaml
As said before, An ArgoCD application resource is a Kubernetes resource that defines an application managed by ArgoCD. This resource specifies the details of the application, including its source repository, the path to the application's manifest files, and any additional configuration settings.
The project tree is composed of two main directories: ArgoCD directory contains files related to ArgoCD, such as application definitions and kustomization file used to define a kustomization in Kubernetes.
Helm contains the chart of the infrastructure and the ApplictionSet chart We will discuss it in detail soon.
Together, these two directories form the structure of the project tree.
Tikal.yaml file in the main application file , it runs the "App-of-app" application .
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: app-of-apps
namespace: argocd
labels:
group: auto3
spec:
project: tikal
source:
repoURL: git@github.com:tikal/tikal_helm_deploy
path: argocd/app-of-apps
targetRevision: auto3
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
prune: true
selfHeal: true
Another folder I want to discuss shortly is the Terraform folder.
This project was installed by terraform basically but it can deployed easily with simple helm installation of ArgoCD .
In the next attachment we can see the relevant values neccesary to apply our ArgoCD system :
values-override.tpl
controller:
# If changing the number of replicas you must pass the number as ARGOCD_CONTROLLER_REPLICAS as an environment variable
replicas: 2
enableStatefulSet: true
env:
- name: "ARGOCD_CONTROLLER_REPLICAS"
value: "2"
resources:
limits:
cpu: 4M
memory: 4Gi
requests:
cpu: 3M
memory: 3Gi
## Redis
redis:
enabled: false
redis-ha:
enabled: true
# Check the redis-ha chart for more properties
resources:
requests:
memory: 200Mi
cpu: 100m
limits:
memory: 700Mi
exporter:
enabled: false
server:
serviceAccount:
# -- Create server service account
create: true
# -- Server service account name
name: argocd-server
# -- Annotations applied to created service account
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::607827849963:role/argocd-get-oci-access-sultan
# -- Labels applied to created service account
labels: {}
# -- Automount API credentials for the Service Account
automountServiceAccountToken: true
replicas: 2
configEnabled: true
config:
repositories: |-
- name: ${repo_name}
type: git
url: ${repo_url}
sshPrivateKeySecret:
key: sshPrivateKey
name: github-repo-secret
- name: sonatype
type: helm
url: https://sonatype.github.io/helm3-charts/
- name: eks
type: helm
url: https://aws.github.io/eks-charts
- name: autoscaler
type: helm
url: https://kubernetes.github.io/autoscaler
- name: external-secrets
type: helm
url: https://external-secrets.github.io/kubernetes-external-secrets/
- name: bitnami
type: helm
url: https://charts.bitnami.com/bitnami
## Projects
## reference: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/
additionalProjects:
- name: tikal
namespace: argocd
description: tikal Project
sourceRepos:
- "*"
destinations:
- namespace: "*"
server: https://kubernetes.default.svc
clusterResourceWhitelist:
- group: '*'
kind: '*'
namespaceResourceWhitelist:
- group: '*'
kind: '*'
additionalApplications:
- name: app-of-apps
namespace: argocd
labels:
group: ${env_name}
project: tikal
source:
repoURL: git@github.com:tikal/tikal_helm_deploy
path: argocd/app-of-apps
targetRevision: ${env_name}
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
prune: true
selfHeal: true
repoServer:
## Repo server service account
## If create is set to true, make sure to uncomment the name and update the rbac section below
serviceAccount:
# -- Create repo server service account
create: true
# -- Repo server service account name
name: "argocd-repo-server"
# -- Annotations applied to created service account
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::607827849963:role/argocd-get-oci-access-sultan
labels: { }
# -- Automount API credentials for the Service Account
automountServiceAccountToken: true
# -- Additional containers to be added to the repo server pod
extraContainers: [ ]
# -- Repo server rbac rules
rbac:
- apiGroups:
- argoproj.io
resources:
- applications
verbs:
- get
- list
- watch
I have selected this configuration for our system because it separates the code responsible for deployment from the charts of our applications. This separation provides a clear separation between the build process and the deployment process.
Each environment has its own branch in our version control system, which contains all of the configurations and settings specific to that environment. This allows us to manage and track the differences between our development, staging, and production environments.
In addition we maintain a separate repository for each chart of our products. This helps us keep track of the chronology of each product which helps us maintain a clear separation of concerns and makes it easier to manage and update our codebase.
ApplicationSet Helm Chart
In our stack, each product is built from several services that are represented in an umbrella chart . To deploy these products with Argo, we use an application manifest for each service. Our application manifests for each service share the same variables and values, in this case we would like to use the Applicationset resource.
This technique can be used to deploy multiple products by converting the Applicationset manifest into an Applicationset helm chart and use it serially by injecting all the products names from our env-values.yaml
applicationset.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: tikal-products
namespace: argocd
spec:
generators:
- matrix:
generators:
- list:
elements:
{{- range $name,$value := .Values.apps }}
{{- if $value.enabled }}
- name: {{ default $name $value.name |replace "_" "-"}}
version: {{ $value.version }}
repo_url: {{ $value.repo_url }}
team: {{ $value.team }}
{{- end }}
{{- end }}
- list:
elements:
- propertiesEnvName: {{ .Values.cluster_name }}
domain: {{ .Values.domain_suffix }}
region: {{ .Values.cluster_region }}
cluster_name: {{ .Values.cluster_name }}
aws_account_id: '{{int .Values.aws_account_id }}'
db_url: {{ .Values.db_url }}
db_name: {{ .Values.db_name }}
db_admin: {{ .Values.db_admin }}
kafka_broker: {{ .Values.kafka_broker }}
external_domain: {{ .Values.external_domain }}
internal_domain: {{ .Values.internal_domain }}
redis_uri: {{ .Values.redis_uri }}
vpc_id: '{{ .Values.cluster_vpc_id }}'
template:
metadata:
name: 'tikal-{{"{{"}}name{{"}}"}}'
spec:
project: tikal
source:
path: ./
repoURL: '{{"{{"}}repo_url{{"}}"}}'
targetRevision: '{{"{{"}}version{{"}}"}}'
helm:
valueFiles:
- 'envs/{{ .Values.cluster_name}}/services-values.yaml'
values: |
global:
propertiesEnvName: {{"{{"}}propertiesEnvName{{"}}"}}
domain: {{"{{"}}domain{{"}}"}}
region: {{"{{"}}region{{"}}"}}
aws_account_id: {{"{{"}}aws_account_id{{"}}"}}
cluster_name: {{"{{"}}cluster_name{{"}}"}}
external_domain: {{"{{"}}external_domain{{"}}"}}
internal_domain: {{"{{"}}internal_domain{{"}}"}}
db_url: {{"{{"}}db_url{{"}}"}}
db_name: {{"{{"}}db_name{{"}}"}}
kafka_broker: {{"{{"}}kafka_broker{{"}}"}}
redis_uri: {{"{{"}}redis_uri{{"}}"}}
vpc_id: {{"{{"}}vpc_id{{"}}"}}
SpotinstRestrictScaleDown:
enabled: false
labels:
team: {{"{{"}}team{{"}}"}}
product: {{"{{"}}name{{"}}"}}
env:
INFLUXDB_HOST: {{"{{"}}propertiesEnvName{{"}}"}}-influxdb.tikal.com
LOG4J_FORMAT_MSG_NO_LOOKUPS: 'true'
ROOKOUT_ROOK_TAGS: {{"{{"}}propertiesEnvName{{"}}"}}
SPRING_PROFILES_ACTIVE: {{"{{"}}propertiesEnvName{{"}}"}}
CPTLS_ENV: {{"{{"}}propertiesEnvName{{"}}"}}
DATABASE_ADMIN: {{"{{"}}db_admin{{"}}"}}
destination:
server: https://kubernetes.default.svc
namespace: ns
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
prune: true
selfHeal: true
As we can see Applicationset resource get all its product names and environment values from the values file and generate an application resource in a matrix manner.
env-valuse.yaml
eks_user_access:
- orenext
- user1
- user2
aws_account_id: account122345
cluster_address: https://kubernetes.default.svc
cluster_name: auto3
cluster_region: eu-west-1
external_domain: example.com
internal_domain: example.is
domain_suffix: example.com
db_url: db-auto3.example.is
db_name: postgres
db_admin: tikalmarke1
kafka_broker: tp7.c9
redis_uri: redis://tikalredis-master.argocd
vpc_id: 'vpc-07b7ad41f697'
certificate: e86fffb2-255725f6fe6f
cluster_vpc_id: vpc-07b7adc04f697
nodegroup_iam_role: auto3-eks20220419093757900800000009
alb_ingress_role_arn: arn:aws:iam::account122345:role/alb-ingress-auto3
external_dns_role_arn: arn:aws:iam::account122345:role/external-dns_role_auto3
external_secrets_role_arn: arn:aws:iam::account122345:role/external-secrets_role_auto3
cluster_url: https://argocd-auto3.example.is
namespace_pod_quota_count: 200
aws_cluster_autoscaler:
enabled: false # Only enable when spot.io ocean is off
logzio:
enabled: false
kube2iam:
enabled: true
datadog:
enabled: false
api_key: 952b3deffce6afa2d389cfcacc4
apm:
enabled: false
argocd:
targetRevision: auto3
project: tikal
repoURL: git@github.com:tikal/tikal_helm_deploy.git
ingress:
cert_arn: arn:aws:acm:eu-west-1:607827849963:certificate/12f49709-d977-41c4-b
domain: example.is
sso:
enabled: false
clientid: e8cb602fb5b5904dd
clientsecret: n6wAj9BZ
tenant: 6cc0f90cac58fd
apps:
product1:
repo_url: git@github.com:tikal/product1-helm.git
enabled: true
version: edge
team: dev-team1
product2:
repo_url: git@github.com:tikal/product2-helm.git
enabled: true
version: edge
team: dev-team2
product3:
repo_url: git@github.com:tikal/product3-helm.git
enabled: true
version: edge
team: dev-team3
env-values.yaml file determines which products will be utilized in the environment, specifies all related values for the environment, and is also used by the application to generate the necessary manifests for each service with the appropriate values.
Helm template tool can assist verifying the output of application set.
helm template ./ -f ../env-values.yaml >test.yaml
test.yaml
---
# Source: products-set/templates/applicationset.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: tikal-products
namespace: argocd
spec:
generators:
- matrix:
generators:
- list:
elements:
- name: product1
version: edge
repo_url: git@github.com:tikal/product1-helm.git
team: dev-team1
- name: product2
version: edge
repo_url: git@github.com:tikal/product2-helm.git
team: dev-team2
- name: product3
version: edge
repo_url: git@github.com:tikal/product3-helm.git
team: dev-team3
- list:
elements:
- propertiesEnvName: auto3
domain: example.com
region: eu-west-1
cluster_name: auto3
aws_account_id: '0'
db_url: db-auto3.example.is
db_name: postgres
db_admin: tikalmarke1
kafka_broker: tp7.c9
external_domain: example.com
internal_domain: example.is
redis_uri: redis://tikalredis-master.argocd
vpc_id: 'vpc-07b7adc04f697'
template:
metadata:
name: 'tikal-{{name}}'
spec:
project: tikal
source:
path: ./
repoURL: '{{repo_url}}'
targetRevision: '{{version}}'
helm:
values: |
global:
propertiesEnvName: {{propertiesEnvName}}
domain: {{domain}}
region: {{region}}
aws_account_id: {{aws_account_id}}
cluster_name: {{cluster_name}}
external_domain: {{external_domain}}
internal_domain: {{internal_domain}}
db_url: {{db_url}}
db_name: {{db_name}}
kafka_broker: {{kafka_broker}}
redis_uri: {{redis_uri}}
vpc_id: {{vpc_id}}
SpotinstRestrictScaleDown:
enabled: false
labels:
team: {{team}}
product: {{name}}
env:
INFLUXDB_HOST: {{propertiesEnvName}}-influxdb.tikal.com
LOG4J_FORMAT_MSG_NO_LOOKUPS: 'true'
ROOKOUT_ROOK_TAGS: {{propertiesEnvName}}
SPRING_PROFILES_ACTIVE: {{propertiesEnvName}}
CPTLS_ENV: {{propertiesEnvName}}
DATABASE_ADMIN: {{db_admin}}
destination:
server: https://kubernetes.default.svc
namespace: ns
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
prune: true
selfHeal: true
Test.yaml specifies the details of three different products (product1, product2, and product3) and the values that should be used to deploy them.
The template section specifies the details of how the products should be deployed, including the source repository and target revision for each product and the values that should be passed to the Helm chart for each product.
getting there...
Thus far, we've become familiar with Argo and Applicationset resource , which greatly simplify the process of managing deployments in our cluster. Our ultimate goal, however, is to find a streamlined and automated method for deploying our products. This is where Helm charts come into play. By using Helm charts to define the desired state of our products in the cluster, we can leverage Argo CD to listen for changes in the Git repository containing these charts and automatically deploy those changes to our specified target environment.
Chart Umbrella and Chart-Library
As an efficiency hunter ,one of my main objectives in this project was to find a way for every team member to easily add or remove new services to our product, or even add entirely new products, without needing a lot of expertise in Helm or Kubernetes. Thankfully, the use of Helm Library charts has been a game-changer in this regard, making it much simpler for me and my teammates to make changes and additions to our products without getting bogged down in the details of Helm or Kubernetes.
In Helm, a Library chart is a collection of templates and helper functions that can be shared and reused among multiple charts. It is a way to organize and manage common templates and functions in a central location, making it easier to maintain and update your charts.
One of the main advantages of using Library charts is that they allow you to avoid duplication of code. Instead of copying and pasting the same templates and functions into multiple charts, you can define them once in a Library chart and then include them as needed in other charts. This makes it easier to maintain your charts and reduces the risk of errors, as you only need to update the templates and functions in one place.
Library charts came to our advantage in this project is that they make it easier to roll out changes across multiple charts. If you need to update a template or function that is used by multiple charts, you can simply make the change in the Library chart and all the charts that depend on it will automatically use the updated version. This can save a lot of time and effort, especially if you have a large number of charts that share common templates and functions.
Our deployment process holds multiple products ,each of which is made up of a group of micro-services called sun-charts. These sun-charts are located in the /services folder, and each product has its own umbrella chart that is composed of dependencies on these sun-charts.
The umbrella chart also includes a common config-map that is mounted to the file system of all of the sun-charts.
This approach allows you to organize your micro-services into logical groups and manage them as a single unit, while still allowing for flexibility and modularity. By using an umbrella chart to define the dependencies for each product, you can easily add or remove micro-services as needed and ensure that all of the necessary resources are properly configured. Additionally, the common config-map can be used to share configuration data across all of the micro-services within a product, making it easier to maintain and update your applications.
One of this umbrella chart repository would be presented like :
├── Chart.lock
├── Chart.yaml
├── README.md
├── apps
│ └── cms
│ └── application-env.properties
├── charts
│ ├── Tikal-main-chart-0.0.5.tgz
│ └── auth-service-1.0.0.tgz
├── envs
│ ├── auto3
│ │ └── services-values.yaml
│ ├── qa-euw1
│ │ └── services-values.yaml
│ └── qa-usw2
│ └── services-values.yaml
├── services
│ ├── auth-service
│ │ ├── Chart.yaml
│ │ ├── templates
│ │ │ ├── deployment.yaml
│ │ │ ├── hpa.yaml
│ │ │ ├── ingress.yaml
│ │ │ ├── service.yaml
│ │ │ └── serviceAccount.yaml
│ │ └── values.yaml
│ └── report-service
│ ├── Chart.yaml
│ ├── templates
│ │ ├── deployment.yaml
│ │ ├── hpa.yaml
│ │ ├── ingress.yaml
│ │ ├── service.yaml
│ │ └── serviceAccount.yaml
│ └── values.yaml
└── templates
└── cm.yaml
Chart.yaml
This is the Chart.yaml file for a Helm chart. It contains metadata about the chart, including its name, description, type, version, and app version.
The chart has three dependencies. The first dependency, tikal-main-chart, Its our Chart library holds all our common functions and configurations, its hosted in an AWS Container Registry (OCI) repository. The second and third dependencies, auth-service and report-service, are hosted in the local file system and are located in the services directory. It contains the source code for individual micro-services, organized into separate directories for each service. Each service has its own chart, with a Chart.yaml file that defines the chart's metadata and a templates directory that contains the resource definitions for the service.
The envs directory contains environment-specific values files that can be used to customize the configuration of the charts for different environments.
By this setup, each product is managed in its own separate repository and has its own version control history.
This separation allows the product code to be independent from the deployment code, enabling the ability to run any version of the product in any environment at any time, enabling the ability to run any version of the product in any environment at any time - what makes it easy to manage the drift of product versions between various environments. Additionally, multiple developers can work on the same product simultaneously, allowing for local testing and debugging before deploying to an environment.
Some extra last thoughts
Such a complex task requires many stages of trial and error. The road to a solution is full of obstacles, edge cases, and the whims of users who may not have the same level of knowledge as the author.
I am sure that this is not the only way and maybe not even the best way to carry out such a complex task, but in my opinion the key to success is to develop the architecture together with the people who are supposed to use and maintain the system, Teach them your way of thinking and understand if it converges with theirs. Put all the study cases in writing and find a solution that is somewhere in the middle between efficient, sophisticated and maintenance-friendly range.
I was happy to share the knowledge with you and I would love to hear your opinion.
Top comments (0)