DEV Community: Chirag Modi

AWS RDS Blue/Green Deployment for Aurora using Terraform

Chirag Modi — Thu, 27 Jun 2024 05:07:08 +0000

Why ?
If you are using AWS RDS and you have any of the following use cases.

Upgrade DB Major/Minor version without impacting LIVE production cluster with zero downtime.
Easily create a production-ready staging environment side by side to production to perform specific tests.
Test database changes in a separate staging environment without affecting the production cluster.
Implement and test new DB features on staging cluster before doing it on production.

Please take a look at this article for detailed information.

How did I upgrade Aurora Postgres RDS clusters in Production using Terraform

Chirag Modi — Mon, 20 Feb 2023 08:35:07 +0000

Step by Step Guide which includes

Planning required for the upgrade
Usage of terraform module and version dependency
Detailed steps for minor/major version upgrade
Problems faced and their solutions as part of upgrade
Best Practices for Database major version Upgrade

Take a look at detailed article - https://faun.pub/how-did-i-upgrade-aurora-postgres-rds-clusters-in-production-using-terraform-ff57aeb16873?source=friends_link&sk=26630c75421b705ad15b36f622feaddb

How to access AWS services from EKS

Chirag Modi — Sun, 27 Nov 2022 15:37:20 +0000

In-depth guide about different solutions to access AWS APIs from Kubernetes...

Kube2IAM
KIAM
IRSA (IAM Role for Service Account)

https://faun.pub/how-to-access-aws-services-from-eks-ab5fa003a1b6?source=friends_link&sk=ac00ca0fad44b7e7363d63c35dc4f0fc

Kubernetes Authentication in AWS EKS Using IAM Authenticator

Chirag Modi — Thu, 21 Jul 2022 04:00:52 +0000

How Kubernetes integrates with AWS IAM authenticator

There is a different way of authentication to Kubernetes based on different cloud provider implementations. I will specifically discuss authentication implemented by AWS EKS. This article should clarify the following questions.

How authentication works in EKS?
What is AWS IAM Authenticator for Kubernetes?
What does “aws eks get-token” do in KubeConfig to access EKS cluster?
What is “aws-auth” Configmap in EKS?
How can I add AWS users/roles to access EKS cluster?
How do AWS users/roles map to Kubernetes users and groups in EKS?
How do I generate KubeConfig for EKS cluster?
How do users get authorized to perform specific Kubernetes actions?

Take a look at this article to find detailed explanation with hands-on use case - https://betterprogramming.pub/kubernetes-authentication-in-aws-eks-using-iam-authenticator-de3a586e885c?source=friends_link&sk=fce36cdc7bbc7de2c9a39f73f013922b

Docker, Containers & Confusions

Chirag Modi — Sat, 26 Feb 2022 08:44:57 +0000

There is a lot of confusion about following common questions around Docker, Containers, Kubernetes and related technologies and tools.

What is Dockershim ?
What is a Container Runtime ?
Why did Kubernetes use Docker ?
What is the difference between Docker, Containerd, CRI-O ?
What is RunC ?
What are OCI and CRI specifications ?
What are different Container Runtimes available ?
Why did Kubernetes remove support for Docker ?
What alternative Container tools are available ?
What is the difference between Docker and Podman ?

Take a look at this article for clarifying details from basics to today's container world - https://faun.pub/docker-containers-confusions-2e2768530144?source=friends_link&sk=da2c8099b6db1bbe88e377da086ac111

A Cloud Migration Questionnaire

Chirag Modi — Mon, 19 Jul 2021 16:53:13 +0000

The questions you must ask your customers before migrating their on-premise workload to AWS Cloud.

Take a look at a detailed post to know why these questions are important and how it helps to lay the foundation for future design solutions and plan migrations better.

Kubernetes Helm Charts Testing

Chirag Modi — Fri, 11 Jun 2021 15:04:33 +0000

Tools to use for Helm Chart Testing during Development to Release

Introduction

Helm Chart is a package management software to write Kubernetes templates and package it as a chart with all its dependencies. A single chart can be used to deploy nginx, memcache or any full stack web application. You can deploy any application chart just by running the following command.

helm install my-release bitnami/nginx

Scope

This article does not cover detailed information about Helm Chart development instead Helm chart has very good documentation which you can go through to learn more about it.

I am going to cover how to test helm charts as part of development and what different types of testing tools can be used to test charts from unit tests to integration tests.

Helm Chart Development — Not a pleasant Experience

Helm Chart is written in go templates and writing those templates to render Kubernetes manifests is a painful experience. There is no good debugger support available and errors are clueless so sometimes you need to spend hours to fix minor indentation related issues. Helm provides a debug flag while rendering templates though it does not pinpoint the exact line where the error is in the code so it’s difficult to find issues. I hope to have better tools available in the future for helm chart development to make developers’ lives easy.

Photo by Ryan Snaadt on Unsplash

Are your chart templates correct ?

As mentioned earlier, Helm Chart templates use go templates so as part of development you need to be sure of syntactical errors so you don’t get last minute surprises when you release your chart.

Helm provides a lint command which finds and reports all these issues related to templates so you can execute this command frequently to find compile time errors as part of your development.

Here is a helm chart deployment template with errors.

Helm lint will report the following issues which were expected.

➜ mychart helm lint .
==> Linting .
[INFO] Chart.yaml: icon is recommended

[ERROR] templates/: parse error at (mychart/templates/deployment.yaml:19): function “Values” not defined

[ERROR] templates/: template: mychart/templates/deployment.yaml:7:16: executing “mychart/templates/deployment.yaml” at <include “namespace” .>: error calling include: template: no template “namespace” associated with template “gotpl”

Error: 1 chart(s) linted, 1 chart(s) failed

I know this is a very simple example but good enough for understating how lint works.

Validate against Kubernetes Manifests

Helm template is the command you can use to render/generate Kubernetes manifests/templates out of your helm chart templates.

There is a command Helm install to install/deploy charts to Kubernetes cluster. Internally, It first executes helm template command and then deploys generated template output to the cluster.

helm template . > deployment.yaml

Are your Kubernetes Manifests Valid ?

If you make mistakes while developing a chart then it might be possible that the generated Kubernetes Manifests generate errors when applied to the Kubernetes cluster but I want to know about those errors before deployment.

Kubeval is the tool to rescue. It’s a tool to validate your generated manifests against official Kubernetes specification and reports issues if any.

Can you spot any issue in this Deployment template ?

Run kubeval against this deployment manifest and look at the issues.

➜ mychart kubeval deployment.yaml

WARN — mychart/templates/deployment.yaml contains an invalid Deployment (myservice.nginx-deployment) — selector: selector is required

WARN — mychart/templates/deployment.yaml contains an invalid Deployment (myservice.nginx-deployment) — containerPort: containerPort is required

WARN — mychart/templates/deployment.yaml contains an invalid Deployment (myservice.nginx-deployment) — spec.replicas: Invalid type. Expected: [integer,null], given: string

##### This is the output if it was valid deployment #####
PASS — mychart/templates/deployment.yaml contains a valid Deployment (myservice.nginx-deployment)

Additionally you can specify Kubernetes version against which you want to validate generated templates using option* “ — kubernetes-version v1.20.4”*

Custom Validations against Kubernetes Manifests

Let’s say I have the following simple requirements.

Containers should not run as root.
Docker images should come from my org repository.

You can address this by implementing an admission controller in Kubernetes when resources get deployed to the cluster but would it not be nice if we can apply this custom validation before deployment ?

Conftest is a framework which allows you to write rules using OPA policies and run it against the Kubernetes manifests.

Run these custom rules against deployment manifests using conftest which will report issues based on configured rules.

➜ mychart conftest test — policy . deployment.yaml

FAIL — deployment.yaml — main — Containers must not run as root
FAIL — deployment.yaml — main — image ‘nginx’ doesn’t come from myorg.com repository

2 tests, 0 passed, 0 warnings, 2 failures, 0 exceptions

You can write any custom policies for all your resources which you can execute against Kubernetes manifests before deployment. That’s Pretty Cool.

Schema Validations for Custom Values

As I explained, you can verify Kubernetes manifests using Kubeval and Conftest tools but when you are creating a helm chart then you need to allow your users who are using the chart to add some custom values for any new feature. you need to validate that those custom values are in correct format to be consumed by the chart otherwise it will fail chart rendering which is a very difficult task to debug. how can we apply the first level of defense to make sure provided custom values are in correct format otherwise error it out with proper validation message.

Helm Chart provides Schema Validation feature for which you need to provide a schema file in a chart containing rules for all your custom values. It validates this schema validation first before executing any of these commands.

helm lint
helm template
helm install
helm upgrade

These are a few of the use cases using custom values.

Users should be able to specify memory and cpu requirements.
Some users want to specify different log locations for application logs.
Users want to supply environment *variables for the application container.*

Let’s implement the first use case where if the user specifies custom values for memory and cpu then it will take it otherwise set default values.

Here is values.schema.json file to validate against custom values.

This is custom-values.yaml which users can provide while consuming the chart.

As the user has provided wrong custom values, it should fail.

➜ mychart helm template . -f custom-values.yaml

Error: values don’t meet the specifications of the schema(s) in the following chart(s):
mychart:
- memory: Does not match pattern ‘^[0–9.]+[M|G]i$’
- cpu: Does not match pattern ‘^[0–9.]+m*$’

You can do any type of validation as long as it’s supported by json schema specifications though the only condition is you should have all json schema rules available in file named values.schema.json in your chart.

Unit Testing

Like any other programming language, Unit tests are the first which developers should consider in the early stage of development. I see a scarcity of good unit testing frameworks available for Helm Charts.

There is a unit test framework helm-unittest. It is a very nice framework for unit testing and lots of active deployment is going on so one should definitely go for it.

There is another hacky way of testing Helm charts which is a mix of unit tests and regression tests.

Idea is very simple. you need to add a binding file containing custom values for each new feature you are implementing in helm chart and generate fixture file out of it using helm template command. You need to commit binding and it’s fixture file in the repository. Now create a simple shell script to be executed in CI which will create a fixture file out of a binding file on the fly against your changes in chart and compare it with existing fixture file. you just fail the test if there is difference in fixture files then you need to either fix your helm chart or update the existing fixture file if it’s expected behavior.

Testing using fixtures is very helpful in code refactoring and also can be considered for unit tests while writing new features.

Integration Tests using Kubetest

Till now, we have tested helm chart templates and Kubernetes manifests using different tools but we did not verify anything by actually deploying Kubernetes manifests in the Kubernetes cluster.

Why do we need integration tests ?

I have mentioned a few of the use cases which can only be verified by deploying resources in a cluster.

I have mounted the volume as writable which I want to verify by creating a file in volume.
There are some custom resources which I want to verify.
I want to verify the health-check of the internal load balancer created as part of Kubernetes service creation.

There are many tools available but I really liked Kubetest which is a pytest plugin. Kubetest makes it easy to write integration tests by providing abstraction on top of Kubernetes client.

It provides many helper functions so you don’t need to write complex code using Kubernetes client unless it’s absolutely necessary.

It’s very intuitive and fun to write integration tests using Kubetest. I was tempted to skip not putting any example code here because it is self explanatory once you look at this example code.

Summary

I have covered the basics about helm charts, various tools we can use for different types of testing including unit test and integration tests during the lifecycle from development to releasing the helm charts.

Don’t miss out to look at my github I have used as an example through out the article if anything is not clear.

Hope you enjoyed it. Cheers !

References

Envelope Encryption using AWS CLI

Chirag Modi — Thu, 10 Sep 2020 12:44:01 +0000

Background

I have already posted article on Envelope Encryption and how it works. Please take a look at that article.

This article covers how can we encrypt/decrypt large amount of data by Envelope Encryption using AWS CLI.

Prerequisite

This hands-on exercise requires AWS account and AWS CLI. You can get more information about installation and configuration of AWS CLI from here

Hands-on Exercise

Generate Customer Master Key

We have AWS CLI setup by now so first step is to create AWS CMK (Customer Master key) using KMS. We have got our Customer Master Key which we will be using for encryption.

aws kms create-key --description "This key is used for envelope encryption"
output:
{
"KeyMetadata": {
    "AWSAccountId": "************",
    "KeyId": "21763c54-353e-4099-8027-************",
    "Arn": "arn:aws:kms:us-east-1:************:key/21763c54-353e-4099-8027-************",
    "CreationDate": "2020-09-10T14:59:44.359000+05:30",
    "Enabled": true,
    "Description": "This key is used for envelope encryption",
    "KeyUsage": "ENCRYPT_DECRYPT",
    "KeyState": "Enabled",
    "Origin": "AWS_KMS",
    "KeyManager": "CUSTOMER",
    "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
    "EncryptionAlgorithms": [
        "SYMMETRIC_DEFAULT"
    ]
}

1. Generate Data Key

Let's generate Data key using CMK we generated earlier. It returns Data Key (Plaintext) and Encrypted Data key (CiphertextBlob).

aws kms generate-data-key --key-id 21763c54-353e-4099-8027-************ --key-spec AES_256
Output:
{
"CiphertextBlob": "************IPQE9CgC3MLxxTR8lu/AFcM2axxufFf5mB81aqlukaAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM5FCtUAUdCHJ72PYNAgEQgDs7EwfgzL4g4/E48AJTKVEKJq8EsuEM6hAlcZ6XWw0AlYpfLyXD910NSd/LasDtI2YkIp7wSitlpdkVuw==",
    "Plaintext": "XIJVOItN8cc40n/H8/Wzbs/u+57/H5ERL/gi/hArqZI=",
    "KeyId": "arn:aws:kms:us-east-1:************:key/21763c54-353e-4099-8027-************"
}

2. Decode Base64 encoded Data Key

Keep note that Data Key and Encrypted Data key generated in previous step are Base64 encoded so we need to decode it first.

echo 'XIJVOItN8cc40n/H8/Wzbs/u+57/H5ERL/gi/hArqZI=' | base64 --decode > ~/plaintext_data_key.txt

3. Encrypt Data using Plaintext Data Key

We are encrypting actual data using Decoded plaintext data key using AES256 encryption.

echo "This is data I want to encrypt using plain data key"  | openssl enc -e -aes256 -k fileb:///Users/chirag/plaintext_data_key.txt > ~/encrypted_data.txt

4. Package Encrypted Data and Data key

We have now Encrypted Data and Encrypted Data Key which we can store together or separately on Data store. Make sure to store Encrypted Data key which will be required during decryption.

5. Remove Plaintext Data Key

We can remove Data key from system after Data encryption as it's sensitive information and we don't require it as we have stored Encrypted Data key so in future whenever required we can get back plaintext data key.

rm ~/plaintext_data_key.txt

6. Extract Data for Decryption

Let's we want our encoded data back so first need to extract Encrypted Data key we stored earlier and then Decode it as it was also Base64 encoded.

echo '************IPQE9CgC3MLxxTR8lu/AFcM2axxufFf5mB81aqlukaAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM5FCtUAUdCHJ72PYNAgEQgDs7EwfgzL4g4/E48AJTKVEKJq8EsuEM6hAlcZ6XWw0AlYpfLyXD910NSd/LasDtI2YkIp7wSitlpdkVuw==' | base64 --decode > ~/encrypted_data_key.txt

7. Decrypt Encrypted Plaintext Data Key

Once we get back Encrypted Data Key, we need to call Decrypt API to get Plaintext Data Key.

aws kms decrypt --ciphertext-blob fileb:///Users/chirag/encrypted_data_key.txt
Output:
{
"KeyId": "arn:aws:kms:us-east-1:************:key/21763c54-353e-4099-8027-************",
"Plaintext": "XIJVOItN8cc40n/H8/Wzbs/u+57/H5ERL/gi/hArqZI=",
"EncryptionAlgorithm": "SYMMETRIC_DEFAULT"
}

8. Decode Base64 encoded Plaintext Data Key

Again Decrypted Data Key we got is Base64 encoded so we need to decode it first.

echo 'XIJVOItN8cc40n/H8/Wzbs/u+57/H5ERL/gi/hArqZI=' | base64 --decode > ~/decrypted_plaintext_data_key.txt

9. Decrypt actual data using Plaintext Data Key

Take actual encrypted data and decrypt it using same AES256 algorithm and we got actual data back.

cat ~/encrypted_data.txt  | openssl enc -d -aes256 -k fileb:///Users/chirag/decrypted_plaintext_data_key.txt
Output:
This is data I want to encrypt using plain data key

10. Remove Plaintext Data Key

Cleanup Plaintext Data Key.

rm ~/decrypted_plaintext_data_key.txt

Note: I have masked all sensitive information here with "*"

Conclusion

So we have gone through full cycle of encryption (steps 1-5) and decryption (steps 6-10) making use of Envelope Encryption using AWS KMS.

This is how AWS internally performs Data encryption for large datasets in S3, EBS, RDS, etc.. when data encryption is enabled.

Thanks for joining me.

AWS KMS Envelope Encryption

Chirag Modi — Tue, 08 Sep 2020 19:21:33 +0000

Background and Introduction

Traditionally applications used to store security keys used for data encryption/decryption in application config files. Drawback of storing it in config files is risk involved if not stored and managed properly.

AWS Key Management Service (KMS) is fully managed service offering which AWS itself is using to encrypt/decrypt data at rest for different AWS services like S3, EBS, RDS, etc..

AWS KMS is highly available key management service to access, store, audit secret keys called CMKs (Customer Master Keys).

There are two types of CMKs (Customer Master Keys).

AWS Managed CMKs

AWS creates keys for each of its services which provides data encryption. These keys are managed by AWS and it's default CMK used to encrypt/decrypt data for particular service.
Customer does not have much control on it as it can only be viewed in KMS.

Customer Managed CMKs

Customer creates keys in KMS and has full control over management of keys like Audit log, Key rotation, Key deletion, etc...
Customer can also upload their own keys to KMS.

How Envelope Encryption works ?

AWS KMS key called CMK (Customer Master Key) is used to encrypt/decrypt data but there is limitation of it as it can encrypt data up to 4KB only so the question pops up then how it's using it for encryption of big datasets in S3, EBS, etc...

You got it right - Envelope Encryption.

Envelope encryption is the practice of encrypting plaintext data with a data key, and then encrypting the data key under another key.

Let's see how envelope encryption works but before encrypting any data customer needs to create one or more CMKs (Customer Master Keys) in AWS KMS.

Encryption

API request is sent to KMS to generate Data key using CMK.
KMS returns response with Plain Data key and Encrypted Data key (using CMK).
Data is encrypted using Plain Data key.
Plain Data key is removed from memory.
Encrypted Data and Encrypted Data Key is packaged together as envelope and stored.

Decryption

Encrypted Data key is extracted from envelope.
API request is sent to KMS using Encrypted Data key which has information about CMK to be used in KMS for decryption.
KMS returns response with Plain Data Key.
Encrypted Data is decrypted using Plain Data key.
Plain Data Key is removed from memory.

Summary

I tried to explain here what is Envelope Encryption in AWS KMS and how can we encrypt/decrypt large amount of data using envelope encryption method.

I am looking forward to put hands-on exercise on this in next article.

Thanks for joining me.

References

https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#enveloping

Everything is Replaceable

Chirag Modi — Sun, 06 Sep 2020 07:56:53 +0000

Many times we think that particular person is very important for organization or company and without him project can not go on as he has been working since long time and he knows everything.

That's just belief everyone has in mind

Everyone is important and same time truth is Everyone is replaceable

We often heard founders are fired from the company which they had built from start so Better everyone should keep in mind that they are replaceable as doing that will not hurt you when you get replaced.

Everyone is important for certain period of time

Sometimes person is very important for particular job, team or company till his work is over or when his replacement is found.

What people do to avoid they don't get replaced ?

Hide the information they know.
Don't treat other people well being egoistic of knowing everything.
Hinder the growth of company and other employees around.
Don't grow themselves by keeping them to only what they know.
They don't put important information on paper to avoid being used by others instead they are tried to remember everything.

How to make ourselves aware about this Truth ?

Just think that you won't be coming to office tomorrow due to you are sick or you left the job or whatsoever it may be.

What would happen to company, team or project ? who would be able to take it over and help the company or project ?

You should have proper plan put down from start and documented everything about tasks you are doing and decision you made like so that others can pick up where you left off. Information kept in your head will be forgotten one day and it does not benefit anyone.

This little practice helps lot to you and the organization.

As everything is documented, you don't need to explain other team members repeatedly.
Helps to grow yourselves so you can focus on other important things.
Keeps yourselves free from remembering everything.
Helps others to grow by delegating your duties to them.
Frees yourselves to take time off as other team member can take up your tasks.

I have analogy to compare this with software architecture.

Don't be like tightly coupled monolithic architecture but instead
loosely coupled microservice architecture

Rightly it has been said in Karmyog of Bhagvad Gita..

"Do Karma but keep yourself free from Karma instead of tied to it"

Cloud Design Principals

Chirag Modi — Tue, 01 Sep 2020 15:16:02 +0000

It's all about how you develop and deploy application instead of where you are deploying application be it public cloud or private data center. Application designed in this way can best make use of offering provided by cloud. First step in that direction is building micro-service based applications and run them in containerized and orchestrated platform like Kubernetes.

Following are the design principals one should consider while creating Cloud applications.

Microservices

Applications are created as micro-service so one can make use of best language, frameworks and tools suitable for different applications. Let's say we can build real-time applications using Nodejs, REST applications using Spring-Boot or Dropwizard and ML applications using python.

Containerization and Orchestration

Containerized application requires less resources then VMs and starts up in fraction of seconds. Containers packages applications into small, lightweight execution environment which shares host operating system. Containers helps isolating different micro services running in same host operating system. Docker has been used heavily for containerization.
Containerized application applications can be deployed using orchestration platform like Kubernetes which helps in container management, application deployment, scaling which has become standard for cloud application deployment.

Immutable and Disposable Resources

Consider resources as disposable as opposed to traditional way of thinking of physical fixed servers in data center where application used to be running.

Consider resources as immutable infrastructure where once application is deployed, it never gets updated instead new resources are provisioned for each deployment which can greatly improves consistency of application deployment/rollback.

Leverage Managed Services

There are many service offered by cloud which can be used by applications so instead of investing resources in setting up that infrastructure in managing it, one can leverage managed services offered by different cloud providers.

Scared of vendor lock in ?

It's trade off operation cost of managing it or using managed services. You can always use many open source managed services offered by cloud and for others you have to decide based on operational costs.

Scalability and Elasticity

Scalability is the ability to scale application without changing design and it achieved by scaling up and down whenever required. Scalability leads to other principals like Stateless applications and Disposable resources.

Elasticity is the ability to use resource dynamically and all cloud providers offer it as pay as you go model. It solves problem of over-provisioning of resources for applications deployed in traditional data center.

Stateless

Stateless applications maintain states outside of application in database or other external entity so nodes can be added dynamically and removed and deployed applications without worrying about state. Application components also should be stateless as stateless components can be easily scaled, repaired, rolled-back and load balanced. Scalability can be achieved easily by designing stateless applications.

Loose Coupling

Loosely Coupled architecture reduces inter dependency between application components so one failed component does not impact whole application.

Service Discovery

Services registers themselves and discovered automatically by other services
Service endpoints does not need to be hard coded in applications.

Asynchronous Communication

Decoupled components does not interact each other directly
Communication between components happen using various message brokers like ActiveMq, SQS, etc..

Fault Tolerant and Resiliency

Application is fault tolerant means it should continue functioning in-spite of fault detected in one of the application component. While Resiliency is about ability to recover from failures to the operation state when it was before failure so it's all about responding to failures without any downtime and data loss.

Resiliency should be at core of any architecture and you should practice it from start when designing any application.

Resilient applications offers High availability and Disaster recovery using load balanced clusters, multi-region deployments, replication, continuous monitoring.

Different ways to make application resilient:

Load balancing to avoid single point of failure
Retry intermittent failures
Circuit breaker to avoid cascading failures
Request throttling to avoid unavailability and DDOS attacks
Distributed transaction rollback using SAGA pattern

Security

Application component should communicate with each other using well defined authentication method without thinking whether it's internal service or external service to avoid any security vulnerability

Automation

Last but very very important is we should strive for automation in every part of application development, build, deployment, monitoring, etc ...

Infrastructure Automation

Infrastructure automation can be achieved by provisioning infrastructure using tools like Terraform, Serverless, Cloudformation etc..

CI/CD Automation
CI/CD pipeline greatly reduces manual efforts and help in automating everything from build, test, package and deployment. There are various tools like Jenkins, Spinnaker, Code pipeline, etc...

Auto Scaling
Automating scale up helps application availability during peak load and scale down reduces cost when application are not used heavily. Sometimes you can scale down to 0 for internal applications which are used rarely to further reduce costs.

Monitoring
Centralized logging and monitoring greatly helps in maintaining services in healthy condition and it should be automated using different tooling so if issue can be detected quickly and fixed.

Backup
Automated backup solution in place so data can be recovered quickly without much loss of data.