DEV Community: Poojan Mehta

No More Hardcoded Secrets: Automatic Database Credential Rotation with Vault, AKS and Postgres🔐

Poojan Mehta — Mon, 17 Feb 2025 02:19:15 +0000

In Part 1 of this series, we set up HashiCorp Vault in an AKS cluster using Terraform, configured ExternalSecrets, and demonstrated how to fetch secrets from Vault's KV engine into Kubernetes.

Now, let's take it a step further. Static credentials are risky—they can be leaked, misused, or forgotten.🤯To mitigate this, Vault provides Dynamic Secrets, allowing credentials to be generated on-demand, time-bound, and auto-revoked after expiration.

✅In this article, we’ll deploy PostgreSQL in our AKS cluster using Helm, explore Vault's database secrets engine to generate short-lived credentials, set externalSecrets and vaultDynamicSecrets to natively sync those credentials in the cluster.

Let's jump right in.!🚀

GIF Credit

👉🏻 1) Setup PostgreSQL using Helm:

We will use the bitnami helm chart, and use the default parameters for the sake of simplicity. Fine-tuning parameters can be added in a separate values.yaml file and applied with the installation command.

helm install my-release oci://registry-1.docker.io/bitnamicharts/postgresql --set hostNetwork=true

This will generate the default Postgres user password and store it as a secret in the cluster. Use the below-mentioned command to fetch the password in decoded format.

kubectl get secret --namespace default my-release-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d

📝Note: The configured password will be ignored on a new installation in case the previous PostgreSQL release was deleted through the helm command. In that case, old PVC will have an old password, and setting it through helm won't take effect. Deleting persistent volumes (PVs) will solve the issue.

Up next, let's create a non-root user in the database which we will be using for all the interactions between Postgres and Vault. Since we are aiming at credential rotations, keeping the root user intact will ensure we never lose access to the database.

# execute interactive bash shell with the database statefulset
kubectl exec -it my-release-postgresql-0 -- /bin/bash

# login to the database 
psql -h 127.0.0.1 -U postgres -p 5432 -W

#Create user with password
CREATE USER vault WITH PASSWORD 'vault123';

# Grant privileges
GRANT ALL PRIVILEGES ON DATABASE postgres TO vault;
ALTER USER vault WITH SUPERUSER;

We can see the vault user is created

👉🏻 2) Create Database Secret Engine from Vault UI:

The Vault database secrets engine dynamically generates database credentials based on configured roles, eliminating the need to hardcode credentials. It supports various databases through plugins and allows for both dynamic and static roles.

Vault's leasing mechanism assigns a Time To Live (TTL) to dynamic secrets and tokens, ensuring they are valid for a specified period. Once the TTL expires, Vault can revoke the secret or token, necessitating periodic lease renewals by clients to maintain access.

Navigate to the Vault UI and create a new secrets engine.

Give an appropriate name and adjust the default lease TTL and Max lease TTL if needed.

Next, select the database plugin as PostgreSQL and pass the connection URL postgresql://{{username}}:{{password}}@localhost:5432/database-name

💡Here, Postgresql is the connection method, username and password of the Vault user created in the previous step, and IP address of the ClusterIP service created while Postgres installation followed by the database as Postgres.

The reason for using private clusterIP is that the database is running the same cluster and can be accessed through the service connected with the statefulset.

To further enhance the security, TLS configuration can also be added in this step.

🤐Let's configure the role for this connection. Dynamic roles generate unique, time-limited database credentials for each service request. In contrast, static roles map Vault roles to existing database usernames, and Vault manages automatic password rotation for these static credentials.

⚠️ Static roles are not recommended for root credentials as rotating them will no longer keep the authentication between Vault and Postgres.

This is the main part. The role is attached to the database connection and it generates a dynamic username and password with 10 minutes of validity (Default TTL) and max TTL of 1 day.

Image Source

We have kept the validity to only 10 minutes to verify the rotation. Based on the sensitivity of the application, the TTL duration can be adjusted.

Here, creation statement and revocation statement consist of SQL queries which would be performed when new credential request is triggered. In our example, it will create a database role with password and expiry and grant select privileges on the given table(s). At the time of expiry, it will revoke all the permissions and drop the user.

As shown in the screenshot below, on clicking "Generate Credentials" from the vault UI, the short-lived credentials are displayed one-time.

Now let's log in with this user and verify if the granted permissions got applied or not. As shown below, it shows temporary users with their expiry dates added as attributes, and any other operation than select would not work.

Okay, so far so good. But how to natively fetch these secrets in the cluster?🤔🤔 There comes the **vaultDynamicSecrets** in the picture.

👉🏻 3) Configure VaultDynamicSecret resource:

Since we already have externalSecrets set up to natively fetch credentials from external secret stores like Vault, integrating these dynamic credentials with the same would make it more flexible to use with other Kubernetes resources through native secrets.

By default, the externalSecrets resource we used in part 1 only supports KV (Key-value) secret engine. Here, we will be using generators which allow to generate values from the given source. Generators can be defined as a custom resource and re-used across different ExternalSecrets.

The VaultDynamicSecret generator specifically integrates with HashiCorp Vault to retrieve dynamic secrets directly from Vault's database secrets engine.

apiVersion: generators.external-secrets.io/v1alpha1
kind: VaultDynamicSecret
metadata:
  name: db-credentials
  namespace: external-secrets
spec:
  path: "/database/creds/dynamicrole"
  method: "GET"
  provider:
    server: "<VAULT SERVER ADDRESS>"
    auth:
        tokenSecretRef:
          name: "vault-token"
          namespace: "external-secrets"
          key: "token"

This resource definition creates a VaultDynamicSecret, which dynamically retrieves database credentials from HashiCorp Vault.

The credentials are fetched from the specified path (/database/creds/dynamicrole) in Vault using the GET method.
The Vault server address and authentication are provided, with the token stored in a Kubernetes secret named vault-token within the external-secrets namespace.

This is one side of the bridge. On the next side, let's create externalSecret resource to connect with vaultDynamicSecrets resource and fetch the short-lived credentials natively.

Here, a native Kubernetes secret named db-credentials will be created keeping the refresh interval to 1 hour and referencing data source from the generator we created in the last step.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: "dynamic-external-secret"
  namespace: external-secrets
spec:
  refreshInterval: "1h"
  target:
    name: db-credentials
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: VaultDynamicSecret
        name: "db-credentials"

Ready.! Let's deploy the resources :)

Image Credits

Let's verify the changes by describing the secret resource in the namespace external-secrets. Two keys are stored with a username and password in it. This action is supported by the creation statement declared in the role attached to the database secrets engine.

How can we move ahead without verifying the rotation?😮‍💨 Run the below commands to fetch the decoded values of the secrets at intervals of 10 minutes and see the changes in action.

# Fetch username from the secret
kubectl get secret db-credentials -n external-secrets -o jsonpath="{.data.username}" | base64 --decode; echo

# Fetch password from the secret
kubectl get secret db-credentials -n external-secrets -o jsonpath="{.data.password}" | base64 --decode; echo

GIF Credit

And it worked. We got rotated credentials with fine-grained privileges which expire at every TTL duration.

If you are still reading, thanks for making it to the end.🥹 We've taken secrets management to the next level by introducing dynamic database credentials with Vault! Instead of relying on boring static, long-lived passwords, we now have ephemeral credentials that are automatic, time-bound with built-in rotation and seamlessly injected into Kubernetes pods via ExternalSecrets.! Wohoooo 🥳

This means no more hardcoded database passwords, reduced security risks from leaked credentials, and no manual rotation headaches—everything is automated! 🚀

With this, our Kubernetes workloads are now safer, more scalable, and fully automated in handling sensitive data.

💡 Let’s Connect!

Thanks for reading! 🎉 I hope this guide helped you understand dynamic secrets and automated credential rotation in Kubernetes. If you have feedback, suggestions, or want to discuss more, feel free to reach out! 💬

Find me on LinkedIn and check out the full project on GitHub. Let’s build smarter, more secure cloud solutions together! 🚀

Secrets Management 101: A technical approach with AKS, Terraform, and Vault

Poojan Mehta — Tue, 14 Jan 2025 00:50:33 +0000

🌟 Welcome, fellow DevOps enthusiasts! 🌟

Welcome to our journey into Kubernetes secrets management! 🚀 In this two-part series, we'll delve into the essentials of securely managing secrets across different Kubernetes clusters.
In this first half, we'll walk you through setting up an Azure Kubernetes Service (AKS) cluster using Terraform, deploying HashiCorp Vault, and utilizing ExternalSecrets to fetch secrets within the cluster. By the end of this article, you'll have a robust setup that ensures your secrets are both secure 🔒 and easily accessible within your AKS environment.

📝 Pre-requisites: Before we dive into the technical details, make sure you have the following prerequisites in place:

An active Azure subscription and Azure CLI installed
Terraform CLI, Kubectl CLI, and Helm CLI installed Ensure you have these tools ready to follow along with the steps outlined in this guide. Let's get started! 🚀

Image Source

Step 1) Setting up AKS with Terraform

Let's understand the terraform code for AKS. I have used the modular approach and the full code can be found in the repository mentioned below.

Provider Configuration:

data "azurerm_client_config" "current" {}

This data source retrieves metadata about the authenticated Azure client, such as tenant and object IDs, which are crucial for configuring role-based access and other resources.

Azure Resource Group:

resource "azurerm_resource_group" "rg" {
  location = var.resource_group_location
  name     = var.rg_name
}

Defines a resource group using variables for location and name, making the configuration flexible and environment-agnostic.

Azure Kubernetes Cluster:

resource "azurerm_kubernetes_cluster" "k8s" {
  location            = azurerm_resource_group.rg.location
  name                = var.cluster_name
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = var.dns_prefix
  kubernetes_version  = var.kubernetes_version

  azure_active_directory_role_based_access_control {
    azure_rbac_enabled = true
    tenant_id          = data.azurerm_client_config.current.tenant_id
   }

  identity {
    type = "SystemAssigned"
  }

  default_node_pool {
    name       = "agentpool"
    vm_size    = var.vm_size
    node_count = var.node_count
  }

  network_profile {
    network_plugin    = "azure"
    load_balancer_sku = "standard"
  }
}

For defining the Kubernetes cluster, we have used the azurerm_kubernetes_cluster resource and added the location, name, resource group, and Kubernetes version details.

🔐Security is a crucial aspect, and we integrate Azure Active Directory (AAD) for role-based access control (RBAC). By enabling Azure AD RBAC, we ensure that access to the Kubernetes cluster is managed using Azure AD. The tenant ID is referenced from the current Azure client configuration, linking the cluster to the correct Azure AD tenant.

Rest, nodepool, load balancer, and network plugin fields are kept default. In the final part, we assign the cluster-admin role to the current Azure client-authenticated user who gets administrative privileges.

Role Assignment:

resource "azurerm_role_assignment" "aks" {
  scope                = azurerm_kubernetes_cluster.k8s.id
  role_definition_name = "Azure Kubernetes Service RBAC Cluster Admin"
  principal_id         = data.azurerm_client_config.current.object_id
}

Since the code is ready with the minimal required configuration, it is now time to deploy it using Terraform cli.

Initialize Terraform and set up the environment, downloading necessary plugins and configuring the backend.

terraform init

Validate Configuration and detect if there are any semantic errors.

terraform validate

Plan the Deployment, and generate the change in the infrastructure with the current defined state mentioned in the code.

terraform plan

Ready to go.! Run terraform apply with the subscription id inline variable followed by auto-approve argument to bypass the runtime confirmation. The benefit of adding this inline variable is to eliminate the risk of exposing sensitive details like subscription ID in the terraform

terraform apply --var=subsciption-id="pass-your-id" --auto-approve

Applies the planned changes to reach the desired state, creating and configuring resources.

Authenticating to the AKS Cluster 🔐
Once your AKS cluster is up and running, you'll need to authenticate to it in order to manage and deploy applications.
Use the az aks get-credentials command to download the kubeconfig file for your AKS cluster. This file allows Kubectl to authenticate to your cluster.

az aks get-credentials --resource-group <resource-group-name> --name <cluster-name>

This command merges the cluster's kubeconfig with your existing kubeconfig file (or creates a new one if it doesn't exist).

Verify Connection: To verify that you are authenticated and can access the cluster, use the kubectl get nodes command:

kubectl get nodes

The screenshot above confirms the cluster-admin role assignment to the logged-in tenant.

Step 2) Setting Up HashiCorp Vault on AKS🕵🏻‍♂️

HashiCorp Vault is a powerful tool for managing secrets and protecting sensitive data. It allows you to store and tightly control access to tokens, passwords, certificates, and encryption keys. Vault's robust API and comprehensive audit logs make it a vital part of any security infrastructure. By using Vault, you ensure that your secrets are managed securely, minimizing the risk of unauthorized access.

We'll deploy HashiCorp Vault using Helm, a package manager for Kubernetes. This deployment will include a custom value.yaml file to enable the Vault UI with a LoadBalancer service, making it accessible from outside the cluster.

-> Create a values.yaml file to override the default configuration:

ui:
  enabled: true
service:
  type: LoadBalancer
  externalPort: 8200

-> Install Vault with Helm:
First, add the HashiCorp Helm repository and update it:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

-> Next, deploy Vault using the custom values.yaml file:

helm install vault hashicorp/vault -f values.yaml

This command deploys Vault into your Kubernetes cluster with the settings defined in values.yaml. Verify the deployed version and resources in the screenshots below.

After deployment, the Vault starts in a sealed state, meaning it can't perform any operations until it's unsealed. Unsealing is a critical security feature that ensures the Vault server remains secure. To unseal Vault, you'll need to use the kubectl exec command to run the unseal process within the Vault pod.

First, initialize Vault to make it operational:

kubectl exec vault-0 -- vault operator init

Once Vault is initialized, you can access the Vault UI through the LoadBalancer service.

-> Get the LoadBalancer IP:

kubectl get svc

Look for the service named "vault" and note the external IP address.

-> Access the UI:
Open your browser and navigate to http://:8200. You'll be prompted to pass 3 unseal keys which were displayed as output for the vault operator init command we ran earlier.

-> Log in to Vault:
Use the root token to log in. The root token was generated during the Vault initialization process.

-> Creating a New Secret Engine:
Vault uses secret engines to store and manage secrets. Let's create a new KV (key-value) secret engine to store our secrets. Navigate to the secret engine option in the UI and create one.

Once completed, store key-value pairs which are supposed to be sensitive data injected into the pods as secret.

(p.s: for simplicity of the demo, we kept other options in the secret engine default and performed the demo through UI, but the same can be done via CLI and API actions)

Step 3) Installing and Configuring ExternalSecrets on AKS

GIF Source

What are ExternalSecrets and Why Do We Need Them?

ExternalSecrets provides a Kubernetes-native way to fetch secrets from external secret management systems like HashiCorp Vault. It helps in maintaining a centralized secrets store, enabling easier management and more secure access controls.

Two primary resources used in ExternalSecrets are ClusterSecretStore and SecretStore. Here's the difference:

ClusterSecretStore: A cluster-wide resource that defines how to connect to an external secret store like HashiCorp Vault. It is accessible across multiple namespaces.

SecretStore: A namespace-scoped resource for defining the connection to an external secret store. It is specific to a single namespace.

Using ClusterSecretStore allows multiple namespaces to share the same external secret store configuration, making it efficient for larger deployments.

Deploying ExternalSecrets Using Helm

-> Install ExternalSecrets with Helm:
First, add the ExternalSecrets Helm repository and update it:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

-> Next, deploy ExternalSecrets:

helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace --installCRDs=true

This command deploys ExternalSecrets into your Kubernetes cluster in a new namespace, and it will install the CRDs as well.

-> Creating ClusterSecretStore Resource
We'll create a ClusterSecretStore resource that defines the connection to HashiCorp Vault.

💡We have used vault's root authentication token, but one can create a non-root token and use it. Don't forget to base64 encode your token before passing in the secret.

Here’s the YAML configuration:

apiVersion: external-secrets.io/v1alpha1
kind: ClusterSecretStore
metadata:
  name: example
spec:
  provider:
    vault:
      server: "http://PUBLIC_IP_OF_VAULT:8200"
      path: "secret"
      version: "v2"
      namespace: "external-secrets"
      auth:
        tokenSecretRef:
          name: "vault-token"
          namespace: "external-secrets"
          key: "token"

---
apiVersion: v1
kind: Secret
metadata:
  name: vault-token
  namespace: external-secrets
data:
  token: ROOT TOKEN IN BASE64 ENCODED FORMAT # "root"

Save this as clustersecretstore.yaml and apply it:

kubectl apply -f clustersecretstore.yaml

-> Creating ExternalSecret Resource
Now we'll create an ExternalSecret resource to fetch secrets from Vault. Here, Here’s the YAML configuration:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: vault-example
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: example
    kind: ClusterSecretStore
  target:
    name: application-sync
  data: 
    - secretKey: PASSWORD
      remoteRef:
         key: secret
         property: PASSWORD

Save this as externalsecret.yaml and apply it:

kubectl apply -f externalsecret.yaml

Verifying the Configuration

To ensure everything is set up correctly, we'll use several kubectl commands.

-> Verify the ExternalSecret Resource:

kubectl get externalsecrets
kubectl get clustersecretstore

-> Verify the Creation of the Secret Resource:

kubectl describe secret application-sync

-> Fetch the Value of the Secret:

kubectl get secret my-secret -o jsonpath="{.data.my-key}" | base64 --decode

This command fetches the value of the secret, decoding it from Base64. It is the same value we stored inside the secret engine in the vault UI.

In short, Kubernetes never knew about this secret in the Hashicorp Vault, but we used externalSecrets to bridge this gap and make those secrets run like native Kubernetes secrets. From here, we can inject this in pods as regular secrets either as an environmental variable, or volume, or secret reference.

Alas.!🤓 We finally made it to the end of the first half of the demo. We've set up an AKS cluster with Terraform, deployed HashiCorp Vault for secure secrets, and integrated ExternalSecrets to fetch them. Your secrets are now safely managed in Kubernetes.
But the adventure isn't over! Stay tuned for more exciting discoveries!

Find me on LinkedIn and check out the full project on GitHub. Let’s build smarter, more secure cloud solutions together! 🚀

Deploy Single-Node RedHat OpenShift 4.9 Cluster on AWS

Poojan Mehta — Tue, 12 Jul 2022 16:02:03 +0000

In this article, I’ll guide you through the installation steps in order to deploy OpenShift 4.9 on a single node EC2 server.

Before Jumping into the details, let’s understand what OpenShift is actually. OpenShift is a enterprise grade container orchestration tool by RedHat built on Kubernetes. This product provides much more than core Kubernetes capabilities and covers almost every phase of the container management lifecycle. In this article, we will install the Opensource flavour of the OpenShift container platform 4.9. Basic understanding of containers and Kubernetes is suggested to learn and work on OpenShift. We will use the OpenSource version of the OpenShift container platform which is also known as OKD project. Also, we will use the Full-stack Automation approach to install the single-node cluster over AWS EC2 service.

Basically, the Full-Stack Automation approach will allow us to utilize the power and availability of the existing cloud services and club multiple things together in a Terraform script and deploy a cluster-based solution with just a few clicks. Also, the user can customize the cluster configuration and set a few parameters within the script to get the customized cluster solution. As we consider OpenShift as an enterprise version of Kubernetes, it comes up with more features and pre-requisites. One of them is a public domain if you’re looking forward to deploy the solution on a public cloud provider and planning to deploy workloads that are publicly accessible.

I’ve already purchased a domain and maintaining a DNS Hosted Zone within Route53 Service which implies the pre-requisite is fulfilled. If you’ve purchased a domain from other provider than Route53, you should consider creating a Hosted zone and alter the nameservers in the original domain provider site.

We will provision a single-node openshift cluster via an EC2 instance as a client node. The client node will simply work as an intermediate between the cluster and the admin and will enable the communication between them.

Step: 1) Create SSH key in the client node

→ The first step is to create one ssh key in the client node to further bind it with the full stack automation script. The SSH key is created in case we need to login inside the master node in future and perform some cluster operations. Follow the steps below to create one.

ssh-keygen

eval “$(ssh-agent -s)”

ssh-add id_rsa

Give appropriate name to the SSH key (optional) and click enter. You can verify the key has been created by going to the ~/.ssh folder. With the ssh-add command, we are adding the identity of the created keyfile with the system.

Generate and add SSH key

Step:2) Configure AWS Profile for the programmatic access

→ In this step, we will configure one AWS profile with Access and Secret keys so that the Full stack automation script can make the API calls to the AWS console on our behalf and preform infrastructure provisioning tasks.

aws configure — — profile dev

export AWS_PROFILE=dev

Configure AWS Profile

After running this command, pass the AWS access and secret keys🔒 and then run the export command to set the current profile for the shell.

Step:3) Install OpenShift binaries from GitHub

→ Considering that we are using the OpenSource version of the OpenShift, in order to get the installation binaries, we will take reference of the officialGitHub Repo of the OKD Project. In this demo we are downloading openshift v4.9 as zip files. Verify the same with the screenshot below. Install the binaries and extract the zip files using the tar command.

tar -xvf openshift-client-linux-4.9.0–0.okd-2022–02–12–140851.tar.gz

tar -xvf openshift-install-linux-4.9.0–0.okd-2022–02–12–140851.tar.gz

OpenShift installer zip file

Step:4) Create a installation config YAML file using the script

→ By using the OpenShift install executable file with the required arguments, we will now create one configuration file in YAML format. Run the below command to initiate the config file creation and select the respective options from the dropdown menu.

./openshift-install create install-config — dir=.

→Don’t forget to select the SSH key we created in Step 1. In the next option, we have to select the resource provider. This Full-Stack automation approach supports almost all major cloud providers and we will select AWS in this case.

→ Afterwards, the script expects AWS region and the domain name for the cluster which I’ve given ap-south-1 and my own domain having a hosted zone in route53. In addition, we’ve to provide a unique name of the cluster which will be binded with the base domain and eventually be provided as a publicly routed URL for the console or the deployed workloads over the cluster.

→ The last remained field is Pull Secret. Earlier, OpenShift 4.x Versions were not opensource. Hence, back then, in order to pull the OpenShift images from the registry, the user had to authenticate with a unique pull secret obtained from their respective RedHat developer account. The option became obsolete after the consideration of the OKD project under OpenSource guidelines but still the script seeks the pull secret. A workaround to bypass this authentication is mentioned in this GitHub Issue . I’ve followed the same and passed null authentication secret which is {“auths”:{“fake”:{“auth”:”aWQ6cGFzcwo=”}}}.

→After feeding all the inputs, a config file named install-config.yml will be created in the target directory. Here we can see the current directory as the target dir.

OpenShift Install script

Step:5) Customize the existing file and edit the number of nodes

→ Open the file editor and change the node number for Worker from 3 to 0. Also, change the node number for master node from 3 to 1. This concludes that the provisioned cluster will operate on a single node which means the same node will work as both master and slave.

→ We are keeping all other fields as default much there is much more to customize based on requirements. You can refer the article for detailed explanation on customized installation.

Edited YAML file

Step:6) Run the create cluster command and initiate the installation

→ We’ve fulfilled the pre-requisites in order to launch our single-node OpenShift v4.9 over an EC2 instance. The script is edited with respect to our current requirement. Now let’s run the cluster creation command.

→ Here we are giving reference of the created config file as the — dir argument. So, the create cluster command will take the inputs from the config file and provision the single node cluster by using the access and secret keys provided by the admin.

./openshift-install create cluster — dir=.

2 nodes provisioned. One master and one bootstrap.

→ The installation will take somewhere around 30–40 mins. Hold tight and grab a coffee until then.

→ Also, the installation script will launch an intermediate node for bootstrapping. This node is responsible for configuring the cluster and making all the cluster components up and running. The bootstrap node will be destroyed after the master node is working and in healthy state.

→During the installation process, the cluster will perform internal health checks to make sure each components are up and running in a operatable state.

→ Also, the installation will come up with a default cluster user named Kubeadmin and the corresponding password (~/auth/kubeadmin-password). We can use this user and password to login in the cluster and perform the operations. Also, a publically routable URL for the OpenShift Web UI will be printed in the shell output.

post Installation output

→ Now, export the Kubeconfig file so that the oc client will be able to fetch the cluster information and context from that file and perform cluster operations to the mentioned endpoints.

export KUBECONFIG=/root/okd/auth/kubeconfig

Step:7) Copy the OC binaries to the executable directory

→ In order to run the OC client from the command line, we have to first copy the client binaries to the executable location in the linux system. Use the below command for the same.

cp oc /usr/bin/

cp kubectl /usr/bin/

oc client

Finally, all the pre-requisites and the cluster installation process is completed and now we have a running single-node OpenShift v4.9 working as expected. We can verify the same by running the status command.

oc status

→Also, with the printed console URL, we can login inside the WebUi using the temporary Kubeadmin user and explore the console.

Done and Dusted. The OpenShift installation is completed. That’s it from my side for today. There is a lot more to learn in the ecosystem.

THANKS, A LOT FOR READING THIS SO ATTENTIVELY

I’ll be grateful to have connections like you on Linkedln ** ** 🧑‍💼

How to Host WordPress with AWS RDS as Database?

Poojan Mehta — Tue, 02 Feb 2021 06:20:24 +0000

In this article, I’ll show how to set up WordPress(frontend) and AWS RDS database step by step.

Description:

🔅 Create an AWS EC2 instance and configure it with Apache web server and compatible PHP and MYSQL environment

🔅 Download and configure PHP-based application name “WordPress”.

🔅 WordPress stores data at the backend in the MySQL Database Server. Therefore, you need to set up a MySQL server using the AWS RDS service.

🔅 Provide the endpoint string to the WordPress application to connect with RDS and make it work.

First things first, let’s create an EC2 instance from the console and configure the Apache webserver.

I’ve used RHEL8 AMI throughout and performed the task by following all the dependencies. Manually performed all the steps but the same setup can be(should be) done using IaC tools like Terraform.

→SSH to the instance and run the following commands to install httpd, PHP, and MySQL(with dependencies) and start the services:

yum module install httpd php mysql

yum install php-mysqlnd

systemctl start httpd

httpd, PHP, MySQL installed and service started for httpd

→ Install the latest version of WordPress(written in PHP)and copy the same in the document root of the httpd(/var/www/html) in the system:

[https://wordpress.org/latest.tar.gz](https://wordpress.org/latest.tar.gz) --output wordpress.tar.gz //install wordpress

tar xf wordpress.tar.gz //extract the downloaded file

cp -rv wordpress /var/www/html // copy the content in doc. root

->If all the above steps are successful then we can access the front end with public-ip/wordpress and the o/p will look like this:

Default page after installation

Leave this till here and let’s configure the AWS RDS database from the console

Selected MySQL flavor

→AWS RDS offers various flavors of relational databases that include MySQL, MariaDB, PostgreSQL, Oracle, Microsoft SQL Server, and Amazon Aurora. RDS is a fully managed service from AWS, so users have to only focus on data and databases.

After selecting the database engine and creation method, set the instance name and admin user and password.

Now select compute and storage unit for the DB instance as per requirement. I have used the free-tier RDS method, so t2.micro flavor and default storage type is selected.

As a next step, select network set up for the instance. I have taken default VPC and subnet. Usually, the database is running in the backend of our application and is not meant for public access. So, it’s suggested to disable public access of RDS. As a security group, good practice to open the specific port and allow only admin IP.

Still, there are additional options that give more customization and post-launch actions. Totally depends on the requirement which facilities you need. I’ve left the else part default. Click on Create Database to launch the RDS.

It takes around 10 mins to launch the DB cluster. As this is a fully managed service, the user need not configure anything.

→Let’s connect to the RDS instance with the unique endpoint. I am using an ec2 instance within the vpc of RDS. So, being in the same network, we can connect.

Use the syntax mysql -h -u -p and pass the password on prompt.

Now create a database to further connect with the WordPress site. Use the syntax create database

→Database created. Now let’s head towards the WordPress dashboard where we left and pass the above database name, username, password, and instance endpoint on that screen.

On submit, internally it will write the above entries in the wp-config file and connect the front-end to the backend database. On a successful connection, the user will have to fill in the WordPress site details and dashboard login details. Run the WordPress installation when done.

😇Perfect. Each requirement for our problem statement is fulfilled. The below screen implies a successful installation and connection with the database. Here, all the user data is stored in a very reliable and powerful RDS instance. So even if the front-end goes down, the most critical user data won’t lose.

That’s all I have for this article. Thanks a lot for reading.🤗

I’ll be grateful to have connections like you on Linkedl *n * 🧑‍💼

Why so Kubernetes?

Poojan Mehta — Sat, 26 Dec 2020 11:36:34 +0000

Credits: ashleymcnamara/gophers

If you’ve been anywhere near the IT industry, you’ve very likely heard the term containers 🚢. The adoption of containers is growing exponentially as they are lightweight, portable, and revolutionary fast. Nowadays, deployments over containers are holding an upper hand over VMs for larger environments.

→ Containers provide isolated runtime environments and allocated resources are exclusively presented to the container, and any alteration won’t affect other running containers.

→ Such environments provide greater efficiency with running, managing, and in resource consumption.

→ With great power, comes great responsibilities ✌️. Imagine a situation where you have been using containers in production and your application starts getting massive traffic. All you need to do is scaling the application. How will you perform this? How will you decide inter-connection of containers within no time? How will you monitor all these containers and manage health-checks😥? We need an orchestration platform to scale and manage our containers.

This is where tools like Kubernetes☸️ comes into play.

What exactly is Kubernetes?

→ Kubernetes(K8s) an open-source project by Google , hosted by Cloud Native Computing Foundation that has become one of the most popular container orchestration (simply container management) tools around; it allows you to deploy and manage fault-tolerant, resource optimal container applications at scale.

→In simple words, you can bundle together groups of containers, and Kubernetes helps you easily and effectively manage those containers. It can span across on-premises, public, private or hybrid clouds. For this reason, Kubernetes is now an undisputed platform to host cloud-native applications that require rapid scaling. While in best practices, K8s is more often used with Docker. It provides :

Service Topology : Routing the traffic based upon cluster topology.
Service Discovery and Load Balancing : Service discovery is the process of figuring out how to connect to a service. Kubernetes has its own mechanism for that. It gives pods(basically one or more containers) unique IP addresses and a single IP to the collection of pods that enable load-balancing across the pods.
Storage Orchestration : This allows you to mount the storage system of your choice. Be it public, private cloud storage, NFS, and many more...
Horizontal Scaling : Scale your application up or down with commands, UI, or via auto-scaling seamlessly!
Automated Rollouts and Rollbacks : Monitoring the health-checks of the application ensures almost no downtime and matches the desired state for deployment solution. Kubernetes will rollout and rollback for you.
Self Healing : Always takes care of the desired state and restart, replace and reschedule the containers and only ready-to-work containers are advertised to the clients.

For a successful open source project, the community is as important as having great code. K8s has a very thriving community across the world. It has more than 3000 active contributors(according to the August 2019 report).

How Kubernetes Works?

→A working Kubernetes deployment is called a cluster. A cluster mainly contains the master node and several workers nodes. The role of the master node is to maintain the desired state and the worker nodes actually run the application workloads.

→If a worker node goes down, Kubernetes starts new pods on a functioning worker node. This makes the process of managing containers easy and simple. Your work involves configuring Kubernetes and defining nodes, pods, and the containers within them. Kubernetes handles orchestrating the containers.

Overview of a basic Kubernetes cluster

Some common terms for batter understanding of K8s:

Node: The machine that performs the tasks
Pod: The smallest communicable unit is a group of containers deployed in a node. Pods abstract network and storage from underlying containers.
Kubelet: It is responsible for maintaining a set of pods and ensures the defined containers are started and running.
etcd: It is a consistent and high-available key-value store.

The adoption of Kubernetes as the Go-To platform for hosting production-grade applications is ever increasing. Big brands like The New York Times, HBO, Reddit, Airbnb, Pinterest, Pokemon — all have their own K8s stories to tell. And many more are on their way to join them.

Kubernetes meets the real world: Airbnb’s story😄

→ Airbnb is an online marketplace for sharing, renting homes, and experiences. The transition from monolithic to the microservice architecture by Airbnb is quite commendable. The organization needed to scale horizontally to ensure continuous delivery and keep upscaling by adding new services. The sole purpose was to enable continuous delivery with a microservices architecture so that the team of over 1000 engineers can put up faster delivery.

→ Airbnb adopted to support the developer's team and configured, deployed over 250 critical production services to Kubernetes. Airbnb managed to scale with the microservice environment having over 20,000 deployments per week(all environments, all apps).

All slides available

→ Initially, the configuration was manual and rigid and not much evolved. Then, they shifted to configuration management with the chef on a monolithic. But the very complex hierarchy of services, inheriting configuration was not working as expected. Modifying the chef recipes was quite frequent and it would take down other services on convergence. And at the latest Airbnb moved to Kubernetes which automated orchestration of containerized setup.

→ More specifically with Kubernetes, the declarative approach seemed more resilient, and an easier scheduling process leads to optimization of cost. Also, it came with all features and advantages of Docker made the environment more granular. And most important, YML configurations are extremely human-understandable and provide a hassle-free development experience.

→Kube-gen, a tool for K8s, helped Airbnb take service parameters (defined in a single YAML file) and generate the complete K8s services manifests containing all the necessary configurations.

The outcome of this shift was quite decent, Kubernetes enabled Airbnb to add a layer of abstraction over the containers and set up an automated management workflow. Today, almost 50% of Airbnb workloads are running on K8s ☸

For more information and insights, go through the below keynote 👇:

Solutions like Kubernetes are buzzing with the spirit of DevOps

Getting Ready for the Kubernetes-driven future🤩:

Container-based microservices applications are the future and Kubernetes is their platform. It has reached a level where organizations can undoubtedly depend on and thrive in the competition. That’s why the big three cloud providers have all launched managed versions of K8s namely EKS by AWS, GKE by GCP, and AKS by Azure. RedHat Openshift is also a contender of Kubernetes distribution that one must not neglect.

== This seems extremely true and a bit relatable with frequently evolving K8s environments.

With its latest release in Dec. 2020, K8s depreciated dockershim(a component of kubelet) that favors the use of container runtime interfaces created for K8s. We are yet to witness the emergence of the implementation of runtimes like cri-o.

I really appreciate your time and attention in reading this piece. I’ll be grateful to have connections like you on Linkedl *n * 🧑‍💼

Setting Up Ansible for EC2 With Dynamic Inventory

Poojan Mehta — Fri, 25 Sep 2020 05:18:35 +0000

Setting Up Ansible for EC2 With Dynamic Inventory🙂

In this article, I will demonstrate how to provision EC2 instance using ANSIBLE and how do set up a more agile environment using the DYNAMIC INVENTORY.

→Pre-requisites:

— >RedHat Ansible downloaded and configured in the local system.

Do check out my previous article for Ansible👇👇:

LINUX AUTOMATION WITH ANSIBLE

~Problem Statement:

♦️ Deploy Web Server on AWS through ANSIBLE!

🔹 Provision EC2 instance through ansible.

🔹 Retrieve the IP Address of instance using the dynamic inventory concept.

🔹 Configure the webserver through ansible!

As Ansible is built on top of python, a Python Software Development Kit (SDK) is required that enables the configuration of AWS services. The package is an object-oriented API named boto3.

pip3 install boto3 //assuming python3 is installed

→STEP-1)

In the first step, I provisioned an ec2 instance with this playbook.
Here, the RedHat system itself calls the API for configuration on AWS, and this procedure is done on the local machine that’s why the host is supposed to be localhost.
For authentication to the AWS account, create one IAM user that has less privileged than the root account. The AWS_ACCESS_KEY and AWS_SECRET key are passed explicitly through an Ansible vault named secret.yml

Encrypted Vault🔒

- hosts: localhost
  vars_files:
      - secret.yml
  tasks:
   - name: Provision os in AWS
     ec2:
      key_name: "keytask" //keypair to be attached to the instance  
      instance_type: "t2.micro"
      image: "ami-0ebc1ac48dfd14136" //amazon linux 
      count: 1
      wait: yes
      vpc_subnet_id: "subnet-e7780dab"
      region: "ap-south-1" //asia-pecific-south region of AWS
      state: present
      assign_public_ip: yes
      group_id: "sg-0512d293cfb4af6e4" //security group 
      aws_access_key: "{{ myuser }}"
      aws_secret_key: "{{ mypass }}"
     register: ec2   

- debug:
       var: ec2.instances[0].public_ip

ansible-playbook ec2.py — ask-vault-pass🚀

Ansible register allows the user to capture the output and store as variables and can be used in different scenarios. The variables will contain the value returned by the task.

The register variable will print the public IP address of the instance from Ansible facts it gathers.

→STEP-2)

The instance has been launched! Next what? 🤔🤔

We need to dump the IP address of this instance into the inventory file and do the further procedure!

Wondering I will simply write the IP in the host file🤭?? NAH ! Not manually 🤫🤫

AND THIS IS WHERE 🔥DYNAMIC INVENTORY🔥 COMES TO PLAY:

→Ansible dynamic inventory is a concept that contains scripts that work as external APIs and pulls the information(facts) of a particular provider.

→The gathered facts will be dynamically dumped into the host file and further, we can create groups of these hosts according to requirement.

→Copy the following files into the controller node to enable dynamic inventory.

[https://raw.githubusercontent.com/ansible/ansible/stable-1.9/plugins/inventory/ec2.py](https://raw.githubusercontent.com/ansible/ansible/stable-1.9/plugins/inventory/ec2.py)

[https://raw.githubusercontent.com/ansible/ansible/stable-1.9/plugins/inventory/ec2.ini](https://raw.githubusercontent.com/ansible/ansible/stable-1.9/plugins/inventory/ec2.ini)

→Both files need to be in executable format:

chmod +x ec2.py
chmod +x ec2.ini

→Also, for account authentication, pass AWS_ACCESS_KEY and AWS_SECRET_KEY in the ec2.ini file. This will contact to AWS on our behalf and retrieve the information of the ec2 instance.

→Edit the inventory file in the ANSIBLE.CFG configuration files too.

→Now, to see the output, run ./ec2.py - - list

→Also, run ansible all — — list-hosts to see the available hosts.

Host added dynamically😃

→STEP-3)

→With a defined host, now the final step is to deploy our application! In this example, I am deploying an apache webserver.

→Before that, enter the key file in the ansible configuration file.

private_key_file= /root/path-to-private-key 🔒

This file also needs to be executable .. chmod 600 key_name.pem

🙌Out of the box yet important information about file access:

These numbers show different types of permissions given to a file or a directory.

the format is: chmod XYZ

x is the root or owner permissions

y is the group permissions

z is the permission for other users

Now let’s get to know what does these numbers mean. So, there are generally three types of permissions: read (r), write (w), and executable (x)

each number denotes some kind of permissions. They are:

0 = no permission

1 = only execute (- — x)

2 = only write (-w-)

3 = write an execute (-wx)

4 = only read (r — )

5 = read and execute (r-x)

6 = read and write (rw-)

7 = all (rwx)chmod 777: here, 7 means all permissions and three 7s means the rwx permission is given to all (owner, group, and other)

similarly, you can calculate the same for all the numbers.

Now, run one playbook that downloads the required packages into the instance and copy the code into the document root of the webserver.

- hosts: all
  become: yes
  remote_user: ec2-user //login as this user in the instance
  tasks:

- name: Download Httpd and Git in remote system
        package:
         name:
           - httpd
           - git
         state: present

- name: Clone code from GitHub
        git:
         repo: '[https://username:password@github.com/poojan1812/Ansible.git']
         dest: "/var/www/html/"

- name: start the services of httpd
        service:
         name: "httpd"
         state: restarted

ansible-playbook server.yml

→The output of this playbook -

Service started and code copied from GitHub to the doc. root

FINAL OUTPUT-

THAT’S IT

→🤗All steps completed and the Problem statement matched successfully!!

THANKS, A LOT FOR READING THIS SO ATTENTIVELY

I’ll be grateful to have connections like you on Linkedl *n * 🧑‍💼

LINUX AUTOMATION WITH ANSIBLE

Poojan Mehta — Sat, 05 Sep 2020 11:13:53 +0000

This article is a demonstration of Automation using ANSIBLE

~WHAT IS ANSIBLE? 🤔🤔

→Ansible is an infrastructure automation tool from REDHAT. It is widely used in the configuration of systems and setting up deployment environments.

→Ansible is an abstraction layer that covers all operating systems under its umbrella that helps configuration of large heterogeneous environment. It is built on top of PYTHON 🐍.

→Ansible has Modules that enables performing various tasks in the system.Power⚡ of Ansible is Playbooks. Playbooks are nothing but YML files containing modules as per user requirements.

Problem Statement For This Hands-On-

Write an Ansible PlayBook that does the following operations in the managed nodes:

🔹 Configure Docker

🔹 Start and enable Docker services

🔹 Pull the httpd server image from the Docker Hub

🔹 Copy the html code in /var/www/html directory and start the web server

🔹 Run the httpd container and expose it to the public

→The system from which user is operating Ansible is called Controller Node and all working nodes under controller node are called Managed Nodes.

→Here I have taken only one Managed node. The file containing information about managed node is called Inventory.

- hosts: DockerSlave
  vars_files:
    - secret.yml //will discuss about secret later in this article

Managed node and inventory file

→Mention inventory file in Ansible configuration file

→STEP -1)

>Configure yum repository for docker in slave system

- name: yum configuration in slave system
    yum_repository:
      name: DockerRepo
      baseurl: [https://download.docker.com/linux/centos/7/x86\_64/stable/](https://download.docker.com/linux/centos/7/x86_64/stable/)
      description: docker repo
      enabled: true
      gpgcheck: no

- name: Docker installation
    command: "yum install docker-ce --nobest -y"

Yum repo configured in slave node

→STEP-2)

Install Docker 🐋 and start the services.
As Ansible is built on top of Python, it is required to download docker package for python.
Ansible command and service modules are used in this step.

- name: Docker installation
    command: "yum install docker-ce --nobest -y"

- name: Start docker services
    service:
     name: "docker"
     state: started
     enabled: yes

- name: Install python36 package
    package:
     name: python36
     state: present

- name: Install python library for docker
    pip:
     name: docker-py

Docker installed

Docker services started

→STEP-3)

→Here, Httpd image for web server is pulled from DockerHub 🐋

- name: Pull docker image
    docker_image:
     name: httpd:latest
     source: pull

→STEP-4)

→In this step, using git module, cloned the code from GitHub. One of the ways is to pass the credentials in the link to get authenticated in GitHub. But, it’s not a good practice to pass as plain text in the playbook😕.

→Use concept of Vault to encrypt the secret information and pass it in playbook as parameters. I have created one Vault that contains my GitHub credentials.

Encrypted vault🤗

- name: Clone code from GitHub
    git:
     repo: '[@github](https://{{gituser}}:{{gitpass}}<a%20href=).com/poojan1812/hybrid-cloud.git'">https://{{gituser}}:{{gitpass}}[@github](http://twitter.com/github).com/poojan1812/hybrid-cloud.git'
     dest: "/root/code_html/"

>This will clone the repository in the destination folder of slave system.

Code cloned in slave node

→STEP-5)

The last step will launch a docker container and expose the port for the public world.
Docker_container module is used here to launch and manage the container.
The HTML code from local system will be copied in the document root of web server.

- name: Launch container
    docker_container:
     name: img_httpd
     image: httpd:latest
     state: started
     exposed_ports:
      - "80"
     ports:
      - "2025:80"
     volumes:
      - /root/code_html:/usr/local/apache2/htdocs/

Launched container🚀

Now, run the following command to apply the playbook in managed node

ansible-playbook <file-name>.yml --ask-vault-pass

DONE🤩🤩🤩

FINAL OUTPUT-

curl <ip-container>/index.html

Thank you guys for taking the time out to read my rant lol!🥰😅

I’ll be grateful to have connections like you on Linkedl *n * 🧑‍💼

DevOps Automation 2

Poojan Mehta — Wed, 12 Aug 2020 05:43:58 +0000

→WHAT IS THE PROJECT?

TOOLS USED — GIT, JENKINS, DOCKER

Problem Statement-

JOB1: Pull the GitHub repo automatically when some developer commits.
JOB2: By looking at the code or program file, Jenkins should automatically start the respective language interpreter install image container to deploy code ( eg. If code is of PHP, then Jenkins should start the container that has PHP already installed)
JOB3: Test your application if it is working or not.
JOB4: If the application is not working, then send an email to the developer with an error message and if it is running file send a success message.
JOB5: To monitor the container where the application is hosted, fails due to any reason then this job will automatically start the container again.

First Create DockerFiles for 3 different image requirements-

Dockerfile for Jenkins image

~Here, CentOs image is taken and git, python, sudo, net-tools, and similar essential tools are to be downloaded.

~Also, install Jenkins and grant Sudo powers to Jenkins user. Also, add the command to start Jenkins services automatically as soon as the container is launched.

Dockerfile for Html image

~Here also, CentOs image is taken and important packages are downloaded and HTTPS service will start after the container is launched.

3.Dockerfile for PHP image

~Same piece of image and packages from the HTML image and PHP package is also added.

I’ve passed -y option in every command to ignore user prompt while downloading.
The final output of all images after build —

Jenkins_task, html_img, and php_img are dockerfile created and Jenkins service is also started automatically

JOB1-

→This job will pul the code from the GitHub and store it in the Host system in the specified folder.

Here, an already running container of HTML of PHP will stop and code will be copied in the folder named data_folder.

Poll SCM will always monitor the Git repository and pull the code instantly after code is pushed in GitHub.

Job1 success and job 2 triggered

JOB2-

→This job will detect the code and will launch a docker container accordingly.

->If HTML code is pulled from Git then html_img image and if PHP code is copied then php_img image will be used

>This job is chained to job1. So, it’ll automatically run just after a successful build of job1.

Job1 pulled HTML data, so a docker container for HTML is launched

JOB3-

→This is the main job!! This is a chained job with upstream job-2 and will trigger after a successful build of job2.

→Here, the HTTP status code of the webserver is used and passed in the condition.

->The status code is to be stored in a variable through the output of the curl command.

>Also, Jenkins mail plugin can be used to send customized mail notification.

Configure Jenkins as a mail client. Jenkins will contact the google SMTP server.

-Jenkins is a third party application and will be considered as a third-party application for a google account. Create a 16 digit password and provide it to Jenkins.

Add POST BUILD ACTION and select MAIL NOTIFICATION and add the recipient mail address.

JOB4-

→This is an independent job that will continuously monitor the Jenkins container and launch one new container if the running one is failed due to any reason!!

THAT’S IT !! AUTOMATIC JENKINS PIPELINE SET UP IS DONE.

FUTURE SCOPE — INTEGRATING THIS PIPELINE WITH MULTIPLE DOCKERFILES AND LAUNCH DIFFERENT OS ACCORDING TO REQUIREMENT ON THE FLY!

GitHub URL for reference — https://github.com/poojan182/jenkins-pipeline

THANKS A LOT FOR YOUR ATTENTION FOR SO LONG!!

Face Recognition Using Transfer Learning

Poojan Mehta — Sat, 06 Jun 2020 06:56:00 +0000

→First of all, many thanks to MR.VIMAL DAGA SIR for mentoring and training in Machine learning from very basic to advance level

→I have completed Face Recognition Using Transfer Learning

→Environment requirements —

Keras, TensorFlow, Numpy, Jupyter, cv2

→In this, I have used VGG16 pre-created dataset and using this pre-trained model implemented face-recognition over my dataset

→I have taken 2 faces of Virat Kohli and Rohit Sharma and collected some images.

→It is required to have Testing and Training data. So, suggested taking 80:20 ratio of training: testing dataset.

→Screenshots of whole code and workflow

dataset

→snapshots of code

→Still, adding more CNN may lead us towards higher accuracy

final output-

GitHub URL for reference- https://github.com/poojan182/face-recognition

→Thank you for your attention!

Machine learning and DevOps integration

Poojan Mehta — Sat, 30 May 2020 11:42:15 +0000

→First of all, many thanks to MR.VIMAL DAGA SIR for mentoring and training in Machine learning and DevOps from very basic to advance level

→Using this valuable knowledge I have completed the task of integration Machine learning with DevOps

→Tools used for DevOps — Git, Jenkins, Docker,

→Concept used for Machine Learning- Deep learning

SYSTEM CONFIGURATION/REQUIREMENTS-

→Base os Windows and virtual os Redhat Linux

→Jenkins installed in your RedHat system (including GitHub plugin)

→Docker installed with an image which can run python code

→Git installed in Windows and Redhat os

Problem statement- While creating a deep learning model, the process becomes too time taking and somewhat manual. And we all know, DevOps is used when we need automation!!

→Here deep learning will keep on creating the model and DevOps will keep on monitoring it and when deep learning gets accuracy lower than certain level Jenkins will automatically enhance the Dl code and will increase accuracy on its own

→STEP 1 →Creating a deep learning code and push it to GitHub with .PY file extension

→I have used cifar-10 dataset

deep-learning code

Steps to get your code into GitHub-

1) git clone — to clone the repository to your local system

2) ml_devops_integration1_py.py — code which is mentioned above

3) git add — to add the file into the staging area

4) git commit -m “message” — to make your changes permanent

5) git push origin — to push your changes into the remote repository

NOW HEADING TOWARDS CREATING A DOCKER IMAGE WHICH IS COMPATIBLE WITH PYTHON CODE

CREATE A DOCKERFILE WITH FOLLOWING

Run docker build command after creating docker file

→NOW THERE IS A PIPELINE OF 6 JENKINS JOBS

JOB1>This job will simply copy code from Github and paste into a specific folder in Redhat os

job 1 detail

JOB2>This job is critical and important as well

→This container will first search for KERAS and it will launch the os with provided docker file so that only deep learning code runs on that os

job 2 detail

Here job 1 is set upstream for this. Because it is important to get code from GitHub

Execute shell for reference-

“if sudo grep Keras | cat /MlOps_workspace/ml_devops_integration1_py.py

then

if sudo docker ps -a | grep img1

then

sudo docker rm -f img1

sudo docker run — name img1 -dit -v /MlOps_workspace:/Mlworkspace python_img:v1

else

sudo docker run — name img1 -dit -v /MlOps_workspace:/Mlworkspace python_img:v1

fi

else

echo “This is not a deep learning code”

fi”

→Here we have to mount permanent volume so that we can have our data persistent

JOB-3> This job will run the PYTHON code for initial level

job3 detail

Execute shell for reference-

sudo docker exec img1 python3 /Mlworkspace/ml_devops_integration1_py.py

JOB-4>>This job will check accuracy in our initial run

job 4 detail

→If accuracy is less than 95% then Jenkins will do changes in code like increasing epochs and adding consolation layers to reach towards higher accuracy

Execute shell for reference-

read=$(sudo cat /MlOps_workspace/result1.txt)

echo $read*100 | bc -l

if (( $(echo “$read > 0.95” | bc -l) ));

then

echo “Achieved accuracy”

exit 1

else

cd /MlOps_workspace/

sudo sed -i ‘/^epoch=.*/a epoch=epoch+5’ ml_devops_integration1_py.py

sudo sed -i “62i model.add(Conv2D(filters=32, kernel_size=3, padding=’same’, activation=’relu’, input_shape=input_shape))” ml_devops_integration1_py.py

sudo sed -i “63i model.add(MaxPool2D(pool_size=2))” ml_devops_integration1_py.py

fi

→sed keyword is used to add specific code into our original file

JOB-5>This job will run again the edited code and find new accuracy

job5 detail

→This job is set-upstream to job4 and it will run until we get 95% accuracy in our model

→It will keep on adding epochs and convolutional layers till achieving accuracy

JOB-6> This is the final job of the whole process

→This job will monitor the docker container

→In any case, if the container goes down or fails to launch then this job will launch a new container

THAT’S IT

→The WHOLE PROCESS IS AUTOMATED WITH CERTAIN ACCURACY

→LEARNED MANY NEW CONCEPT THROUGH THIS TASK AND LOOKING FORWARD TO ENHANCING THIS EVEN BATTER

GitHub link for reference- https://github.com/poojan182/workspace_repo

Thanks for reading!! Kudos to you

Getting Started with CI/CD

Poojan Mehta — Thu, 07 May 2020 08:58:00 +0000

→WHAT IS THE PROBLEM STATEMENT?

TOOLS USED — GIT, JENKINS, DOCKER.

→Jenkins plays 3 jobs in this system

job-1) Jenkins will keep on monitoring and keep on deploying out the site on a TESTING server

job-2)Jenkins will keep on monitoring and keep on deploying out the site on the PRODUCTION server

job-3)It will run this job only when it is triggered by the testing team and it will merge branched and run job 2 and finally destroy the testing server!

SYSTEM CONFIGURATION/REQUIREMENTS-

→Base os Windows and virtual os Redhat Linux

→Jenkins installed in your RedHat system (including GitHub plugin)

→Docker installed with https configuration

→Git installed in Windows and Redhat os

→Ngrok (optional)

DETAILED DESCRIPTION ABOUT THE WHOLE PROCESS-

→ Starting with GIT

| → Create a git repository on GitHub and clone it into your local system

| →Create files in the local system and push them to GitHub

COMMANDS USED-

1) git clone — to clone the repository to your local system

2) notepad file1.txt — to create a file for our use

3) git add file1.txt — to add the file into the staging area

4) git commit -m “message” — to make your changes permanent

5) git push origin — to push your changes into the remote repository

6) git branch — to create a branch in the local system and then make changes and commit then push them too

7) git merge — to merge the branch with the master branch

some useful commands

1)git log — to find all versions of our files

2)git status — to get track information of all files

JOB 1-

→ Job 1 of Jenkins is to take files from GitHub and copy deploying it into TESTING SERVER

→ Click on “new item” to get started

→Click on “freestyle project” and add name

→ Select GIT and then link you GitHub profile where you want to download file

→Build trigger with “POLL SCM” in make Jenkins watch your GitHub every time

→Last step to “EXECUTE SHELL” and inform Jenkins where to put downloaded files

→In Linux, Jenkins user does not have all the privileges to manipulate files

| →By using “SUDO” command will give all power to Jenkins

→With this job, Jenkins will deploy files to the testing server (Here we launch new os only if our testing server is not running)

JOB 2-

→ Job 2 of Jenkins is to copy the file and deploy it to “PRODUCTION SERVER”

→Create a new job and give an appropriate name

→Link job2 with GitHub and

→Use POLLSCM to keep monitoring GitHub

→As shown in image use commands which copy files from GitHub and deploy into the webserver

→Here job 2 is completed

→But job2 is an “UPSTREAM PROJECT” which will only work if job 3 builds successfully

JOB 3-

→Job 3 is the main part of this system

→Job 2 is totally dependent on job 3. This is called “JOB CHAINING”

→First link your repository with storing credentials

→Then specify the branch you need to monitor and in advanced options give “MERGE BEFORE BUILD”. So this will merge branches and then build the job

→Job 3 is most sensitive because if this job gets wrong execution then our whole site in the production server gets harmed.

→In this job we use “BUILD TRIGGERS REMOTELY”

→So it will trigger just after the token is provided manually by the testing team

→As soon as the testing team pass and run the trigger, job 3 will run and our site will be in production server

→After the testing server approves, we no longer need the testing server

→So in EXECUTE SHELL we run the

script to destroy the testing server

THAT’S IT!! OUR WORK DONE THROUGH AUTOMATION SYSTEM

→FINAL OUTPUT OF OUR FILE

->OUTPUT OF DESTROYED TESTING SERVER!!

→GITHUB URL FOR REFERENCE- https://github.com/poojan1812/testing

→HAD A GOOD EXPERIENCE OF LEARNING AND BUILDING THIS PROJECT!!

FUTURE SCOPE -

→FURTHER WE CAN INTEGRATE THIS SYSTEM WITH VARIOUS TOOLS AND AUTOMATION SYSTEM.

→WE CAN INTEGRATE THIS WITH MACHINE LEARNING MODELS AND LINK THIS WHOLE WITH CLOUD COMPUTING TO TO CREATE A MASTERPIECE AUTOMATION SYSTEM WITH STORAGE EFFICIENCY INCLUDING ARTIFICIAL INTELLIGENCE

→THIS RAW SYSTEM CAN BE DIRECTLY USED EFFORTLESSLY IN DevOps WORLD