DEV Community: Hrittik Roy

Check this new post out!

Hrittik Roy — Mon, 03 Mar 2025 13:44:06 +0000

Technical Guide: Syncing Ingress Resources from various Virtual Cluster on GKE with vCluster

Hrittik Roy for LoftLabs ・ Mar 3 '25

Technical Guide: Syncing Ingress Resources from various Virtual Cluster on GKE with vCluster

Hrittik Roy — Mon, 03 Mar 2025 13:43:35 +0000

Kubernetes Ingress is the most widely used Kubernetes resource for exposing an application to the outside world. Understanding the concepts and Layer-7 load balancing may sound difficult, but with this article, it won’t be.

This article uses Google Kubernetes Engine (GKE) as the host Kubernetes cluster, where you will install the Nginx Ingress controller and cert-manager to get TLS certificates for your web apps. With that, you can run an application with proper TLS.

This article touches on how to create a virtual cluster using vCluster to reuse the host cluster ingress controller and cert-manager to create ingress. This approach allows your virtual clusters to reuse the ingress controller running on the host cluster, GKE in our case, and the cert-manager.

If you are hearing about virtual clusters for the first time, then read more here.

Prerequisites

Ensure these are installed:

Google Cloud SDK (gcloud) – To interact (create/delete) with GKE. Make sure to have a project and a billing account linked and authenticated. Install [Note: You can use the UI if you prefer]
Kubernetes CLI (kubectl) – To manage Kubernetes clusters. Install
vCluster CLI – To create virtual clusters within a single Kubernetes cluster. Install
A Domain Name – This is used to configure A Records, DNS, and TLS.
Basic Kubernetes Knowledge – Familiarity with Deployments, Ingress, and Services.

Setting Up GKE Cluster

Once you have completed the prerequisites, you can proceed to the next step: creating your Kubernetes cluster. This cluster will serve as the environment where you deploy your controllers, deployments, services as well as your virtual clusters.

With gcloud installed, you can create your cluster using the command gcloud container clusters create, along with a few parameters to specify the project and location of the deployment:

Command:

gcloud container clusters create test  --project=hrittik-project --zone asia-southeast1-c

After you apply the above command, your gcloud SDK will begin creating the cluster. This process will take a few minutes, and once completed, a kubeconfig entry will be generated that allows you to interact with the cluster using kubectl.

Successful cluster creation will look something similar:

Creating cluster test in asia-southeast1-c... Cluster is being health-checked (Kubernetes Control Plane is healthy)...done.                                                                     
Created [https://container.googleapis.com/v1/projects/hrittik-project/zones/asia-southeast1-c/clusters/test].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/asia-southeast1-c/test?project=hrittik-project
kubeconfig entry generated for test
NAME  LOCATION           MASTER_VERSION      MASTER_IP       MACHINE_TYPE  NODE_VERSION        NUM_NODES  STATUS
test  asia-southeast1-c  1.30.8-gke.1051000  34.124.254.219  e2-medium     1.30.8-gke.1051000  3          RUNNING

Setting Up the Ingress Controller

Once your GKE cluster is set up and operational, the next step is to install an Ingress Controller to handle the routing of external traffic to your services. In this section, we will use NGINX as our Ingress Controller. NGINX is a popular and reliable choice for Kubernetes ingress management, renowned for its flexibility and performance.

Ingress will act as a reverse proxy to route traffic outside the cluster to specific Kubernetes Services based on the Ingress Rules.

Cluster Admin Permissions

To configure Ingress on GKE, the first step is to provide the user cluster-admin permissions to carry out the operational tasks:

kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole cluster-admin \
  --user $(gcloud config get-value account)

Success in this step will create a cluster-admin-bindingin your cluster like below:

clusterrolebinding.rbac.authorization.k8s.io/cluster-admin-binding created

Ingress Controller Deployment

With the appropriate permissions configured, the next step is to deploy the controller which will manage all of the routing logic. The step’s as simple as running the command in your cluster:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0/deploy/static/provider/cloud/deploy.yaml

Success will create resources in an ingress-nginx namespace as shown below:

namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created

Testing Ingress Controller

The key point to focus on is that the ingress-controller will be exposed as a LoadBalancer by default, which we will explore later in the configuring A Record Section. For now, to verify the installation, check if your pods are running by using the command below:

kubectl get pods -n ingress-nginx

The Status as Running means everything is configured correctly:

~ ❯ kubectl get pods -n ingress-nginx                                  
NAME                                       READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-bh7nl       0/1     Completed   0          100s
ingress-nginx-admission-patch-bvlhf        0/1     Completed   0          100s
ingress-nginx-controller-cbb88bdbc-5dkxt   1/1     Running     0          101s

Installing Cert-Manager on the Host Cluster

Now that your Ingress Controller is set up and running, the next step is to install Cert-Manager to automate the management and issuance of TLS certificates. Cert-Manager is a powerful Kubernetes project that simplifies obtaining and renewing SSL/TLS certificates from various certificate authorities (CAs) like for eg: Let's Encrypt.

This tutorial uses Cert Manager to create and store certificates as Kubernetes Secrets automatically.

Installation of cert-manager

The installation process is straightforward; simply execute the command below on your cluster:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.3/cert-manager.yaml

Successful installation will look similar to the screenshot below:

Verifying cert-manager Installation

Once installed, you can verify that Cert-Manager is running by checking the pods in the cert-manager namespace:

kubectl get pods --namespace cert-manager

Successful creation will display a manager pod, a cainjector pod, and a webhook pod to verify domain authority within that namespace, as shown below:

~ ❯ kubectl get pods --namespace cert-manager                                                                                                

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-6c7fdcbcd5-6lp94              1/1     Running   0          2m11s
cert-manager-cainjector-64d77f8498-f6p7d   1/1     Running   0          2m12s
cert-manager-webhook-68796f6795-59sqq      1/1     Running   0          2m11s

Creating a Certificate Issuer CRD

A Certificate Issuer instructs the manager on how to obtain certificates. There are two primary types of issuers:

1. ClusterIssuer: This issuer is available throughout the entire cluster and is recommended for most cases.

2. Issuer: This issuer is limited to a specific namespace.

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod
 namespace: cert-manager
spec:
 acme:
   # The ACME server URL
   server: https://acme-v02.api.letsencrypt.org/directory 
   # Email address used for ACME registration
   email: test-hrittik@example.com # Replace with your email
   # Name of a secret used to store the ACME account private key
   privateKeySecretRef:
     name: letsencrypt-prod
   # Enable the HTTP-01 challenge provider
   solvers:
   - http01:
       ingress:
         class: nginx
EOF

In this example, we will use a ClusterIssuer with Let's Encrypt as the verification server. Don’t forget to update your email address in the YAML below before applying:

Success will look similar:

clusterissuer.cert-manager.io/letsencrypt-prod created

Verify the Cluster Issuer

To verify successful installation, use the below command to get the clusterissuer CRD:

kubectl get clusterissuer letsencrypt-prod

Successful output will be similar to:

NAME AGE 
letsencrypt-prod 3m

Configuring A Records for Domain

Now that you have set up your NGINX Ingress Controller and are issuing certificates with Cert-Manager, it’s time to configure the A Records **for your domain. In Google Kubernetes Engine (GKE), A Records **are used for mapping.

An A Record (Address Record) in DNS points a domain or subdomain to a specific IP address. In this case, you need to direct your domain (for example, hrittikhere.live) to the external IP address of your NGINX Ingress Controller. This will allow external traffic to be properly routed to your cluster.

Getting the External-IP

After installing the NGINX Ingress Controller, a LoadBalancer type service will be created that automatically allocates an external IP address. To find the allocated IP, use the following command:

 kubectl get svc -A | grep -E "NAME|ingress-nginx-controller"

The output will display a list of services, from which you should select the one that has an external IP. Keep the IP address readily available:

Configuring A Name Record on Domain Provider

The next step is to log in to your domain provider and navigate to the DNS Record section. Depending on the provider, the UI might change but you just need to configure four three things once you click on Create New Record:

Name/Host: * # Wildcard to capture all Hosts

Type: A # A record for GCP

Value: 34.142.255.42 # External IP address of your NGINX Ingress Controller

TTL: 300 # Use the default value.

On the Dashboard, this will look something similar to below:

Click on Add Record and you will be all ready for the next step!

Creating an Application on the Host Cluster

Now, you have all of the moving pieces in place: domain to serve your application, ingress controller for Load Balancing and Cert-Manager for TLS. The only missing piece is application so the next step is to deploy one simple game to your cluster.

The following YAML deployed three things together:

Service: Exposes the game application via a Kubernetes Service.

Deployment: Deploys the game application.

Ingress: Defines routing rules for the Ingress Controller to route external traffic to the Service.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: game-2048
  namespace: default
spec:
  ports:
    - name: http
      port: 80
      targetPort: 80
  selector:
    app: game-2048
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: game-2048
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: game-2048
  template:
    metadata:
      labels:
        app: game-2048
    spec:
      containers:
        - name: backend
          image: alexwhen/docker-2048
          ports:
            - name: http
              containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: game-2048-ingress
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: game.hrittikhere.live
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: game-2048
                port:
                  number: 80
  tls:
  - hosts:
    - game.hrittikhere.live
    secretName: letsencrypt-prod
EOF

Success will create three objects in your cluster:

service/game-2048 created
deployment.apps/game-2048 created
ingress.networking.k8s.io/game-2048-ingress created

Verification of Ingress

To find the ingress Host you can use the following command:

kubectl get ingress

Success will look something like this:

~/Code/ingress ❯ kubectl get ingress                                                                                               
NAME                CLASS    HOSTS                   ADDRESS         PORTS     AGE

game-2048-ingress   nginx    game.hrittikhere.live   34.142.255.42   80, 443   51s

If you go to the following URL, you will find your application running on the host:

If you check the network tab, the Remote Address corresponds to the LoadBalancer assigned by the Ingress Controller. In simple terms, the Ingress Controller assigns the subdomain and serves the service smoothly.

Syncing Ingress Resources with vCluster between your Virtual Clusters

After completing this tutorial, you will see that it’s a bit complicated to set up the backbone to create ingress resources. However, imagine the IT headache if you’re operating 100s of Clusters and managing all these certificates and secrets while making sure your domains are linked well. Sounds Challenging?

vCluster saves you from these and many similar problems. What vCluster does is create virtual clusters on top of your host clusters in specific namespaces. This will help you create a virtual cluster for each of your teams with full admin privileges in seconds instead of the hours it takes to create a normal cluster and configure all the required resources.

You just define a virtual cluster and what host cluster features you want in your virtual cluster. For example, if we want to ingressClasses from our Host Cluster and sync ingresses back to our host cluster from the virtual cluster to back to your host it’s as simple as defying it in the vcluster.yaml file like below:

sync:
  fromHost:
    ingressClasses:
      enabled: true
  toHost:
    ingresses:
      enabled: true

With this, whatever application you’re running on your virtual cluster can request and run an ingress smoothly. Let’s see a demo but the first step is to create the virtual cluster using the vCluster CLI [Installation Step]:

 vcluster create my-vcluster -f vcluster.yaml

Success will look something like this:

11:35:20 done Successfully created virtual cluster my-vcluster in namespace vcluster-my-vcluster

With that, make sure that you’re in the** virtual cluster context** and you can apply the same application again with a new subdomain path:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: game-2048
  namespace: default
spec:
  ports:
    - name: http
      port: 80
      targetPort: 80
  selector:
    app: game-2048
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: game-2048
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: game-2048
  template:
    metadata:
      labels:
        app: game-2048
    spec:
      containers:
        - name: backend
          image: alexwhen/docker-2048
          ports:
            - name: http
              containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: game-2048-ingress
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: game1.hrittikhere.live
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: game-2048
                port:
                  number: 80
  tls:
  - hosts:
    - game1.hrittikhere.live
    secretName: letsencrypt-prod
EOF

The success will look like this again when you have three things configured:

service/game-2048 created
deployment.apps/game-2048 created
ingress.networking.k8s.io/game-2048-ingress created

On the CLI, if you do kubectl get ingress you will see a new ingress resource created with the new specified host: \

Now, if you check game1.hrittikhere.live, you will find a similar game running alongside. With just one vCluster configuration, you can create multiple new clusters, and all of them will have ingress functioning seamlessly.

The IP is also again the same as your host cluster, ensuring all the networking is working smoothly:

Clean Up

With everything working the cleaning steps are very simple. You can remove the DNS Record from your service provider through the UI by clicking on delete record and for the GKE Cluster use the following command with your own parameters:

gcloud container clusters delete test  --project=hrittik-project --zone asia-southeast1-c

Successful deletion will look something like this:

With that, you have cleared all the resources, including your virtual clusters.

Final Thoughts

With this guide, we’ve explored how to configure an Ingress Controller using NGINX in a GKE (Google Kubernetes Engine) environment and to set up TLS certificates with Cert-Manager. We’ve also seen how to configure DNS A Records to point to a Kubernetes cluster’s external IP and route traffic efficiently to our services.

In addition to all of this, you’ve learned how powerful vCluster can be, especially when dealing with multi-tenant Kubernetes environments. If you're managing several teams or clusters, using vCluster allows you to create isolated, lightweight Kubernetes clusters (virtual clusters) within a larger host cluster, providing the flexibility of separate clusters without needing to pay for the control plane cost of each cluster.

More Questions? Join our Slack to talk to the team behind vCluster!

Supercharge your remote development environment with DevPod

Hrittik Roy — Tue, 19 Dec 2023 04:53:23 +0000

DevPod is that new kid in town that works on the same standard of devcontainers.json that Codespaces uses but is on the infrastructure of your choice and is open source. The project was just launched this May and has gathered more than 5.3K stars in this period. The advantage is the lower costs (around 5-10 times cheaper than cloud VMs) with auto-shutdown.

This blog talks about the tool, how you can get started with DevPod quickly on GCP, and the benefits of Cloud Development Environment in general.

About DevPod

DevPod is a client-side tool, which means you can install either the Desktop application or the CLI locally and easily get started. The steps are simple:

Install DevPod
Configure your Provider or your infrastructure connection, which can be your local Kubernetes Cluster, Docker Instance, or a Cloud Provider. Here in this tutorial, we’ll use GCP.
Choose your IDE among the ones like VScode, Jupyter Notebook, Fleet, or others from JetBrains.
Click on Start and get your development environment ready.

But again, there’s a simple question: Why go through this hassle when you can do things locally on your local machine?

Well, let me ask you a few questions:

What’s the most complicated thing while onboarding a new project or team? Is it setting it up and ensuring everything, including the dependencies, works smoothly?

Is managing a lot of dependencies and fixing broken dev environments a waste of time?

Do you feel you need a powerful machine or more bandwidth for your day-to-day operations?

Suppose the answer to these questions is Yes. In that case, you understand there’s a very big requirement for a system that can use the flexibility of bigger machines and easier dev environment creation. Here, the combination of devcontainer and DevPod comes up.

Features that you want

The central problem that DevPod addresses is creating a smooth and consistent developer experience (DevEx) that is smooth and helps developers get to work without configuring broken environments. Let’s see some other features:

Local development: You get the same developer experience locally, so you only need to rely on a cloud provider if you need the extra boost.

Rich feature set: DevPod already supports prebuilds, auto inactivity shutdown, git & docker credentials sync, and many more features.

Time for a Tutorial?

Now, with all the features set and praise about Dev Containers, you might want to know how you can get started with it as soon as possible. Remember the four steps from above where we discuss the simplicity that’s exactly how you get started. Before diving deeper, ensure you have a Google Cloud account, an unrestricted local machine, and an active internet connection.

Step 1: Install DevPod

DevPod supports any OS of your choice, so go to the following links according to your choice and install the Desktop Application on your system. The CLI can be installed independently or via the App.

If you prefer just a CLI, here’s where you’ll find additional instructions.

Step 2: Get your backend sorted

Now, assuming if you have Docker installed, you can use your local machine. But if you require to use some extra resources for extra compute power then you can use the remote backends via SSH or cloud.

The steps are simple: Have a Cloud Account (Here it’s GCP) and then two steps:

Install and Configure gcloud CLI locally using the documentation
Click on Add Provider in the Providers sidebar.

Select your desired Provider (Here Google Cloud):

Select the Project ID and your desired region on the UI and click on Add Provider:

Congrats 🎉 Your Provider is now available 👏

Step 3: Get Started with Your Workspaces

Now, workspaces are where you will code and develop your application, i.e., your development environment. Let’s move to the Workspace tab on your side panel and then click on Create Workspace:

Add your Repository URL and choose your preferred IDE among the many options:

Once done, click Create Workspace and the process starts. If you scroll down, you can choose the name of workspace prebuilds, but we’ll keep this for a more advanced tutorial.

Wait for the daemon to be installed; in a few minutes, your workspace will be running on your browser.

Get Started coding

Now, you can install your required extensions and start your development quickly. Also, if you want to switch your default IDE in between development, you can easily do that from the workspace tab:

Step 4: Clean UP your workspace

Navigate back to your workspace Tab to see all your workspaces running. Here you have an option to Rebuild, Stop, and Delete.

Choosing Delete will destroy your workspace:

Confirm, and it will remove the cloud resources as well:

Final Thoughts

Having a reliable method to create and reproduce setups can greatly benefit teams. In this tutorial, you learned how effortless it is to generate a workspace and utilize it for isolated workspaces. For more complex scenarios, you can implement pre-built configurations to reduce boot times and also utilize templates for more consistent workflows.

Also, remember there are data protection laws that prevent data from leaving the site so solutions like DevPod are very helpful. Like here’s a tutorial for a small-scale representation of on-premise Kubernetes and how you can use DevPod with VS Code Browser.

Join the DevPod Slack if you face any issues while trying it out.

Software Delivery Foundation: How you can crack it?

Hrittik Roy — Fri, 28 Oct 2022 17:29:39 +0000

SDF, or Software Delivery Foundation, is the brand new certification from Harness that is intended for software delivery practitioners with at least 1-2 years of experience and about 2-3 months of Harness experience. The certification lets you demonstrate your knowledge of basic software delivery concepts.

I passed my cert last month, and this blog can be a good fit for how you can excel in your certification attempt as well!

Who needs the Certification?

If you’re interviewing with Harness, then this will be one of the rounds which will help you to understand the breadth of Harness products on which you will be working, so this certification acts as an important knowledge base for those who want to work on Harness products or use them.

Moreover, if you were already an employee before the certificate launch, this certification gives you a look into all the other awesome harness products that your team can complement and helps you understand other modules that Harness offers.

Also, if you’re the curious kind and always love upskilling yourself this certification will be very helpful to grade yourself and learn new technologies.

What is the Harness Software Delivery Foundation?

The certification program is designed to ensure that Hanress’s customers, partners, and employees have the skills and knowledge necessary to be successful with Harness. These goals put the certification in a hard category, and it will take you some time to prepare if you’re starting to learn the product from scratch.

The exam is designed for software delivery practitioners, costs $150 with a validity of one year, and consists of four domains:

Domain 1: Software Delivery Fundamentals

Domain 2: Harness Second Generation Architecture

Domain 3: Security

Domain 4: Account Level Information

Passing Criteria

To pass the exam you need to get at least 42 out of 60 questions correct so covering all the domains is essential! In the exam, there’s no negative marking and you get 65 questions, which you need to complete in 90 minutes. 5 of the 65 questions don’t carry any marks and are seeded.

For seeded questions, whether you attend them or not, you don’t get any marks from that. They are just added to maintain the quality and later decide if they want to keep it or not. You, as a candidate, will have no idea which is seeded among those 65. :)

Prerequisites:

The prerequisites as mentioned on the website are:

Basic computer skills
Basic networking skills
Basic computer security knowledge
Fundamental understanding of the Linux Operating System and the capability to configure and deploy workloads using command-line interfaces
Fundamental understanding of and experience configuring and deploying workloads
Fundamental understanding of containers and the capability to configure and deploy workloads using command-line interfaces
Fundamental understanding of container runtimes
Fundamental understanding of DevOps practices and methodologies

In my experience, having all of the prerequisites isn’t necessary but learning them before starting the preparation can save you a lot of time and energy.

Preparation Strategy

Harness has a guide that you can use as a checklist to prepare for the exam! Covering all of the modules isn’t necessary if you have previous knowledge of the product, but if this is your first time, I highly recommend checking all of the content on the list: https://university.harness.io/exam-prep-sdf/1191834

Don’t miss the courses from Harness University as they are very important to clear your basics and some direct questions from the content can appear in your certification attempt.

However, to excel, the list won’t be enough and you need to focus on some specific areas as well. The areas are:

Understanding Pricing and what one plan offers on top of another for all the different modules: https://harness.io/pricing?module=cd#

Understanding the Drone License and its limitations: https://docs.drone.io/enterprise/

Organizations and Projects Overview with*_ Resources Across Scopes_*: https://docs.harness.io/article/7fibxie636-projects-and-organizations

Delegate in details with architecture: https://docs.harness.io/article/h9tkwmkrm7-delegate-installation

https://docs.harness.io/article/2k7lnc7lvl-delegates-overview

https://docs.harness.io/article/lwynqsgxt9-delegate-requirements-and-limitations

FAQs for all the modules: https://docs.harness.io/article/320domdle1-harness-security-faqs

SaaS authentication: https://docs.harness.io/article/gdob5gvyco-authentication-overview

The best way to learn about the certification will be to follow the tutorials and try out the steps, which will help you to remember the process and options. Memorizing all the combinations will be hard.

If you get time, going through all the documentation can be very handy as it will help you understand the bigger picture of the whole platform, and that’s what the certification aims towards and increases the chances of you getting a good score.

Shiny Credly Badge:

Once you sit for the exam and pass, you get the shiny badge that you can brag about on social media! 🥳🥳🥳

Some Tips:

All the best for your certification, and remember the following tips to have a good experience:

Schedule your exam at least 72 hours in advance, giving yourself time to schedule a proctor. Read this Guide for more details on desk setup, scheduling and planning: https://university.harness.io/live-sdf/1268412
Make sure that you are using the current version of Chrome and have downloaded the ProctorU Chrome extension available at http://bit.ly/proctoruchrome.
Keep your ID handy and make sure your desk is clean
Use your personal laptop and not the one issued by your organization.
Consider all the alternatives before settling on one. Some may be misleading.
If there are any questions that are difficult for you, flag them and schedule time to review them at the end.

To learn more about the certification program, make sure to subscribe to the harness-community YT channel and to follow on GitHub for more updates on the exam! 🤩

The DevOps Roadmap: Containerization, Containers and why do you need them?

Hrittik Roy — Wed, 11 May 2022 11:39:29 +0000

Containers are a popular term in the industry and help developers develop and deploy apps a lot faster. While by using virtualization, you can run various operating systems on your hardware containerization to run multiple instances or deploy multiple applications using the same operating system on a single virtual machine or server.

This capacity to run multiple applications on the same resource summarizes into making our development life cycle efficient. In this post, we would summarize everything you must know about these unfamiliar words and how exactly containerization makes our life easy.

Let’s dive in!

What is Containerization?

Containerization is simply packaging all the required environment, libraries, frameworks, directories, and application code to create a container. Citrix defined it as:

Containerization is defined as a form of operating system virtualization, through which applications are run in isolated user spaces called containers, all using the same shared operating system (OS). A container is essentially a fully packaged and portable computing environment that already has all the necessary packages to run the application inside.

Simply put, the process of creating containers is called containerization.

What is a Container?

The word container has its origin form contain, and it does the same thing, i.e., it contains everything you need to run an application.

Containers packages and contains code and all its dependencies, so the application runs quickly, portably, and reliably from one computing environment to another.

How does a Container work?

Containers run via a containerization engine over a single host operating system by sharing the operating system kernel with other containers. This sharing is achieved by limiting parts of the OS to a read-only mode. This sharing makes containers extremely lightweight on the resources as you don’t need to configure a new operating system for each new container.

Docker a containerization engine running multiple containers. Image Credits: Docker

As you see, containers differ a lot from virtual environments (you’d need a separate OS per system here), and if you’d want to know more, we have put a blog discussing the core differences elaborately.

We would also discuss why VMs are more secure than containers and other drawbacks in the same blog.🤯

Why do you need Containers?

Development is a complicated task, and as a developer, you need to tackle issues and keep in mind many things while you develop. Containers help you keep your focus on the code and care less about other substances like the environment during development.

A few reasons you should use a container to make your dev life easy are:

Consistent Environment

Development is a critical task, and you must take into account a lot of factors ranging from libraries, frameworks to network configuration or directory management while deploying the application. Think about how different Linux and windows manage their directories.

Problems arise when the supporting software environment is not identical, says Docker creator Solomon Hykes. “You’re going to test using Python 2.7, and then it’s going to run on Python 3 in production and something weird will happen. Or you’ll rely on the behavior of a certain version of an SSL library and another one will be installed. You’ll run your tests on Debian and production is on Red Hat and all sorts of weird things happen.”

Containers have the opportunity for developers to build predictable environments isolated from other applications. The application’s software dependencies can also be contained in containers, such as particular versions of programming language runtimes and other software libraries.

This all turns in favor of a developer. The focus stays on quality code and not on the bugs that might creep while migrating code from his system to the server.

Less overhead

A container can be just tens of megabytes in size, while it may be several gigabytes in size for a virtual machine with its entire operating system. Because of this, far more containers than virtual machines will hold a single server, which boils down to consuming less resources for one individual container.

Just In Time

It can take several minutes for virtual machines to boot up their operating systems and begin running their host applications. At the same time, it is possible to start containerized applications almost instantly. That implies that when they are needed, containers can be instantiated in a “just in time” mode and can disappear when they are no longer required, freeing up resources on their hosts.

Modularity

Containers can split applications into modules (such as the database, the application front end, and so on) instead of running an entire complex application within a single container.

This is the method of so-called microservices. It is simpler to handle applications designed in this way because each module is relatively easy, and improvements can be made to modules without the whole application needing to be rebuilt. Since containers are so lightweight, it is possible to instantiate individual modules (or microservices) only when required and almost immediately accessible.

Run Anywhere

Containers can run almost anywhere, making development and deployment much easier: on Linux, Windows, and Mac operating systems; on virtual or bare metal machines; on the developer’s machine or on-site data centers; and, of course, in the public cloud.

Container images like that of docker help enhance portability as they are widespread and well supported.

You can make use of containers anywhere you want to run your apps.

What is Container Image?

To rebuild a container, you need some file or a template that contains the instructions of what the replication must include.

Container images are the same templates that help you to rebuild a container. The template consists of unchangeable static files that can be shared, and upon building, using the shared image yields a similar container.

Docker Image is an example for Container Image used to build containers.

An image comes in handy when you need to recreate a consistent working environment within another container without dealing with traditional long and boring manual configurations.

What are the popular Container Engines?

Docker

Docker’s open-source containerization engine is the first and still most popular container technology among various competitors. Docker works with most commercial/enterprise products, as well as many open-source tools.

You can read more about docker and docker images in this curated beginner-friendly post.

CRI-O

CRI-O, a lightweight alternative to docker, allows you to run containers without any unnecessary code or configuration, directly from Kubernetes, a container management system.

Kata Containers

Kata Containers is an open-source container runtime with lightweight virtual machines that feel and function like containers but use hardware virtualization technology as a second layer of protection to provide more robust workload isolation.

Microsoft Containers

Positioned as an alternative to Linux, Microsoft Containers supports Windows OS in very specific conditions. Typically, they run on a real virtual machine rather than a cluster manager like Kubernetes.

Final Thoughts ⭐

Containers are becoming essential as the direction of development is moving towards cloud-native. The advantages to using containers like the flexibility and agility are unmatched, and adoption is on the rise.

I hope this blog helped you understand containerization and containers in-depth, and if you want to experiment with containers, you must know more about the best practices used by various companies of all sizes while adopting containers.

Happy Containerizing!

Prometheus: As Simple As Possible

Hrittik Roy — Tue, 15 Mar 2022 04:34:31 +0000

Distributed systems help an organisation absorb countless benefits but at the cost of complexity. With the rise of the adoption of container orchestrators like Kubernetes, a need for monitoring and alerting systems came.

One such system is Prometheus which is famous for being “the Kubernetes monitoring solution.”

This post will explore Prometheus in a beginner way without any intricate details that generally scare a novice.

Let’s Start!

What is Prometheus?

SoundCloud first developed Prometheus in 2012 after determining that their existing monitoring tools were insufficient for their needs. However, the first release of Prometheus v1 would not be available to the public until 2016, but now it’s fully open sourced and a CNCF graduate project.
Prometheus is a time series database-based Monitoring and Alerting tool. Prometheus gathers data from apps and systems and allows you to visualise it and set up alerts. We will go deeper with this in a minute, but why do we need monitoring again?

The Stroy Behind Prometeus and Cloud

Long before you start reading this blog, there used to stay giant creatures called Monoliths. They were slow and had complexity. It was hard to understand them and harder to diagnose problems. But then we had an evolution, and micro creatures was a better option. They had a lot of benefits and had less complexity with them. But managing one creature is more demanding than managing hundreds or thousands.

You’re correct!

So, we need someone to orchestrate the herd and someone to look over or monitor the flock. The orchestrator is Kubernetes, and Prometheus manages the monitoring. Suppose your micro creatures had one telephone in their camp. Prometheus helps you look for how many hours they’re talking, whether they can connect to the telephone service and how many times they can’t call someone. All without going to every individual’s tent and enquiring!

Your micro creatures are microservices in a container (or Virtual Machine) that performs a job.

Prometheus Architecture

The Prometheus Server consists of 3 components:

Time series database (TSDB)

All of the metrics get stores in a time series database which is optimised for time-stamped data that needs measuring changes over time and also help query efficiently. So, simply it holds every generated metric for you to reference later.

The architecture of Prometheus and its components Source: Prometheus

Data Retrieval Worker

It collects (by pulling/scraping) metrics from external sources and stores them in the TSBD.

Web UI

It provides a simple web interface UI for configuring and querying the database to help you visualise your queries. We receive centralised management and configuration using the server that can track when we show new data from each unique target.

How does Prometheus work?

Querying and Scraping

Prometheus scrapes metrics from our apps and services regularly via HTTP endpoints from the target systems that have the client library installed. So, in order to collect metrics, you don’t need to install custom software or configure anything on your physical servers, nor do you need to do so on your container images.

This pull based approach use by Prometheus is better than the traditional push based approach because:

"You can run your monitoring on your laptop when developing changes. You can more easily tell if a target is down.
You can manually go to a target and inspect its health with a web browser." — Prometheus — FAQ

Service discovery

Prometheus was built from the bottom up to function in dynamic environments like Kubernetes and requires very minimal configuration when first installed. As a result, it undertakes automatic discovery of operating services in order to make a “best guess” as to what it should be monitoring.

Prometheus Exporters Source: Devconnected

Snapshots and Querying

Now, as your data is scraped what’s next? We want to store the metrics.

Prometheus stores a database record with each scraped metric’s snapshot of the data. You can use PromQL queries and the Prometheus web UI, or other tools like Grafana to explore how metric data evolves over time by querying and analysing metric data snapshots.

Also, you can label your metrics to manage the metrics. Now you don’t have to worry about searching the metrics name manually after changes because you can search them by labels. How cool is that?

Different Type of Metrics

Metrics is the specific set of data from an endpoint. It consists of TYPE and HELP Attributes which are described below.

HELP

For a token like HELP, it is expected that another token would follow, which is the metric name. The docstring for the metric name consists of all of the tokens that are left over. HELP lines should be composed of a metric name at a time.

TYPE

There are two more tokens required if the token is TYPE. The first identifier is the metric name, and the second identifier is a qualifier specifying what kind of metric it is (for example, a counter, gauge, histogram, summary, or type unknown). A single TYPE line is allowed for each metric name. Metric names should have their TYPE lines, which specify the metric’s unit of measurement, positioned first before any metric samples are given. If a metric name does not have a TYPE line, the type is left as untyped.

Metrics Type

Counter

It simply counts the metrics. We’ll count these by things like the number of errors or the number of requests. This type is recommended unless your metric value can fluctuate and go down.

Gauge

It is ideal for metrics that fluctuate, like CPU utilisation.

Histogram

A histogram is used to measure information by counting information in specific observed buckets. Additionally, it presents a complete account of the total amount of all observed values, so it is one of the most challenging types of metrics to read.

Summary

For every observation, a summary takes a sample (usually things like request durations and response sizes). It likewise gives you the total number of observations and a list of all observed values, but it is capable of generating a customisable number of quantiles across a sliding time window.

You can only utilise four sorts of metrics (Counter, Gauge, Summary and Histogram), therefore choose the best one for the purpose.

Fail Proofs

When “pulling” metrics is not possible (for example, short-lived jobs that will not live long enough to be scraped), Prometheus provides a Pushgateway that allows applications to still push metric data if necessary. Essentially, we get the best of both worlds.

Alerting

You can use PromQL, the querying language used by prometheus to retrieve metrics via the database and use Prometheus Web UI, API clients and Grafana to visualise. Setting up external alerting services like pagerduty and email is even possible via Alertmanager.

Final Thoughts

Prometheus is popular, and everyone in the CNCF landscape knows this. The features help it become one of the most promising monitoring systems and beat the likes of Amazon CloudWatch, ApplicationInsights, NewRelic, which are push based platforms. But, Prometheus itself states that the pull based factor shouldn’t be the major point when considering a monitoring system. Also, even if Prometheus serves as Kubernetes best friend it’s capable of more!

If you want to get started with prometheus no other place is better than the First Steps with Prometheus documentation.

Helm: Package Manager for k8s

Hrittik Roy — Sun, 13 Feb 2022 07:31:19 +0000

Kubernetes was started inside Google to provide a layer of abstractions with containers for the modern infrastructure. Now, the technology is adopted by the masses and has become a de facto standard for any cloud native application. The open source system provides management, deployment, and scaling of your containers.

Kubernetes is hard to beat in orchestration, but one of the most significant drawbacks is its lack of reproducibility. Here comes Helm: A package manager for Kubernetes and a CNCF Graduate Project.

I was thrilled to speak at DoK Talks on the 114 Edition about Helm and how it tackles the reproducibility problem for Kubernetes. This post will be a summary of the beginner focused event which took place on 27th January 2022.

It's not required for you to go through the recordings as this is a very extensive summary and going through this will give you a basic understanding of Helm!

What’s the reproducibility problem?

After you have deployed your application with numerous objects: Deployments, Services, ConfigMaps, etc., how do you help your friend get to a similar state? Of course, you will share your YAML files with your friend. Correct?

Yes and No!

You can do the hard work of copy and pasting for a small application, but what if your application is a full-stack web app with 100s of configuration files. Can you still copy them? No, you can’t, as it’s prone to errors with that many large numbers.

Even if you send the object manifest to your friend, there’s one question still available. How would your friend convert site to site, or maybe change how much resource each application consumes? Will he go through the 100 manifests? No, that’s not scalable, prone to errors, and wastes a lot of time.

Now suppose there’s a security flaw on one of your dependencies. How will you update them? Find your YAML, or edit your live resources? Absolutely NO!

You need a saviour. You need Helm to your rescue.

What is Helm?

Helm is your saviour for the reproducibility problem, a package manager, and a CNCF graduate project. It was launched in 2016 and has seen massive adoption among organizations, individuals since then. Under the CNCF Umbrella, Helm has become the de facto package manager for your clusters.

What does Helm do?

Helm helps you to achieve reproducibility in the following ways:

Provides an easy way to deploy complex application
Provides easy way to update specific values for your deployments
Provides a way to version a particular package
Provides a way to share your templates across organisation, Internet
Provides an easy way to manage dependency
Provides an easy way to rollback changes

Architecture of Helm

The Helm Repository contains all the charts (packages) created by you or other people that you can use to reach the desired state. The Helm CLI pulls the package, unarchives it, and then converts the charts to a valid YAML, which is then pushed to Kubernetes API server, which creates a release.

Basic Components of Helm Charts

A basic Helm Chart has the following structure:

package-name/

charts/

templates/

Chart.yaml

values.yaml

charts/: This directory can be used to store manually maintained chart dependencies.

templates/: These contain the template files which would be used to create the final manifest after combining with values.yaml.

Chart.yaml: This file contains information about the chart, such as the name and version of the chart, the maintainer, dependencies, a related website, and search terms.

values.yaml: This contains the default configuration for your charts. You can edit this for updating values and remove the complexity of finding specific editable items in the different manifests.

The below example shows a deployent.yaml from templates being rendered with the custom values from values.yaml to produce the valid YAML.

Helm Templates Creation with values.yaml

How to edit the default values?

Manually pulling the charts and unzipping it to edit your values.yaml is not that straightforward. We have Portainer here, which does all the heavy lifting for you and helps you to get straight to editing the default configuration.

First Navigate to Helm from the Menu and then add a repository. In a standard installation of Helm, you need to add a repository, but with Portainer, you get Bitnami by default and can add more when required. Select a namespace and an application name. Then you need to select the chart you want to deploy to your cluster.

Once selected, you can navigate to a chart, and Portainer will load the values on your dashboard for you to edit. Editing values are straightforward and abstract the complexity you had to deal with manual installation and initialization via a simple GUI.

The Install button would install your chart with the specified values to your cluster. If you want to go with the default values, click on Install without editing the values.

After installation, Portainer detects and shows you Published URLs to access your applications and secrets for you to access default passwords. Forget digging through commands to get to your Services and Secrets.

Final Thoughts

Helm abstract the complexity of installing applications to your cluster. Portainer abstracts the complexity of managing your cluster. This post went through how Portainer can help you simplify your Kubernetes workflows with Helms.

Try Portainer now and learn more about the different ways to streamline managing Kubernetes with our documentation.

Portainer Documentation for Helm here: https://docs.portainer.io/v/ce-2.11/user/kubernetes/helm

Recordings are here:

DVC (Git For Data): A Complete Intro

Hrittik Roy — Thu, 09 Sep 2021 11:13:22 +0000

As a data scientist or ML engineer, have you ever faced the inconvenience of experimenting with the model? When we train the model, the model file is generated. Now, if you want to experiment with some different parameters or data, generally people rename the existing model file and train the model again and this process goes on.

The final directory looks something as shown in the below figure 😁. The same thing goes with data as well.

Fig 1: Typical models dir after experimentation

But if we talk about code, we don't have this problem there because we have Git to version the code. I mean that I can create a separate branch for changing some code to see its behaviour without altering the previous one. What if we can get a Git kind of versioning control system for data and model? Wouldn't that be amazing? Here comes the DVC (Data Version Control) which exactly does the thing which solves this problem.

Although, DVC is much more than just data and model versioning. It also helps create end-to-end pipelines, capturing metrics related to pipelines and experimenting with ML models. But in this blog, we will see the data versioning aspect of DVC.

Installing DVC

DVC can be used as a typical Python library and can be installed using package managers like pip or conda.

To install DVC using pip, you can execute below command in the python supported terminal.

$ pip install dvc

To install DVC using conda, you need to execute below commands in conda terminal.

$ conda install -c conda-forge mamba # installs much faster than conda
$ mamba install -c conda-forge dvc

Apart from these conventional installation techniques, DVC can be installed in other different ways too as described here on the official documentation website.

Getting Started

Code versioning is not really a new thing for us because we use Git in our daily life but the problem with Git is it becomes extremely slow when the tracking file size is huge as 100 GB. That is where DVC comes in. Git tracks the changes in code whereas DVC tracks the changes in data as well as models.

The foundation of DVC consists of a few commands you can run along with Git to track large files, directories, or ML model files. In short, you can call DVC "Git for data".

Let us understand data versioning by dvc using a simple demo project. Let us assume that we want to create an end-to-end pipeline to prepare, train and evaluate the MNIST dataset and we have code as well as data available and structured as shown in the below figure.

Fig 2 MNIST pipeline file structure.

By observing the file structure, we can see the MNIST dataset is stored in the data directory as a data.xml file and the code corresponding to every stage of the pipeline is stored in the src directory. Now as we understand, the code tracking (src) is the responsibility of Git and the data tracking (data) is the responsibility of DVC.

Before we version the data, we need to initialise the repository as a DVC repository. Just like we run command git init to initialise the current directory as a Git repository, we need to execute the below command which will initialise the current directory as a DVC repository.

$ dvc init

This command will create a new directory .dvc just like git init creates .git directory. Two files .gitignore and config will be generated in .dvc directory. Now, we have successfully initialized this as DVC repository.

Start Tracking

In Git, if we want it to track the changes in the code, we run a git add command which adds the code changes in the local repository. DVC has a similar command as shown below to start tracking the data files.

$ dvc add data/data.xml

Note here that a dvc add command accepts arguments which are data or model file names that you want DVC to track. All the filenames must be space-separated.

By executing this command, DVC will create two files data/data.xml.dvc and data/.gitignore.

File data/.gitignore looks something like this:

/data.xml

It means that it forces Git not to track original data.xml file which is expected because the tracking of data.xml is now responsibility of DVC and not Git.

All files ending with extension .dvc are special files which contains the information about where the actual data files will be stored and cached. If you take a look at data/data.xml.dvc, it just stores an MD5 signature as shown below.

md5: a7cd139231cc35ed63541ce3829b96db

Now looking at the complete picture, Git will not track data/data.xml because it is added in data/.gitignore but Git will track data/data.xml.dvc which will have information about where the actual data.xml is stored.

Now the Git tracked 2 files mentioned above will be stored in Git remote repository on Github, BitBucket or Gitlab and DVC tracked data file will be stored in DVC remote repository on any file system that we are going to describe in next section.

Once we do this, the directory structure for MNIST pipeline project looks something like this.

Fig 3 File structure after dvc add

Storing & Sharing

The actual data files (which are not tracked by Git) will be stored in any kind of file system which can also be called DVC repository. File system for DVC repository can be AWS S3 bucket, Google Drive, Google storage bucket, Azure storage, Object Storage Service or any custom created file system. Depending on where you will store the data, you will need to install external dependencies like dvc-s3, dvc-azure, dvc-gdrive, dvc-gs, dvc-oss, dvc-ssh. You can know more about installation here on the documentation website.

To push the data files in DVC repository, first we need to configure the remote origin of the repository where the data will be stored. We can do that using below commands.

$ dvc remote add -d storage s3://mybucket/dvcstore
$ git add .dvc/config
$ git commit -m "Configure remote storage"

dvc remote add command adds the remote origin of the DVC repository where the data will be stored. Executing this command will actually add this origin information in .dvc/config file which Git tracks. That is why we also need to commit the changes done in .dvc/config file as well.

Now, adding the remote origin doesn't automatically push the data in DVC remote repository. We can push the entire data file using below command:

$ dvc push

It pushes all the data and model files on which dvc add command is applied.

Retrieving

There is a reason behind dvc remote add stores remote origin information in .dvc/config. It does so because when next time somebody clones this Git repository, It will not have any data. It will only have MD5 signature of the data in corresponding .dvc file and it will also have DVC remote repository location information stored in .dvc/config file. So, the complete data can be pulled using below command:

$ dvc pull

It pulls all the data and model files stored in DVC remote repository.

ML Pipeline & Versioning

While working with DVC, we have to create two files dvc.yaml which contains information about the stages the ML pipeline will have and params.yaml which contains the parameters that different stages of the ML pipeline would use. You can take a look at the structure of both files on the example repository.

DVC is so flexible that you can use command dvc exp run to run the complete pipeline with just single command. The command is not only limited to this.

If you want to run your experiments with different parameters, you can mention the values of the parameters which are defined in params.yaml as shown below.

$ dvc exp run -S prepare.test_split=0.2 -S train.learning_rate=0.0001

Note here that whenever you run the experiments using the above command, DVC caches the output so well that it will only run those states which are affected by the parameters. Moreover, if you think one of the stages in the pipeline are too much time consuming and you want to checkpoint it so that in case the program crashes, it can resume from where it crashed, you can define checkpoint: true in the output of that stage in dvc.yaml file and it will keep checkpointing your results for you.

DVC keeps track of all the experiments that you perform and even you can see the history of the previously run experiments using dvc exp show. It will show you all the previously run experiments. You can filter this history as well. For example, dvc exp show --include-params=train will only show you the past experiments where any of the parameters of train stage was changed. Not just that, but you can also see the difference between two experiments' parameters and their resultant metrics as well using dvc exp diff exp-1dad0 exp-1df77. where exp-1dad0 and exp-1df77 are tags of two experiments as shown in history given by dvc exp show. Moreover, You can navigate to any previously run experiment using dvc exp apply. For example, if you run command dvc exp apply exp-1dad0, then the files will be changed as per the parameter changes specified in that particular experiment.

DVC gives us the flexibility to commit the experiment into a separate Git branch. As shown in the below command, using dvc exp branch command you can commit the experiment exp-1dad0 to separate branch my_branch.

$ dvc exp branch exp-1dad0 my_branch

Alternatively, you can also commit different experiments in different branches just by switching the branches as described below.

As we already understand switching between different versions of code by changing Git branch. Similarly, we can also change the version of the data as well.

$ git checkout experiment_v2
$ dvc checkout data/data.xml

We can observe that changing the version of data is now as simple as changing the Git branch. Note here that to switch to another version of data, we first need to switch to that particular Git branch where that version of data resides. Also, note here that you must apply dvc checkout to those files for which you want to switch the data version since git checkout only changes the code version.

How data versioning simplifies experimentation?

As described in the beginning of this blog, we don't need to keep multiple copies of model files anymore and keep renaming them. Now, we can have single model file which can be versioned using DVC during experimentation. Below image explains the same.

Fig 4 Model versioning for experimentation

Let us suppose we are working in Git branch v1 where the hyper-parameter n_estimators to train model has value 50 and we have trained a model which is stored as model.pkl. Now, we want to experiment with value 100 for the same hyper-parameter n_estimators. So, we don't need to rename the old model file anymore. We will switch to another Git branch v2 using git checkout v2 command and we will switch to different model file version using dvc checkout model.pkl. We will change the value of hyper-parameter n_estimators to 100 now in code in v2 branch and we can train the model again. The new model.pkl file generated in v2 branch is corresponding to n_estimators = 100 and model.pkl file in v1 branch is corresponding to n_estimators = 50.

Although, this is very small experiment but the same thing can be applied in complex experimentation process. And with data versioning, the experimentation process becomes very flexible. We don't need to worry about tracking of data and model files.

Large datasets versioning

When the dataset is too large, we need very efficient mechanism in terms of space and performance to share different versions of data. That is why it is suggested to store the data in shared volume or on an external system. DVC supports both of these mechanisms as described below.

A shared cache can be setup to store, version and access lot of data on a large shared volume efficiently.
As we described earlier, the more advanced approach is to store and version the data directly into remote storage (ex. S3 bucket, Google Drive, GCP storage bucket etc.). You can look here to know more about how to configure external data storage for DVC.

Conclusion

With this, I hope you were able to get a quick overview of how DVC does data versioning. DVC also has features to streamline ML reproducibility and support cloud provisioning through cloud providers. This would come in handy for the entire MLOps community as now they could interact with each other without worrying about monolithic tools.

If you like to know more about how to do it and get your hands dirty, the documentations are a great way to start.

If you prefer videos then they have a youtube channel with detailed tutorials!

Happy Versioning!