DEV Community: KCD Chennai

Updates about KCD Chennai 2024

KCD Chennai — Tue, 08 Oct 2024 06:13:25 +0000

Hey KCD Chennai Community! 👋

Many of you have been asking about KCD Chennai 2024. There's been some exciting developments in the KCD world this year! 🎉

Firstly, Cloud Native Computing Foundation (CNCF) has implemented a new guideline for KCD events within the same country/region. To ensure a vibrant ecosystem and avoid scheduling conflicts, KCDs need to be at least 2 months apart, and not held within 2 months of other major CNCF/LF events. 🥇

2024 has seen the incredible launch of brand new KCDs in India! KCD Kerala, KCD Pune and KCD Hyderabad. 🎈

Additionally, India is hosting its very first dedicated KubeCon event in New Delhi from December 10th-12th, 2024! This flagship event deserves the spotlight, so we've made the difficult decision to forgo KCD Chennai 2024. ✨

This decision allows other KCD communities in India to flourish and grow, fostering collaboration and knowledge sharing – the core values of open source! 🦄

Open-source communities thrive on the principles of openness, collaboration, and cooperation. These values are fundamental to nurturing innovation, knowledge sharing and community growth. CNCF and hence the KCD Chennai community exemplifies these principles, creating a space where other communities thrive as well. Together, we're stronger! 🤝

We'll keep you updated on KCD Chennai 2025 soon. Stay tuned! 🔊

Storing and Searching TeraByte Scale logs in SnappyFlow with Secondary Storage

KCD Chennai — Wed, 19 Jul 2023 07:22:18 +0000

The premise

Log management in modern organizations

In most enterprises today, big, and small, it is not uncommon to have a tech stack comprising 50 or more different technologies across their applications and infrastructure, and this number is likely to increase as companies embrace microservices, multi/hybrid clouds, and containerization.

All these individual components generate logs, and lots of the. These logs serve as invaluable sources of information, providing insights into the health of individual components, transaction details, timestamps, and other critical data. By analyzing these logs, SREs and DevOps engineers can gain a comprehensive understanding of their systems, diagnose issues promptly, and optimize performance. Development teams rely on these logs to understand and address issues before they affect customers and businesses.

Each log entry represents a specific event that happened at a precise moment in time, allowing for accurate tracking and analysis of system behavior. For instance, when a fault occurs, logs enable developers to identify errors and look for related logs, system performance metrics, and application traces and drill down to the exact line of code to troubleshoot.

Challenges in managing Terabyte and Petabyte scale logs

As more logs get generated, it quickly becomes a “storage” and “search” problem. Although individual logs are tiny – just a few Bytes, the cumulative volume of logs across your stack, multiplied over several days can quickly reach Terabytes or Petabytes. Efficient search and storage mechanisms become crucial for developers and engineers in handling this large log volume.

Log retention, defines how long the logs are stored and in turn determines the total log volume. Factors such as security, regulatory compliances and cost have to be taken into account, to arrive at an optimal log retention period. Striking a balance between cost-effectiveness and fulfilling operational, analytical, and regulatory needs is key to optimizing log storage.

However, retaining logs for extended periods, spanning months or years, introduces complications. The common approach of compressing and storing logs in cost-effective solutions like AWS Glacier hinders real-time log retrieval and search capabilities. While suitable for auditing, this method limits developers' ability to efficiently analyze and troubleshoot logs in a timely manner.

To overcome these limitations, engineers require a solution that allows quick access to archived logs without sacrificing real-time search functionality. This ensures developers can effectively analyze logs, even in long-term retention scenarios, enabling timely analysis and troubleshooting.

Introducing SnappyFlow's Secondary storage feature

SnappyFlow provides an elegant solution to ingest, store and search large volumes of logs for extended periods of time using what we call the “Secondary Storage” feature. Secondary Storage allows massive streams of logs to be ingested and stored in a cost-efficient manner without losing the ability to easily search logs.

So, is there a Primary Storage?

Yes. By default, all logs sent to SnappyFlow are stored in a “Primary Storage”. Think of Primary storage as fast and responsive storage system capable of handling a large volume of searches at lightning-fast speeds. These are typically fast SSD-type storages and are as expected, expensive.

How does Secondary Storage work?

Different log sources can be configured to send logs to Primary Storage, Secondary Storage, or both.

This is available under Log Management > Manage Logs. In the screenshot below, you can see a list of rules for the project apmmanager-opensearch. Note that in this example, you are looking at project-level rules. Similar views are available at Application and Profile levels.

Project level view of Secondary Storage Rules for the project apmmanager-opensearch

Application level view of Secondary Storage Rules for the project apmmanager-k8

The default rules are set to send all logs to both Primary and Secondary storage with a retention period of 7 days and 30 days respectively. New rules can be added using the Add Rule button and it takes a couple of minutes for these rules to get activated.

Once the rules are applied, these can be viewed under Applied Rules

Adding a new secondary storage rule for server logs

Searching logs in Secondary Storage

Search for logs in secondary storage is available under the respective applications. To access go to any application and select Log Management > Secondary Storage

In the Secondary Storage page, live search is available for data from the last 30 minutes. Logs can be filtered using the log type or using simple search strings. The Search History tab allows you to create search jobs and these jobs run in the background. Once a search job is completed, the search results can be accessed instantly any time.

Live search and search history for secondary storage logs

Limitations

All logs in secondary storage can be searched in real-time (only the last 30 minutes) or search jobs can be set up and the indexed results can be accessed instantly.
It is not possible to create dashboard out of logs in secondary storage.
Secondary storage logs are not part of the usual log workflow i.e trace to log

An illustration of the benefit of using secondary storage for log management

Secondary storage can compress logs by upto 40 times the original size and can provide significant cost benefits. Consider a medium-sized SnappyFlow deployment with an average daily ingest volume of 1 TB with a retention of 15 days. At any given time, 15 TB of primary storage is required simply to hold this data. If we were to use Secondary storage to move say 60% of all the logs, we would need to incrementally store only 400 GB of logs on a daily basis and this works out to 6 TB of primary storage.

At the time of writing, the cost of EBS storage on AWS is

15 TB, GP3 - $2001/mo

6 TB, GBP - $800/mo

Here, there is a straightforward reduction in monthly costs of $1200 simply by routing 60% of logs to Secondary storage. Do note that there will be an additional cost of storing data in Secondary Storage but this is significantly lower as we will be using an object-based storage service like S3.

With a compression factor of 40x and a log retention period of 60 days, total log volume in secondary storage will be (1 TB/day * 60% * 60 days) / 40 = 0.9 TB

S3 storage cost is just $20 for ~1TB of compressed logs.

Explore Secondary Storage today!

Secondary Storage features are available to all SaaS and Self Hosted Turbo customers. Secondary Storage is the easiest and simplest way to control your logs storage costs and stay compliant to long term regulatory and security requirements. What’s more? This feature comes at no extra cost.

To try SnappyFlow, start your 14-day free trial today.

Kubernetes Monitoring Simplified

KCD Chennai — Wed, 19 Jul 2023 06:54:17 +0000

Introduction

Today, organizations are transitioning from monolithic architecture to microservice architecture. When we speak of microservice architecture, there is no way to ignore Kubernetes (K8’s). K8’s is the leading open-source Container Orchestration Platform, and it holds a prominent position for certain reasons, such as: Robust, Scalability, High Availability, Portability, Multi-Cloud Support, Self-Healing, Auto-Scaling, Declarative Configuration, Rollbacks, Service Discovery, Load Balancing, among other features.

We all know Kubernetes is a powerful tool, but the question here is, how do I get the most out of it? The answer is simple: by proactively monitoring your Kubernetes cluster.

Challenges in Monitoring Kubernetes Clusters

Monitoring K8’s clusters can be a complicated task. Here are few difficulties faced by SRE’s.

Complexity: Kubernetes operates in a multi-layer environment, requiring the collection and correlation of metrics from pods, nodes, containers, and applications to obtain an overall performance metric of the cluster.

Security: Monitoring tools collect sensitive data from the Kubernetes cluster, raising security concerns that need to be addressed to protect the cluster data.

Data Flow: As the Kubernetes cluster grows, whether it's on the cloud or on-premises, tracing the data flow between endpoints becomes increasingly difficult.

Ephemerality: Kubernetes scales up and down based on demand. Pods created during scale-up disappear when no longer required. To avoid data loss, monitoring tools must collect metrics from various resources such as deployments, daemon sets, and jobs.

Benefits of Monitoring Kubernetes Clusters

Despite the challenges, monitoring K8’s environment can lead to significant benefits and improvements:

Enhanced Visibility: Monitoring Kubernetes clusters helps you gain enhanced visibility about overall health of your system and its components.

Proactive Issue Detection: Monitoring Kubernetes clusters enables you to proactively detect issues such as application failures, abnormal behaviours, and performance degradation. With the help of this you can prevent potential downtime and service disruptions.

Resource Utilization: By tracking the resource usage of your containers, pods, and nodes, you can fine-tune resource allocation, enhance efficiency, reduce costs, and maximize the utilization of the infrastructure.

Performance Optimization: Monitoring Kubernetes enables you to identify the slowness in the response times, inefficient resource usage and helps you to optimize scaling specific to components and optimize network settings to improve overall system performance.

Root Cause Analysis: Monitoring Kubernetes facilitates you to pinpoint the root cause, isolate problematic pods, or nodes, and analyse relevant logs and metrics.

Here comes the next question, how do I monitor my Kubernetes cluster? Monitoring Kubernetes is challenging task, but this is where monitoring tools like SnappyFlow come in handy.

Simplify Kubernetes Monitoring with SnappyFlow

SnappyFlow is a full stack application monitoring tool. By integrating your Kubernetes cluster with Snappyflow, it starts collecting the data from your cluster and enables efficient cluster monitoring. SnappyFlow monitors various aspects of the cluster which will help you to take an informed decision. Its alerting system notifies any deviation that happens in the cluster through your preferred communication channel.

What can you monitor with SnappyFlow?

Resource Utilization: SnappyFlow monitors CPU, memory, and disk usage of nodes, pods, and containers to identify resource bottlenecks. This helps to ensure efficient utilization of the resources and prevents overloading.

Pod Health: SnappyFlow tracks the status and health of individual pods, including their readiness and liveness probes. It monitors the pod failures, crashes, and restarts.

Cluster Capacity: SnappyFlow keeps an eye on cluster capacity to avoid resource exhaustion. It monitors the number of nodes, available resources, Daemon set, Deployment, Replica set, stateful set and the ability to schedule new pods. It also monitors how many critical and warning events are occurring in the cluster.

Network Performance: SnappyFlow monitors network traffic, latency, and throughput at node level.

Persistent Volumes: SnappyFlow monitors the status and capacity of persistent volumes (PVs) and their associated claims (PVCs). Ensure that storage resources are available and accessible as required. It also monitors Read and Write operations.

Service Discovery and Load Balancing: SnappyFlow monitors the health and availability of Kubernetes services and their endpoints. Track load balancing across pods to ensure even distribution of traffic.

Kubernetes API Server: SnappyFlow keeps an eye on the API server's performance, latency, and response times. Monitors for any errors, throttling, or potential bottlenecks in API communication.

Logging and Event Monitoring: SnappyFlow sets up centralized logging and monitoring to capture container logs, Kubernetes events, and system-level metrics. check for issues related to image pulling, container startup, or resource allocation. This enables quick troubleshooting and identification of issues.

Node Summary

Cluster Resource Utilization

Kubelet operation details

Conclusion

In summary, SnappyFlow offers a simplified and effective solution for monitoring Kubernetes clusters. By leveraging SnappyFlow, you can ensure the health and performance of your Kubernetes cluster while reducing the complexity and effort involved in monitoring.

KCD Chennai 2023 Blogathon Contest 🎉

KCD Chennai — Sat, 08 Apr 2023 11:21:12 +0000

Blogathon

Either write something worth reading or do something worth writing. --Benjamin Franklin

Blogging is a powerful means of sharing knowledge and promoting your thoughts and views.As of 2022, there are more than 572 million blogs on the internet (and this number is constantly growing). Over 7 million blog posts are published daily. 77% of people regularly read blogs online.[Source:firstsiteguide.com]

Kubernetes Community Days (KCD) Chennai 2023 is hosting a Blogathon. If you are a seasoned blogger, or an occasional writer, or the one searching for right reason to start blogging, we invite you to participate!! Unleash the writer in you.

How do I Participate?

Register yourself here
Upon registration you'll receive an email containing further steps to be followed within few hours

Important Dates

17-April-2023 : Blogathon begins
19-April-2023 : Workshop on Blogging Tips
31-May-2023 : Blogging ends
20-June-2023 : Know the winners

What topics can I write about?

We invite Technical blogs on a wide variety of topics. You are allowed to write about any topic you are passionate about, as long as the content is technical.

Publishing your blog

When you create a new blog, under 'Create Post' choose 'Kubernetes Community Days Chennai' as shown below.

Rules

Blogs published during 17-April to 31-May alone would be considered. (Tip: Keep publishing as many good-quality blogs as possible)
You are allowed to publish any number of blogs; there is no restriction.
You are allowed to cross-post blogs you've already published in other blogging sites e.g. medium, hashnode, wordpress etc.
You are allowed to publish blogs you've already published in DEV under your personal account.
Published blogs would be reviewed by our team. If the blog is offensive, violates copy rights or we detect plagiarism, such blogs would be removed.

Can I promote my blogs?

Of course yes! Feel free to promote your blogs via Social media, Emails, Newsletters, WhatsApp etc. This would help increase the views and reactions!!

Contact Us

If you have further questions or need help, post it in the #kcd-chennai channel in https://cloud-native.slack.com

Fostering Innovation with a Kubernetes Platform

KCD Chennai — Thu, 26 May 2022 13:17:20 +0000

Author: Vishal Ghariwala, SUSE (Platinum sponsor, KCD Chennai 2022)

Hybrid and multi-cloud are now the established order in the tech world. According to SUSE’s recently commissioned Insight Avenue report, Why Today’s IT Leaders are Choosing Open, more than 800 IT leaders believe the biggest benefits of a hybrid and multi-cloud approach are cost-effectiveness (45%), increased flexibility and agility (44%), and being able to take advantage of best-of-breed solutions (35%).

Yet unlocking these benefits is difficult when many IT leaders haven’t adequately factored the prevalence of containers and Kubernetes in their multi-cloud strategy. As a result, while they will undoubtedly have differing Kubernetes distributions within their environment, they may lack a unified platform for managing, governing and having visibility of the varied distributions.

In the same way that Linux became the data center’s operating system, Kubernetes is now widely regarded as the operating system of the hybrid and multi-cloud – ultimately because Kubernetes makes it easier to manage software complexity.

In the early days of Kubernetes, companies would have experimented with their own DIY Kubernetes stack to run their cloud native applications. However, as these enterprise applications became more complex, it became harder to manage them. Today, the cloud and container market has matured significantly, so we believe it’s time for enterprises to rethink their Kubernetes approach.

What do IT leaders want from a Kubernetes platform?

IT leaders love Kubernetes because it fosters innovation. It helps to significantly increase the agility and efficiency of their software development teams, enabling them to reduce the time and complexity associated with putting differentiated applications into production.

According to our report, 42% of organisations currently run containers for production workloads – with a further 41% planning to do so in the next 12 months. 57% of organisations that are running containers for production workloads use Kubernetes.

There has been an ongoing evolution in the build vs buy debate regarding a Kubernetes platform. Our survey found that 66% of IT leaders now prefer a commercially curated and supported distribution of an open-source Kubernetes platform vs a homegrown Kubernetes platform. This is a significant shift from just last year, where 87% of IT leaders still preferred a DIY approach. What has changed? Is the growing complexity of applications becoming too difficult for them to manage? Is it due to a shortage of Kubernetes skill sets? Is it due to implementation cost?

What we do know is that IT leaders are embracing open source. When we asked IT leaders for the factors they look for in a Kubernetes platform, the top three are:

100% open source (36%)
support for multi-cluster and edge deployments (34%)
ease of installation (34%).

Development and operations teams are pivotal to innovation. However, it is also quite well known that the priorities of these two teams can be diametrically opposed, for good reasons. Development teams want to focus on writing code and rolling out their applications quickly. Operations teams work hard to manage stability, security and control their computing environments. We call this the agility-stability paradox. How do we create a balance between the freedom to innovate and ease of management and governance?

Kubernetes can help to cope with these competing desires between development and operations teams by decoupling application development and operational stability. As a result, development teams can build what they need, optimised for innovation while still aligning to continuous delivery and automation processes defined by the operations teams. To do this successfully however, companies need to leverage an enterprise Kubernetes management platform. Such a platform must be able to meet key requirements that will foster innovation and collaboration between development and operations teams.

The platform should provide development teams with a rich catalogue of services for building, deploying, and scaling containerized applications, app packaging, CI/CD, logging, monitoring, and service mesh. It should also empower operations teams to automate processes and apply a consistent set of operational, governance and security policies for their Kubernetes clusters which may be running on any CNCF-certified Kubernetes distribution, in the data center, in the cloud or at the edge.

Where are you on your Kubernetes adoption journey, and which factors are most important for you in a Kubernetes platform? How do you envision development and operations teams using a Kubernetes platform in your organisation? What priority do you place on the value of open source solutions for giving you the freedom to innovate everywhere?

About the author

Vishal Ghariwala is the Chief Technology Officer for SUSE for the APJ and Greater China regions. In this capacity, he engages with customer and partner executives across the region, and is responsible for growing SUSE’s mindshare by being the executive technical voice to the market, press, and analysts. He also supports the global Office of the CTO to assess relevant industry, market and technology trends and identify opportunities aligned with the company’s strategy.

Prior to joining SUSE, Vishal was the Director for Cloud Native Applications at Red Hat where he led a team of senior technologists responsible for driving the growth and adoption of the Red Hat OpenShift, API Management, Integration and Business Automation portfolios across the Asia Pacific region.

Vishal has over 20 years of experience in the IT industry and holds a Bachelor’s Degree in Electrical and Electronic Engineering from the Nanyang Technological University in Singapore.

Preparing for the new needs of edge computing

KCD Chennai — Thu, 26 May 2022 13:03:25 +0000

Author: Vishal Ghariwala, SUSE (Platinum sponsor, KCD Chennai 2022)

The proliferation of distributed and data-intensive workloads is spurring rapid growth in edge computing. According to Gartner, around 10 percent of enterprise-generated data is created and processed outside a traditional centralised data center or cloud, with this figure expected to reach 75% by 2025. The adoption of intelligent edge applications is already transforming a range of industries through autonomous vehicles, industrial robotics, and industrial IoT devices, as the number of global IoT connections is set to reach 83B by 2024.

In SUSE’s recent global survey of more than 800 IT leaders, 93% say they are excited or interested in the possibilities that edge computing presents, and 79% agree that COVID-19 has accelerated moves to edge computing. As part of our report: Why Today’s IT Leaders are Choosing Open, these IT leaders told us the most compelling benefits of edge computing are:

reliability (45%)
enhanced security and privacy (43%)
enhancing the entire IT ecosystem (41%)
fostering the use of innovative new IT services (40%)
real-time insights (39%)

As edge computing becomes more ubiquitous, there are several questions that IT leaders will need to ask to reap its benefits:

Where should edge solutions replace/augment public clouds, as they may be too far away to provide the required latency?
How can I create consistency across traditional and edge environments in cloud-native development and deployment?
Is my infrastructure platform future-ready for the rapid growth of both edge computing and hybrid and multi-cloud use cases?

Success criteria for edge-related innovations

Innovation in edge computing is leading to digital transformation across industries with autonomous vehicles and equipment, industrial robotics, and IoT devices – each connecting operational functions and outputs with organisational management technologies to create an intelligent and interconnected network. There are three key factors undergirding the success of many of the modern edge-related innovations.

The first factor pertains to defining how the edge infrastructure will intersect with existing on-premises and cloud infrastructure. This will largely depend on the use case and can be segmented into three logical tiers namely the near edge, far edge and tiny edge. Near edge is closest to centralized services such as university compute facilities. Far edge is furthest from the data center and close to the edge device. Tiny edge is the edge device itself which includes sensors and actuators. As an example, the rapid rise in remote working and remote learning over the past year has clearly shown the importance of good internet latency and bandwidth especially for those living in rural areas. In such a scenario, service providers can deploy a near-edge infrastructure as close as possible to their customers.

The second factor is about having a consistent way to develop, deploy and maintain your cloud native applications regardless of where they will be living be it in the data center, cloud or at the edge. You should also be able to perform all the routine maintenance functions like applying patches, updates, configuration changes and roll backs seamlessly. Kubernetes will be a central technology here as it helps to abstract underlying heterogenous infrastructures and provides common APIs to manage software complexity across these environments.

The final factor is around building an infrastructure that is able to support the diverse set of edge use cases in addition to existing hybrid and multi cloud applications. Compute resources at the edge are generally scarce and not always connected to the Internet. Hence we will need to leverage lightweight cloud-native technology stacks that are fit for resource constrained environments and able to operate in remote locations.

SUSE edge solutions

SUSE provides Kubernetes-ready open source, edge solutions for full-lifecycle edge infrastructure management. This includes:

A secure and lightweight Linux operating system
A secure and lightweight edge-ready Kubernetes distribution
A distributed, software-defined storage platform for Kubernetes that can run anywhere
GitOps tooling for continuous delivery of containerized applications at the edge

With SUSE edge solutions, customers can orchestrate containerized workloads at the edge. The solutions employ a unified management layer to empower technology leaders to understand where and how their containerized applications are running across traditional, cloud and edge infrastructures. Finally, SUSE edge solutions are architected using lightweight technologies that are fit for resource-constrained and remote environments.

Where are you on your edge computing adoption journey?
Which edge computing use cases are relevant to your organization?
How is edge computing influencing your cloud strategy?

About the author

Vishal Ghariwala is the Chief Technology Officer for SUSE for the APJ and Greater China regions. In this capacity, he engages with customer and partner executives across in the region, and is responsible for growing SUSE’s mindshare by being the executive technical voice to the market, press, and analysts. He also supports the global Office of the CTO to assess relevant industry, market and technology trends and identify opportunities aligned with the company’s strategy.
Prior to joining SUSE, Vishal was the Director for Cloud Native Applications at Red Hat where he led a team of senior technologists responsible for driving the growth and adoption of the Red Hat OpenShift, API Management, Integration and Business Automation portfolios across the Asia Pacific region.

Vishal has over 20 years of experience in the IT industry and holds a Bachelor’s Degree in Electrical and Electronic Engineering from the Nanyang Technological University in Singapore.

Top 7 reasons to adopt microservices architecture

KCD Chennai — Sat, 21 May 2022 10:51:55 +0000

Author: Abhinav Dubey, Devtron Labs (Platinum sponsor, KCD Chennai 2022)

Microservice is an architectural style that structures an application as a collection of nearly independent, loosely coupled components that are developed and deployed independently unlike monolith architecture. When a software is designed, architects check for several different architectural styles suitable for their application that need to be followed by developers, while building the application. For a long time, monolithic architectures were used, where the applications were designed, developed, and deployed as a single unit. Even today, many of the enterprises use monolithic architecture.

With the continuous evolution of technology, the need to serve data on different platforms like, mobile phones, tablets, smartwatches, etc has arisen and it becomes difficult to manage the source code by adding more and more lines of programming into the same code base, to cater to the different types of clients. There are several constraints with monolithic architecture some of which are,

Serving the application on different gadgets at runtime with the same build
Scaling the application
Continuous testing of the application
Deployment issues - as all components are to be deployed as a single unit
Delay in delivery

To address all these and many more challenges, micro-services came into the picture. It breaks down the complete application into smaller parts making it easier to develop and release applications.

Top 7 reasons to adopt microservices

1. Easy Development: Since microservices are smaller units, they provide ease of development for technical teams. There’s no need to assign large teams to work on a single project, risking delays and errors.

2. Scalability: Smaller components are always easier to scale instead of one single monolith. Microservices enable organizations to scale up their applications at a far lower cost, with easy integration with third-party services.

3. Infrastructure: Microservices are Cloud-based and thus provide agility for any organization that adopts them. Components and Services can be spread across many servers and data centers, offering innovation at a lower cost.

4. Continuity: Software can be broken down into component services and each of these services can be deployed without compromising application probity.

5. Decentralized Data: In microservices, all the components have their own data stores which help to maintain the agility of applications with minimal to zero impact on new releases.

6. Technological Flexibility: Adopting microservices enables the organizations to cope up with technological changes, making them independent to choose any tech stack for an individual service.

7. Containerization: Containerization is one of the most efficient ways of scaling applications. Containers provide a runtime environment for applications from development to testing and finally, deploying the microservices wrapped under it. Furthermore, for orchestrating containers in production, Kubernetes is one of the best and most adopted tools in the market.

Organizations like Netflix, Microsoft, Google, Uber, Amazon, PayPal, Twitter, Delhivery, Bharatpe, etc have adopted the microservice architecture and witnessed great results. Netflix has been able to service 93.8 million users globally and stream more than 10 billion hours of movies and shows. Amazon has transitioned to continuous deployment and its engineers have been able to deploy code every 11.7 sec.

Now that you know the benefits of the microservices architecture, why don’t you get started and deploy your applications on Kubernetes in the easiest possible way? Devtron on its first open-source anniversary, has announced to help the first 100 organizations to accelerate 🚀 their journey of adopting Kubernetes FREE OF COST. Check out this Open Source project and do sign-up on #AdoptK8sWithDevtron.

Decreasing Carbon footprints using Kubernetes

KCD Chennai — Sat, 21 May 2022 10:19:45 +0000

Author: Prakarsh & Abhinav Dubey, Devtron Labs (Platinum sponsor, KCD Chennai)

If you are running microservices, there is a high probability that you're running it on a Cloud platform like AWS, GCP, Azure, etc. The services of these cloud providers are powered by data centers, which typically comprise thousands of interconnected servers and consume a substantial amount of electrical energy. It is estimated that data centers will use anywhere between 3% and 13% of global electricity by 2030 and will be responsible for a similar share of carbon emissions. This post will step through an example with a case study of how to use Kubernetes to minimize the carbon footprint of your organization's infrastructure.

Carbon Footprint

What exactly is carbon footprint? It is the total amount of Greenhouse gases (Carbon dioxide, Methane, etc) released into the atmosphere due to human interventions. Sounds familiar, isn’t it? We all have read about the different ways greenhouse gases get released into the atmosphere and the need to prevent them. But why are we talking about it in a technical blog? And how can Kubernetes help to reduce your organization's carbon footprint?

In the era of technology, everything is hosted on cloud servers, which are backed by massive data centers. In other words, we can say data centers are the brain of the internet. Right from the servers to storage blocks, everything is present in these data centers.

All the machines require energy to operate, i.e, electricity, irrespective of the source of generation, renewable or nonrenewable. According to a survey conducted by Aspen Global Change Institute, data centers account for being one of the direct contributors to climate change due to the release of greenhouse gasses. According to an article by Energy Innovation, a data center that contains thousands of IT devices can use around 100 megawatts (MW) of electricity.

How can Kubernetes help reduce the carbon footprint? Before answering the above question, let’s learn how to calculate the carbon footprint of a server that is created with a public cloud provider like AWS. In AWS, we create instances of different sizes, computation power, storage capacity, etc as per our needs. Let’s calculate the carbon footprint emitted by initializing an instance of type - m5.2xlarge in the region - ap-south-1 which runs for 24 hours. We can calculate using the Carbon Footprint Estimator for AWS instances. As you can see in the below image, after giving the values, we can easily estimate the carbon footprint for the respective instance which is 1,245.7 gCO₂eq.

Factors considered during the calculation

There are primarily two factors that contribute to carbon emissions in the Compute Resource carbon footprint analysis, which are discussed below:

Carbon emissions related to running the instance, including the data center PUE

Carbon emissions from electricity consumed are the major carbon footprint source in the tech industry. The AWS EC2 Carbon Footprint Dataset available, is used for the calculation of carbon emissions based on how much wattage is used on various CPU consumption levels.

Carbon emissions related to manufacturing the underlying hardware

Carbon emissions from the manufacturing of hardware components are another major contributor when it comes to carbon footprint calculations within the scope of this study.

How K8s helps

Kubernetes is one of the most adopted technologies for running containerized workloads. As per the survey by Portworx, 68% of companies have increased the usage of Kubernetes and IT automation, and the adoption metrics are increasing, day by day. For any application to be deployed over Kubernetes, we need to create a Kubernetes cluster that comprises any number of master and worker nodes. These are nothing but instances/servers which are being initialized, where all the applications will be deployed. The number of nodes increases as the load increases, which eventually will contribute more to the carbon footprint. But with the help of Kubernetes Autoscaling, we can lower the count of nodes i.e, reduce the number of instances created as per the requirements.

Let’s try to understand it with a use case.

A logistic company uses Kubernetes in its production to deploy all its microservices. A single microservice requiring around 60 replicas, can initialize around 20 instances of m5.2xlarge type in ap-south-1 region. If we calculate the carbon footprint of a single microservice in 24 hours, it would be around 24,914 gCO₂eq. This is the amount of carbon footprint emitted just by a single microservice, and there are thousands of microservices in an organization that runs 24*7 workloads. Now, if you are wondering how this can be reduced, you are at the right place.

The maximum traffic that a logistic company's first / last mile app experiences are during the daytime when the deliveries happen. Generally, from morning around 8:00 AM, there’s a steep increase in the traffic and it experiences its peak in the afternoon hours. That's when it would have the most number of pod replicas and nodes/instances. Then post 8:00 PM the traffic decreases gradually. During the off-hours, when the traffic decreases, the count of replicas also drops to 6 from an average of 60 for each microservice, which requires only 2 nodes/instances. The microservice uses throughput metrics based on horizontal pod autoscaling using Keda.

Now, if we talk about the carbon emissions by 1 micro service after deploying it on Kubernetes paired with horizontal pod auto-scaling, it would drop to around

51.9 gCO₂eq x 2 (nodes) x 12 (hrs 8pm to 8am non-peak hours) + 51.9 gCO₂eq x 20 (nodes) x 12 (hrs 8am to 8pm peak traffic hours) = 13,701.6 gCO₂eq

51.9 gCO₂eq x 20 (nodes) x 24 (hrs no autoscaling) = 24,912 gCO₂eq

That's a whopping 11,211 gCO₂eq (45% reduction) in carbon emissions [1], and this is just by 1 micro-service in 24hrs. This translates to 4092 KgCo2eq per year! To understand how much this is, a Boeing 747-400 releases about 90 KgCo2eq per hour of flight when it covers around 1000 Kms [2].

Now think of how much your organization is capable of reducing per year by migrating 1000s of microservices on Kubernetes paired with efficient autoscaling.

Migrating to Kubernetes?

Kubernetes is the best container orchestration technology out there in the cloud-native market, but the learning curve for adopting Kubernetes is still a challenge for the organizations looking to migrate to that platform. There are many Kubernetes-specific tools in the marketplace like ArgoCD, Flux for deployments, Prometheus and Grafana for monitoring, Argo workflows for creating parallel jobs, etc. Kubernetes space has so many options, but they need to be configured separately, which again is complex.

That's where the open source tool - Devtron, can help you. Devtron configures and integrates all these tools that you would have to otherwise configure separately and lets you manage everything from one slick Dashboard. Devtron is a software delivery workflow for Kubernetes-based applications. It is trusted by thousands of users and used by organizations like Delhivery, Bharatpe, Livspace, Moglix, etc, with good user communities across the globe. And the best thing is, it gives you a no-code Kubernetes deployment experience which means there's no need to write Kubernetes YAML at all.

Devtron is running an initiative #AdoptK8sWithDevtron where it offers expert assistance and help to the first 100 organizations that want to migrate their microservices to Kubernetes.

References
[1] When compared to micro-services which are not configured to scale efficiently.
[2] https://catsr.vse.gmu.edu/SYST460/LectureNotes_AviationEmissions.pdf

Packet flow in OpenShift SDN

KCD Chennai — Fri, 20 May 2022 11:20:41 +0000

Author: Red Hat Inc. (Platinum Sponsor, KCD Chennai 2022)

OpenShift Container Platform uses Software Defined Networking (SDN) as the default Cluster Network Interface (CNI) provider. OpenShift SDN sets up cluster networking which facilitates the communication between the pods. It does so by configuring an overlay network using Open VSwitch (OVS).

In every node, an OVS bridge (br0) is created. Whenever any packet reaches the OVS bridge, the packet is checked against the flow rules in the flow-tables. After processing the packet through the flow-tables, all the actions corresponding to the matching flow rules are applied to the packet.

An internal interface (tun0) is configured on each node and is connected to the OVS bridge via a port. A route is added in the node’s routing table for sending all traffic destined for the pod network CIDR to the tun0 interface. The tun0 will send it to the OVS bridge for processing the packet through the flow tables.

OpenShift SDN uses Virtual Extensible LAN (VXLAN) tunnels to set up an overlay network between all the nodes in a cluster. The VXLAN tunnels are used to send traffic from one node to another, thus also enabling communication between pods belonging to different nodes. In every node, a VXLAN interface (vxlan0) is configured and is also connected to the OVS bridge via a port. Whenever a packet is needed to be forwarded from one node to another one, the bridge (br0) on the first node sends the packet to the corresponding vxlan0 interface. The vxlan0 interface then encapsulates the packet and sends it to the other node. When the packet is received by the vxlan0 interface on the other node, it first removes the encapsulation from the packet and then sends the packet to the bridge (br0) for further processing.

Every pod that is provisioned on a node is configured with a virtual ethernet pair, one interface in the pod’s network namespace and the other interface in the node’s network stack. The ethernet interface that is configured in the node’s network stack is connected to the OVS bridge via a port.

There can be 3 different types of communications a pod can have: 1) Pod to pod communication, 2) Pod to service communication, and 3) Pod to external host communication. The packet flow from source to destination for these different scenarios are described in the following sections.

Pod to pod communication

In pod to pod communication, a pod tries to communicate with another pod directly. For example, pod A wants to communicate with pod B. In this case, directly communicating would mean that either pod A uses pod B’s DNS name or pod B’s IP address. The two pods may be in the same node or in different nodes. In the following sections we will see how both scenarios are handled in OpenShift SDN.

In the same node

Fig. 1 shows how pod to pod communication takes place when both the pods belong to the same node.

Fig. 1: Pod to pod communication in the same node.

In different nodes

Fig. 2 shows how pod to pod communication takes place when the communicating pods belong to the different nodes.

Fig. 2: Pod to pod communication in the different nodes.

The pod A sends a packet to its own ethernet interface (eth0) with pod B’s address as the destination. From eth0, the packet is sent to the corresponding virtual ethernet (veth0) interface in the node’s network stack. As veth0 is connected to the OVS bridge (br0) the packet is forwarded to it. The actions of the matching flows in the flow-tables are executed and the bridge forwards the traffic to the Virtual Extensible LAN interface (vxlan0) of the node. The reason behind this is that the packet is destined for a pod in a different node.
The vxlan0 encapsulates the packet and sends the traffic from node 1 to node 2 through the VXLAN tunnel. Once the packet is received by the vxlan0 interface in node 2, the encapsulation is removed. Then, the vxlan0 interface forwards the packet to the bridge (br0) of node 2. The packet is checked for matches in the flow-tables. As the packet is destined for the local pod B the corresponding flows match, and the packet is forwarded to the virtual ethernet interface (veth0) in the node’s network stack corresponding to pod B’s ethernet interface (eth0).

Pod to Pod communication via service

In this scenario, a pod tries to communicate with another set of pods by using the DNS name or IP address of a service. For example, pod A wants to communicate with a set of pods which are represented by service S. Lets say, packets sent by pod A to service S get redirected to pod B. Pods A and B may be in the same node or in different nodes. In the following sections we will see how both scenarios are handled in OpenShift SDN.

In the same node

Fig. 3 shows how pod to pod communication via service takes place when both the pods belong to the same node.

Fig. 3: Pod to service communication in the same node.

The pod A sends a packet to its own ethernet interface (eth0) with service S’s address as the destination. As eth0’s corresponding virtual ethernet (veth0) in the nodes’s network stack is connected to the OVS bridge (br0), the packet is forwarded to it. The packet is checked for matches in the flow-tables and the corresponding actions are executed. As the packet is destined for the service S, it is forwarded to the internal interface (tun0). Before forwarding the packet, the iptables rules are processed and the backend pod (pod B) is chosen for service S. The destination of the packet is now modified to pod B from service S. Thus, the packet is sent to tun0 as it is destined for a pod which, in turn, forwards the packet to the bridge (br0). The packet is matched with the flow rules in the flow-tables and it is forwarded to the virtual ethernet interface (veth1) in the node’s network stack corresponding to pod B’s ethernet interface (eth0).

In different nodes

Fig. 4 shows how pod to pod communication via service takes place when the communicating pods belong to the different nodes.

Fig. 4: Pod to service communication in different nodes.

The pod A sends a packet to its own ethernet interface (eth0) with service S’s address as the destination. As eth0’s corresponding virtual ethernet (veth0) in the nodes’s network stack is connected to the OVS bridge (br0), the packet is forwarded to it. The packet is checked for matches in the flow-tables and the corresponding actions are executed. As the packet is destined for the service S, it is forwarded to the internal interface (tun0). Before forwarding the packet, the iptables rules are processed and the backend pod (pod B) is chosen for service S. The destination of the packet is now modified to pod B from service S. The packet is sent to tun0 as it is destined for a pod which in turn forwards the packet to the br0 interface. As the packet is destined for a pod B in a different node the corresponding flow rules match, the bridge forwards the traffic to the virtual extensible LAN interface (vxlan0).

The vxlan0 encapsulates the packet and sends the traffic from node 1 to node 2 through the VXLAN tunnel. Once the packet is received by the vxlan0 interface in node 2, the encapsulation is removed. Then, the vxlan0 interface forwards the packet to the bridge (br0) of node 2. The packet is checked for matches in the flow-tables. As the packet is destined for the local pod B the corresponding flow rules match, and the packet is forwarded to the virtual ethernet interface (veth0) in the node’s network stack corresponding to pod B’s ethernet interface (eth0).

Pod to external host communication

In this case, a pod tries to communicate with an external host. Traffic can be sent from a pod to an external host in different ways: 1) through special pods called Egress Router pods, 2) through the same node using the node IP, and 3) through Egress IP (either on same node or different node). In the following sections we will see how all the scenarios are handled in OpenShift SDN.

Through Egress Router

Fig. 5 shows how pod to external host communication takes place via an Egress Router pod.

Fig. 5: Pod to external host communication through Egress Router.

The Egress Router pod is configured with an additional MACVLAN interface (macvlan0) in the node’s network stack. The macvlan0 is assigned an IP address and any traffic sent out through the interface is NAT-ed using the assigned IP address.

The pod A sends a packet with the Egress Router pod’s address as the destination. In this example the Egress Router pod is provisioned on a different node. However, the Egress Router pod can also be on the same node as well. The packet flow, in both the cases, from pod A to Egress Router pod would happen in the same way as explained above in the pod to pod communication section. Once the packet is received by the Egress Router pod, it will send the packet to the External Host (based on how the Egress Router pod is configured). The source address of the packet would be the IP address of the macvlan0 interface.

Through the same node

Fig. 6 shows how pod to external host communication takes place through the same node using the node IP.

Fig. 6: Pod to external host communication through the same node.

Whenever a pod tries to send a packet to an external host, the default method is sending the packet from the same node. When pod A sends a packet destined for the external host, the packet is matched against the flow rules after it reaches the OVS bridge (br0). As the destination is an external host, the packet is sent to the internal interface (tun0) in the node’s network stack. Then, the packet is sent through the eth0 interface after NAT-ing the packet using the node IP assigned on eth0 interface.

Through Egress IPs

Fig. 7 shows how pod to external host communication takes place through an Egress IP.

Fig. 7: Pod to external host communication through Egress IP.

As mentioned above, the default method of sending a packet, destined for an external host, is through the same node using the Node IP. However, a namespace can be configured with Egress IP(s) for sending packets to external hosts. The Egress IPs will be provisioned on the ethernet interfaces of the nodes in addition to the Node IP. When a pod which belongs to a namespace configured with Egress IP(s) sends a packet destined for an external host, then the packet would first be sent to the node on which the chosen Egress IP is provisioned. If the chosen Egress IP is provisioned on the same node then it would be similar to sending the packet using the Node IP, except the NAT-ed packet would have the chosen Egress IP as the source address. If the chosen Egress IP is provisioned on another node, then for sending the packet from one node to another, the VXLAN tunnel is used. Once the packet reaches the OVS bridge (br0) on the node with the chosen Egress IP, the packet is matched against the flow rules. The packet is then sent to the tun0 interface which, in turn, sends the packet to the eth0 interface to be sent to the external host. Before the packet is forwarded to the external host, it is NAT-ed using the chosen Egress IP.

What’s next

More details can be found here: https://github.com/openshift/sdn
Official Red Hat OpenShift documentation for OpenShift SDN:
https://docs.openshift.com/container-platform/4.10/networking/openshift_sdn/about-open shift-sdn.html

Difference between Helm2 and Helm3

KCD Chennai — Fri, 20 May 2022 11:01:25 +0000

Author : KodeKloud (Platinum Sponsor, KCD Chennai 2022)

In this blog, we will see what is the difference between Helm2 and Helm3.

As administrators had to manage more and more Kubernetes objects in their clusters to run their apps and infrastructure, a natural need for something to manage all of this complexity arose. So, the project called Helm appeared under the CNCF’s (Cloud Native Computing Foundation) umbrella. Since the initial launch in 2016, the project has matured and it got better and better. The improvements were also made possible by the fact that Kubernetes itself was improving, so Helm had more tools at its disposal it could leverage right off of Kubernetes.

No More Tiller

When Helm 2 was around, Kubernetes lacked features such as Role-Based Access Control and Custom Resource Definitions. To allow Helm to do its magic, an extra component, called Tiller, had to be installed in the Kubernetes cluster. So, whenever you wanted to install a Helm chart, you used the Helm (client) program installed on your local computer. This communicated with Tiller that was running on some server. Tiller, in turn, communicated with Kubernetes and proceeded to take action to make whatever you requested happen. So, Tiller was the middleman, so to speak.

Besides the fact that an extra component sitting between you and Kubernetes adds complexity, there were also some security concerns. By default, Tiller was running in “God mode” or otherwise said, it had the privileges to do anything it wanted. This was good since it allowed it to make whatever changes necessary in your Kubernetes cluster to install your charts. But this was bad since it allowed any user with Tiller access to do whatever they wanted in the cluster.

After cool stuff like Role-Based Access Control (RBAC) and Custom Resource Definitions appeared in Kubernetes, the need for Tiller decreased, so it was removed entirely in Helm 3. Now there’s nothing sitting between Helm and the cluster.
That is, when you use the Helm program on your local computer, this connects directly to the cluster (Kubernetes API server) and starts to work its magic.

Furthermore, with RBAC, security is much improved and any user can be limited in what they can do with Helm. Before you had to set these limits in Tiller and that was not the best option. But with RBAC built from the ground up to fine-tune user permissions in Kubernetes, it’s now straightforward to do. As far as Kubernetes is concerned, it doesn’t matter if the user is trying to make changes within the cluster with kubectl or with helm commands. The user requesting the changes has the same RBAC-allowed permissions whatever tool they use.

Three-way strategic merge patch

Now, this is probably a more important change than what we discussed before with Tiller.

The name might sound intimidating, but don’t worry, at the end of this section we’ll see it’s actually simple, but a very smart thing that can prove quite useful.

Helm has something like a snapshot feature. Here’s an example:

You can use a chart to install a full-blown WordPress website. This will create revision number 1 for this install. Then, if you change something, for example, you upgrade to a newer chart to upgrade your WordPress install, you will arrive at revision number 2. These revisions can be considered something like snapshots, the exact state of a Kubernetes package at that moment in time. If there’s a need, you can return to revision number 1, do a rollback. This would get your package/app to the same state it was at when you first installed your chart.

New revisions are created whenever important changes are done with the Helm command. For example, when we install a package, a revision is created. When we upgrade that package, a new revision appears. Even when we roll back a new revision is created.

This is not like a typical backup/restore feature, in the sense that you do not get old data back. If you deleted a database in a persistent volume, this does not restore the persistent volume with the old data. What this does is bring all Kubernetes objects back to their old state, their old declarations, as they were at the time the revision was created.

Another way to think about this is that a rollback restores pretty much everything back to the way it was, EXCEPT for persistent data. Persistent data should be backed up with regular methods.

Helm 2 was less sophisticated when it came to how it did such rollbacks. To understand what was lacking, the official documentation page gives us this example:

You install a chart. This creates revision number 1 and it contains a deployment with 3 replicas. But someone, for some reason, brings down the number of replicas to 0, with a command like:

kubectl scale --replicas=0 deployment/myapp

Now, this does not create a new revision, as it wasn’t a helm upgrade, install, or rollback command. It was just a “manual” change done without Helm.

But we still have revision 1 available, with the original state. So no problem we would think, we just roll back to the original:
helm rollback myapp

But Helm 2 compares the chart currently used with the chart we want to revert to. Since this is the original install, revision 1, and we want to roll back to revision 1, the current chart and the old chart are identical. We did not change any chart, we just manually edited a small Kubernetes object. Helm 2 considers nothing should be changed so nothing happens. The count of replicas remains at 0.

Helm 3 on the other hand is more intelligent. It compares the chart we want to revert to, the chart currently in use, and also, the live state (how our Kubernetes objects currently look like, their declarations in .yaml form). This is where that fancy “Three-way strategic merge patch” name comes from. By also looking at the live state, it notices that the live state replica count is at 0, but the replica count in revision 1 we want to revert to is at 3, so it makes necessary changes to come back to the original state.
Besides rollbacks, there are also things like upgrades to consider, where Helm 2 was also lacking. For example, say you install a chart. But then you make some changes to some of the Kubernetes objects installed. It all works nicely until you perform an upgrade. Helm 2 looks at the old chart and the new chart you want to upgrade to. All your changes will be lost since they don’t exist in the old chart or the new chart. But Helm 3, as mentioned, looks at the charts and also at the live state. It notices you added some stuff of your own so it performs the upgrade while preserving anything you might have added.

You can visit the link for a more in-depth explanation.

What was mentioned above are probably the biggest changes in Helm 3. There are some other smaller changes, but they don’t really affect how you’ll work with the newer version of Helm, especially if you’re a new user.

Checkout the Helm for the Absolute Beginners course here
Checkout the Complete Kubernetes learning path here

Your Roadmap to Become a DevOps Engineer in 2022

KCD Chennai — Fri, 20 May 2022 09:08:16 +0000

Author : KodeKloud (Platinum Sponsor, KCD Chennai 2022)

DevOps Engineer has become an emotion and not just a job profile. People from all sorts of professions are willing to get their hands dirty and make a shift to this career as a DevOps engineer. Apart from the buzzword bingo, DevOps holds a unique position in the software field, even though many technologies evolved over time and vanished, this path doesn't seem like it will disappear very soon. DevOps is here to stay, and we will witness a large number of companies embracing this approach sooner or later.

What exactly is DevOps?

DevOps is a process that emphasizes communication and learning from a technical standpoint between software developers and IT professionals, like dev and ops, managing production environments, while automating the deployment of software and infrastructure changes with utmost care and collaboration. At the core, DevOps means combining development and operations into one unified team so that continuous process of learning, knowledge sharing, and shared responsibilities happen seamlessly between the two.
The idea of DevOps grew out of the Agile methodology and first gained attention in 2009.

The need for DevOps

Many departments in the companies are usually siloed and carry their own procedures. Especially when it comes to a software powered organization, Devs will have no idea what is happening with Ops and vice versa, and it creates a lot of confusion among the teams and impacts the overall company growth and individual productivity. The idea of DevOps is to bridge the gap between development and operations to support other departments, so the workflow within the organization is smooth. This allows companies to fail early and learn early and thereby quickly deliver software features and security updates. The ultimate goal of DevOps is to bring products faster to the market with more quality and reliability.

Skills required to become a DevOps engineer:

DevOps is a cultural phenomenon rather than an individual job role. It is more of a team sport and can't be done alone. So, contradicting my own statement, there is no concept of DevOps engineer, it is just that firms have created this role for their understanding.

So, is it about the tools?

Nope:)
But having the understanding of all the DevOps tools like Docker, GitHub, Kubernetes, Terraform, Ansible, Puppet, etc is necessary because that is what companies are looking for when hiring a DevOps engineer.

But most of all, it is all about learning the DevOps culture and framework rather than tools. Most people might also stress the automation aspect, it is not all about automation. DevOps engineers should have a basic knowledge of scripting, programming, and framework. People coming from other departments to DevOps should understand what developers are trying to do in the development phase and then how they are managing the versions of their code, how they are testing, integrating them, and deploying them to servers and finally, how end users are getting the software to use. Once they understand how things are done theoretically and manually with no automation tools, the concepts will get clear and easier.

In Addition to this, Ops knowledge is also necessary to become good in DevOps. A DevOps engineer, not only writes code or automates but he also has to know other related aspects too, for example

Scripting and Linux basics
Knowledge of different cloud providers
Knowledge of how software development life cycle works (SDLC)
Familiarity with source control and versioning and tools like GitHub & Bitbucket
Experience with building tools
Artifacts management tools like JFrog artifactory & Sonatype
Infrastructure design and microservices
Better communication skills
Automation testing skills
Understanding about infrastructure as code
Troubleshooting skills
Understanding the concepts of CI/CD and tools
Knowledge of DevOps pipeline and how it works
Knowing how systems scale - Horizontal scaling and vertical scaling
Virtualization concepts
Understanding of different DevOps success metrics like deployment frequency, lead time to change, change failure rate, time to restore services back, etc
Containerization concepts and tools like Docker
Container orchestration and tools like Kubernetes
Software release cycle and management

What is CI, CD?

Continuous integration (CI), as the name itself suggests, focuses on combining the work of individual developers together into a repository or a codebase to streamline the continuous efforts. This can be done several times a day; the primary objective is to enable early detection of integration bugs and also to allow for tighter cohesion and more smoother development collaboration. The goal of CI is to quickly make sure a new code change from a developer is good and suitable for further use in the codebase.

The aim of continuous delivery (CD) is to minimize the friction points that are inherent in the deployment phases. Typically, a team's implementation involves automating each of the steps to build deployments so that a safe code release can be done at any moment in time.

Continuous delivery is the repetitive practice of building, testing, and making delivering improvements to software codebase with the help of automated tools. The key result of the continuous delivery (CD) is the code that is always in a deployable state.

Many people confuse between the two, continuous delivery and continuous deployment, whereas both are different in the DevOps space.

Continuous deployment is a DevOps process in which a much higher degree of automation is involved, where a build/deployment occurs automatically whenever a major change is made to the code. Here, developer code changes are automatically detected and prepared for a release to production instantly.

Where to start your DevOps journey?

The problem is, very rarely companies hire freshers to work as DevOps engineers but that being said, there is a huge skills gap in the industry. Firms struggle to hire a good DevOps candidate and they often fail because of the scarcity of talent.
Listing down below some resources and courses you can opt-in to move into DevOps career path,

Read the book ‘The Pheonix Project’
Take this course DevOps pre-requisite course
Watch this video by Rackspace that explains the meaning of DevOps in simple English
Read these interesting answers on Quora by experts on becoming a good DevOps engineer
Watch and learn side by side with this Docker for beginners full free course by Mumshad (One of the top Udemy instructors)
Follow these 100 DevOps influencers on Twitter
Join Developer community forums like dev.to, Hashnode, Dzone, DevOps subreddit, Stackoverflow, DevOps StackExchange, Changelog, etc

DevOps is taking the center stage and as we have mentioned before, it is becoming the epitome of software development. DevOps engineers are one of the highest-paid professionals in the world and this is the demanding tech job currently around the world. DevOps is a good career path and a proper plan and approach will get you a good job but once you get into it, it is highly recommended to always keep learning since the DevOps space is always evolving and new tools are emerging day by day.

BTW, sometimes it can be difficult to get hired as a DevOps engineer without any prior work experience or knowledge of different tools and automation techniques, we at KodeKloud have come up with the KodeKloud Engineer to help you gain free DevOps work experience by solving real DevOps problems and challenges, with which you can get hired for DevOps role. Sign up for Free here.

Migrating Stateful Applications between Kubernetes Clusters using Crane

KCD Chennai — Tue, 17 May 2022 15:43:05 +0000

Author : John Matthews, Savitha Raghunathan , Red Hat (Platinum Sponsor, KCD Chennai 2022)

This post will introduce Crane, a tool under the Konveyor community that helps application owners migrate Kubernetes workloads, and their state between clusters.

Migrating an application between Kubernetes clusters may be more nuanced than one would imagine. In an ideal situation, this would be as simple as applying the YAML manifests to the new cluster and adjusting DNS records to redirect external traffic, yet often there is more that is needed. Below are a few of the common concerns that need to be addressed:

YAML manifests - do we have the original YAML manifests stored in version control or accessible so we can reapply to the new cluster?
Configuration drift - if we do have the YAML manifests, do we have confidence they are still accurate and represent the application as it’s running in the cluster? Perhaps the application has been running for a while, been modified, and we no longer have confidence we can reproduce it exactly as it’s currently running.
State - we may need to address persisted state that has been generated inside of the cluster, either small elements of state such as generated certificates stored in a Secret, data stored in Custom Resources, or gigabytes of data in persistent volumes.
Customizations needed for a new environment - we may be migrating across cloud vendors or environments that require transformations to the applications so they can run in the new environment.

Crane helps users do more than just handle a point in time migration of a workload, it is intended to help users adopt current best practices such as onboarding to GitOps by reconstructing redeployable YAML manifests from inspecting a running application. The project is the result of several years of experience performing large-scale production Kubernetes migrations and addressing the lessons learned.

Crane follows the Unix philosophy of building small sharply focused tools that can be assembled in powerful ways. It is designed with transparency and ease-of-diagnostics in mind. It drives migration through a pipeline of non-destructive tasks that output results to disk so the operation can be easily audited and versioned without impacting live workloads. The tasks can be run repeatedly and will output consistent results given the same inputs without side effects on the system at large.

The process uses a few tools:

crane: The command-line tool that migrates applications to the terminal.
crane-lib: The brains behind Crane functionality responsible for transforming resources.
crane-plugins: Collection of plugins from the Konveyor community based on experience from performing Kube migrations.
crane-plugin-openshift: An optional plugin specifically tailored to manage OpenShift migration workloads and an example of a repeatable best practice.
pvc-transfer: The library that powers the Persistent Volume migration ability, shared with the VolSync project. State migration of Persistent Volumes is handled by rsync allowing storage migrations between different storage classes.
crane-runner: A collection of resources showing how to leverage Tekton to build migration workflows with Crane

A Walkthrough Migration using Crane

Crane works by exporting the application-specific resources from one cluster, transforming the manifests to create re-deployable manifests, and applying the transformed manifests into the destination cluster. We will be using crane-runner to migrate a stateful application.

Requirements

Linux VM with a minimum of 2vCPUs and 8GB of RAM. If using an Amazon EC2 instance, the recommended VM size is t2.large.
Ensure the following tools are installed and available in the PATH: Podman, kubectl, minikube, kustomize, jq, yq

Stateful Guestbook App Migration

We will migrate a simple PHP Guestbook application with a Redis database that will use the underlying persistent storage. The diagram below shows the high-level operations that will be performed by Crane as a part of this migration,

Figure: Stateful App Migration using Crane

Follow this step-by-step guide to complete the migration. This example demonstrates how simple, yet powerful Crane can be and how easily it can be configured to perform complex operations with ease.

What’s Next?

Check out more examples using crane-runner.
Participate in Konveyor community meetings to learn more about the projects in the ecosystem.

About Konveyor:

Konveyor community is home for several open source projects that help with your Application Modernization & Migration journey. The community is actively working on expanding the tools to cater to various real world migration scenarios. If you would like to get involved with the community, reach out to us via Kubernetes slack channel. You can also subscribe to our mailing list from the Konveyor website to hear about what's happening in the community, the latest meetups, and read about new migration stories.

Acknowledgements:

Huge shout out to David Zager & Erik Nelson for contributing to this blog post with their wonderful crane-runner examples.

Authors:

John Matthews:
John Matthews has a 20+ year background in software engineering specializing in Linux systems. He's been focused primarily on Kubernetes since Kubernetes 1.6. At Red Hat, John serves in a hybrid role of technical architect and people manager for solutions related to Kubernetes Migration and Data Protection. He lives in Raleigh, NC with his wife, daughter, and a growing pack of Dobermans.
LinkedIn: https://www.linkedin.com/in/johnwmatthews/

Savitha Raghunathan:
Savitha is a Senior Software Engineer at Red Hat, working on Kubernetes/OpenShift Data Protection software solutions. She is an upstream Kubernetes contributor and passionate about mentoring individuals new to open source communities. She likes to read, travel, and crochet in her free time.
LinkedIn: https://www.linkedin.com/in/savitharaghunathan/