DEV Community: cnu1812

What is DataOps?How can kubernetes help here!

cnu1812 — Sat, 29 Jan 2022 06:19:37 +0000

Introduction to DataOps

What is DataOps?

DataOps is the orchestration of people, processes, and technology to deliver trusted, business-ready data to data citizens, operations, and applications throughout the data lifecycle. With properly governed data, businesses can comply with complex regulations, data privacy and ensure artificial intelligence (AI) model accuracy by monitoring data quality.

This is the definition available on the web. It’s very complex for beginners, isn't it? So to put it in a brief:

DataOps brings together data scientists, analysts, developers, and operations to work on the entire product/service lifecycle, from the design stage to production support.
Sounds like the DevOps definition...But! the goal of DataOps is to make data analytics more efficient. To do so, DataOps adopts Agile Development principles, thereby improving the efficiency and effectiveness of the data teams and users.
DataOps combines principles from DevOps, Agile Development, and Lean manufacturing. Think of it as a water pipeline; your goal is to keep the water flowing in spite of all the plumbing work you carry out.

And here comes two new concepts/topics Lean Manufacturing and Agile Development. Let's see one by one:

What is Lean Manufacturing?

Lean manufacturing is a production process based on an ideology of maximising productivity while simultaneously minimising waste within a manufacturing operation. The lean principle sees waste is anything that doesn’t add value that the customers are willing to pay for.
Manufacturing happens in pipelines—raw materials flow through various manufacturing workstations to be transformed into finished goods.

What is Agile?

Let us say Mr. X is a developer. So, one day Bart(Client) comes to Mr. X and said:

Bart: Hi Mr. X! I want an application for the DOK community.
Mr. X: Definitely Bart! What are the features that you would like to see in the application?
Bart: Yeah! The primary feature is it has to collect all the comments from all socials of dok community.
Mr.X: Fine! I will start working on your application.

After a year… He delivers the project in one go.

Mr.X: Hi Bart! Your application is ready.
Bart: Yeah well, Why don’t you add another feature like collect all comments and filter the most common comments.

This would be frustrating. Because developing software may take months or years.

But what if happened like this. While Mr.X is doing his application Bart may call or email him that I want this so and so feature.

So this is where agile comes into play:

Agile is a time-boxed, iterative approach to software delivery that builds software incrementally from the start of the project, instead of trying to deliver it all at once near the end.
Scrum is a framework of rules, roles, events, and artifacts used to implement Agile projects. It is an iterative approach, consisting of sprints that typically only last one to four weeks. This approach ensures that your team delivers a version of the product regularly.
A sprint in Scrum is a short period of time wherein a development team works to complete specific tasks, milestones, or deliverables. Sprints, also referred to as “iterations,” essentially break the project schedule into digestible blocks of time in which smaller goals can be accomplished.
It works by breaking projects down into little bits of user functionality called user stories, prioritizing them, and then continuously delivering them in short two-week cycles called iterations.
This means that data teams can publish new analytics in shorter increments, referred to as sprints, which significantly reduces wait times. Hence DataOps adopt an agile environment.

What are the challenges dataops trying to resolve?

The problem is that many organizations aren’t exactly clear on the most efficient approach to data collection and analytics. They often take a seemingly all-encompassing approach based on the principle of “we’ll collect the data and then figure out what to do with it,” that can do more harm than good.
They then have a data team who is supposed to miraculously turn garbage into gold, which generally requires far more effort than necessary and rarely leads to the desired results. Of course, this makes it virtually impossible to deliver actionable insights on a schedule that can keep up with the demands of a DevOps team that’s pushing to get their code to market.
DataOps takes this jumbled mess and turns it into a smooth process where data teams aren’t spending their time trying to fix problems. They aren’t wasting their time trying to turn poor raw data into something usable. Instead, they can focus on what matters, namely providing actionable insights.
DataOps ensures that the raw data coming in is useable, it ensures that the results are accurate, it focuses on the value of people and working together, and it keeps the data team at the center of the company’s strategic objectives. After all, they no longer take months to come up with the required insights.

Now here comes the big question of how Kubernetes help DataOps. Before that let us know what is kubernetes in a nutshell.

Let’s say you created an application(e.g Pokemon go) and decided to share it with the world. How can you do that? by deploying it to the internet. So you deployed this application into 5 different servers using docker. And your application starts getting massive traffic even you are not imagined. Millions of users started using. You are happy on one side simultaneously you are sad because you have to scale up the servers because 5 are not enough for such massive traffic. Now you need to scale up fast and make sure all containers restart if they die. And you will be gone out of control.
This is where Kubernetes comes into play.

Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management.

A Kubernetes cluster consists of a set of worker machines, called nodes, that run containerized applications.
Development teams and data science teams are increasingly interested in moving their build-test and model training environments to Kubernetes in order to drive efficiency and avoid vendor lock-in. When they make this migration, it is absolutely necessary for them to be able to take their clone-based processes with them so that they don’t lose all of the lifecycle efficiencies that they have achieved over the years. In fact, the lack of ability to clone a volume to a different namespace is often a showstopper. It is considered a best practice for different developers to work in different namespaces. This is done for security reasons, and there is typically no compromising when it comes to security. If clones are going to be included in the development lifecycle, the ability to clone workspaces to different namespaces is vital.

Every cluster has at least one worker node. Hence if a node fails, your application will still be accessible from the other nodes as in a cluster, multiple nodes are grouped.

Why should you use Kubernetes for DataOps?

DataOps has many things in common that relate them to Kubernetes. It serve the same business goals as devops when it comes to speeding up application software development, deployment, and delivery. In a nutshell, the DataOps practices focus on improving application production through software development workflow optimization powered by data-driven insights.

Kubernetes follows the same principles being a powerful tool that provides a rock-solid technological ground for enhancements on both software and human levels. DataOps engineers can leverage the accuracy of massive data assets for their BigData analytics through robust data orchestration, timely updates, persistent data storage, and streamlined data pipelines across platforms and departments.

K8s is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.

The benefits Kubernetes brings to organizations in slightly more generic concepts. This is due to the fact that K8s can positively affect the workflows of various teams and departments at a time. What’s more, the benefits have an accumulative effect as you can capitalize on utilizing Kubernetes in the long run. Because of K8s open-source ecosystem.

 > Service deployment and adjustment.
 > Storage Alternatives.
 > Self-healing.
 > Automatic Selection.
 > Recovery Mechanisms.
 > Constant Monitoring. and many more...

Kubernetes environment is persistently becoming the leading platform for cloud computing. Consumers looking for dynamically-scheduled container-oriented DataOps solutions should consider Kubernetes-as-a-Service.

Just like Devops, DataOps is also a methdology it's not a tool.

Top DataOps Projects

Genie
Piper
Airflow
Naveego
Delphix Delphix is the industry leader in Data Ops and provides an intelligent data platform that accelerates digital transformation for leading companies around the world. Uses kubernetes.

Finally...

There is no doubt that Kubernetes is a true game-changer in today's software development. The platform helps you boost the overall performance and productivity when it comes to efficiently managing containerized services and workloads. Not to mention its huge time- and cost-saving potential. With Kubernetes at your fingertips, you can drastically enhance the efficacy of your DevOps, DataOps, and GitOps workflows while creating automated CI/CD pipelines

Report on kubernetes

cnu1812 — Tue, 04 Jan 2022 07:22:28 +0000

Report on kubernetes.

Before getting deep into k8s let us understand what is the problem that k8s is trying to solve!

Suppose when we are talking about things like search (Google search, Bing, Yahoo), YouTube. Architectural solutions that worked before may longer work. Either the scale is so large that things simply break under load. Initially k8s was uniquely designed for Google as project7. To allow the data-centers to run processes at very high efficiency. However the primary goal of Google is cost saving. And one of the benefits is it increases productivity enormously.

Let us talk about the survey(report)conducted by the DOK(Data on kubernetes) community.
For the purpose of this report, the research
firm only surveyed individuals whose
organizations whose organizations are
currently using, evaluating, or planning
to use Kubernetes. This included an
international audience of 502 respondents.

AS PER THE REPORT:

k8s is highly adapted i.e 49% in the IT industry.
Productivity of the organization has increased enormously after adapting k8s.majority are 2x or more productive.
Because of consistency, standardization,and simple management many organizations are okay to run stateful workloads on k8s.
To be fact the adaptation of kubernetes is most in the pandemic i.e 2020-2021 to say during this pandemic there is a digital transformation everywhere hence the more traffic,more adapted to k8s. And the fun fact is COVID takes kubernetes to the next level.

There were some of the challenges for tech-leaders of running data on kubernetes.
One of the challenges is the lack of qualified talents.
So, there are many opportunities for k8s learners.

The state of data on kubernetes says that 90% believe that k8s is ready for stateful workloads whereas 70% of them are already using it on production.

Question: I just wanna know the answer form dok.

Such a hype is created on kubernetes nowadays. Will it be in the same position in future like let us say 5 years.?

2.
Suppose if your grandmother asks you what kubernetes is? how will you explain her in her way?

Report link

INTRO TO KUBERNETES!

cnu1812 — Mon, 20 Dec 2021 17:13:21 +0000

Intro to Kubernetes..!

$Let us talk a bit about application architecture.

Before any product has to be delivered.The engineering team needs to decide application architecture (eg:monolith,Microservices)

In monolith architecture, application tiers can de described as:
!part of same unit, managed in single repository,sharing existing resources (eg memory),developed in one programming language, released using single binary.

In Microservices architecture, application tiers can be described as:
!!Managed in separate repositories,own allocated system, implemented using the programming language of choice, released using its own binary.

Let us talk a bit about Docker!

Docker is a set of platform as a service (Paas) products to deliver software in packages called containers. There are plenty of tools used to containerize services. however, Docker has set the industry standout for many years. To containerize an application using docker, we use: Dockerfiles, Dockerimages, Docker registries.

Why docker came into existence..?

whenever you are designing a project it works fine on developer's machine.but as soon as that project is moved to someone's else computer usually fails to work with same performance so the classic problem is "It works on my machine".To solve this problem docker came into existence.

Let us talk a bit about Kubernetes..!

Kubernetes is a container orchestrator framework is capable to create, manage, configur thousands of containers on a set of distributed servers.
(To be simple: It is used to automate the deploy/release process and healing)

Kubernetes currently is a graduated CNCF project. The success for Kubernetes is because it solutionize:

Portability
Scalability
resilience
service discovery
extensibility and operational cost of containers.

IS KUBERNETES CERTIFICATION WORTH IT..?

cnu1812 — Thu, 16 Dec 2021 18:03:34 +0000

****Cockroach Labs is the creator of CockroachDB, the most highly evolved cloud-native, distributed SQL database on the planet. Helping companies of all sizes—and the apps they develop— to scale fast, survive anything, and thrive everywhere.

CockroachDB: A distributed SQL database built for Kubernetes!
¶ CockroachDB is the only database architected and built from the ground up to deliver on the core distributed principles of atomicity, scale and survival so you can manage your database IN Kubernetes, not along the side of it.
¶¶ Using StatefulSets, CockroachDB is a natural for deployment within a Kubernetes cluster. Simply attach storage however you like and CockroachDB handles distribution of data across nodes and will survive any failure.

•••CERTIFICATION•••
Any certification that you earn is worth it for a very simple fact that it adds credibility to your knowledge. Certifications are the proof that you gained hands on experience on a particular technology as well as it shows that you worked on various industry standard projects which gives assurance to the hiring personnel that you can hit the ground running as you have knowledge on the domain.
Talking specifically about Kubernetes certification.Since kubernetes has already become the new platform, there is a huge demand for people who can implement solutions with Kubernetes right now, and is going to remain so for next few years.

Difficulty of the Exam: Till date CKA exam conducted by CNCF, remains moderately difficult. Its practice oriented and not theoretical. People pass the exams with multiple attempts.

In simple:
•Kubernetes is here to stay for foreseeable future ( I could bet my career on it ).

••Its worth learning. The most exiting technology I have personally learning!

•••Even if you are not certified, if could demonstrate you know it well, its going to hugely help your career.

And finally SKILLS>>>>CERTIFICATION

!!!Free and golden resources to learn Kubernetes. Do checkout this link https://lnkd.in/e_8nNt3

What is Docker!

cnu1812 — Tue, 08 Jun 2021 06:20:56 +0000

Let’s get started…

Whenever a product is being designed the first and the only goal with that product is “I want to solve a specific problem (or) somehow I want to reduce the friction that users are currently facing”.

The problem statement that docker is trying to resolve…

Whenever a developer develops any product there are certain issues which are probably almost every time occurs. Well, that problem is whenever you are designing a project it works fine in your machine(the developer machine). But as soon as that project is moved on the production state may be on the servers (or) maybe somebody’s else computer in that case the project usually fails to work with the same performance (or) optimization when it is moved from one computer to another. So the classic problem is “it works on my machine”. Docker is designed to specifically address this exact problem “it works on my machine”.

Docker is just completely a different thing and it’s compatible with almost any programming language (or) any project that you are working on. It’s like almost a sheet of paper on which you can put up anything. Docker allows you to have a sealed airtight container and these containers are the absolute heart of the docker. These containers wrapups your entire code and these and these are portable portability is one of the charms of the docker. Now we can take the container and whenever you are going to work absolutely and exactly like how it worked on your machine. The containers are shared like your social status.

These containers are super powerful and super packed up and consist of a lot of things. These containers are packed up with your code, dependencies, configuration, some of the processes that you are running.

To summarize, we can divide this docker into three main essential things that it does for you:
1.It’s a client-side application program you can just install the docker and it can do all things for you include designing a container for you.
2.It can also act as a service and can be deployed on to ant server.
3.It also acts as a social networking platform where you can share your code easily.

Moreover, docker is also a company that does and manages all of this for you.

Getting started with Portainer...

cnu1812 — Mon, 31 May 2021 07:11:20 +0000

Intro:

Portainer is an orchestrator-agnostic tool for managing containers. It makes deploying and managing applications inside any Kubernetes environment quick and easy thanks to its intuitive UI. In addition to simplifying app management, Portainer provides deep visibility into the performance of apps/containers, overlays the security and governance necessary to deploy and manage containers at scale, and offers the platform management features necessary to get the most of out of Kubernetes. Portainer allows organizations to unlock the full power of containers without having to retrain existing dev/ops teams which helps drive down cost and speed up the process of technological transformation.

Portainer is open-source:

Portainer is an open-source tool for managing containerized applications. It works with Kubernetes, Docker, Docker Swarm, Azure ACI in both data centers and at the edge.
Portainer removes the complexity associated with orchestrators so anyone can manage containers. It can be used to deploy and manage applications, observe the behavior of containers and provide the security and governance necessary to deploy containers widely.
Portainer CE (open source) is trusted by more than 500,000 users worldwide. Portainer Business builds on the open-source foundation and enables organizations to run containerized applications at scale without needing to hire new people or re-skill existing teams
Specific features:

1.Application management:
The application deployment form is the easiest and quickest way to get an application up and running. You don’t need to know how to write complex deployment code or how best to deploy your application inside Kubernetes. You simply need to be able to answer some natural language questions and Portainer will determine the best way to deploy it for you.

2.Observability and Monitoring:
Deploying an application is only half the challenge when it comes to containers. Unlike monolithic apps, container-based applications are widely distributed and highly dynamic, which can create monitoring challenges for organizations. To monitor container-based apps properly you need to have direct and deep visibility into the underlying container platform. Containers can crash and be rescheduled in seconds, often meaning failures could go unnoticed by end-users, but this doesn’t mean there isn’t a problem. Through its close integration with the underlying container platforms, Portainer is able to help users not only identify issues in the application deployment but also identify issues in the container platform itself and provide a live visualization of what’s running where and how.

3.Governance and Security
Portainer is designed to be used in a centralized manner, at scale, across entire organizations. To achieve this safely, Portainer provides a raft of security-related features that ensure only authorized users can access Portainer and can only operate within the boundaries of their assigned privileges.

4.Platform Management
The core value of Portainer is its strength in application deployment. Regardless of how you choose to engage with Portainer, be that via the GUI, our extensive API or Git integrations, the ease with which you can deploy container-based applications is unparalleled.

Why Portainer..?
1.Simplify your app management.
2.Reduce your operational cost.
3.Secure your container platform.
4.Future proof of your IT strategy.

Conclusion:
This is an open-source tool with a business-specific upgrade, it is an immensely powerful tool with the potential to transform the way you deploy and manage containerized applications .whether you are setting out on your Kubernetes journey or you are well advanced, It can simply and improve your workflows, helping you to save time, reduce errors and unleash your container potential.

Complete details and demo will be presented on the next blog.
You can connect me on:
https://www.linkedin.com/in/v-sreenivas-985088203/

Open mainframe Project and perks for contributing to open-source

cnu1812 — Sat, 29 May 2021 13:14:54 +0000

About open mainframe work

Open mainframe project which really focuses on the opensource ecosystem. On mainframe particularly focuses on the projects that are really interconnecting the entire enterprise. Enterprise is very hybrid and heterogeneous in their IoT infrastructure at least what they see in forward-looking enterprises. So the projects and efforts they host are meant to be sort of the interconnecting piece of the mainframe within the world.

Linux Foundation

The Linux foundation host over 400 open-source projects. Pretty much every horizontal and vertical out there and a lot of it really focus is that concept of innovation built on sort of that decentralized trust. The projects that are hosted here are all focused on the area that they all come here because they need that neutral area to collaborate.

Why would students do contribute?

The basic thing is you’re just learning these technologies and the best way to always start in any open-source project is to show up to contribute to starting playing with code, to start adding some features. Even just correcting little errors adding some tests (or) documentation. This is getting you to sort of familiar with a bit of the code and at the same time you’re getting familiar with the community. When people are really getting involved in open-source projects there’s the one half of participating. You use this code in such a way and you want to help contribute some of the use cases the code behind the use cases that you drove and bring those back into the open-source. And really amazing thing is to build your resume. You sort have a unique opportunity to be working with leaders in industry and leaders and leading companies and finally getting hired!

Difference between open-source and Internship...

Open-source community stand point is the output at your work is open-source, so people can see exactly what you’ve done the impacts that you have had and you have a true portfolio building begin.

Internship shows you can work with people in an office setting and have an idea of how offices work, a good recommendation from the company you intern for can show that you have programming skills and other important soft skills. Internships also have the potential to lead to a full-time position within the company. Contributing to open-source projects really just shows you can program. Basically working on open source projects is similar to community service, its helpful and can make you feel good, but it’s not the most efficient/best way to improve a resume.

What is mainframe work..?

It is basically a computer system that is designed really around four key design principles:
(a)Security, (b)Stability, (c) Performance, (d) Scalability.
And we see that showcased in a number of key industry areas where you need all of those aspects of your computing system turned up to an algorithm.

Example: Credit card operation, booking tickets, etc.

Open Mainframe Project

It was founded six years ago as of 2021, was founded on the case that open-source is such a critical part of where the mainframe community is going, really to be fair a lot of open-source actually came from the mainframe world. When you go back to 1955 with an organization called “share” that is really the root of where open-source came from for lack of return I mean it’s words roots computer operators f the time had this new Ibm system and they needed to know how to use it, they needed some help in being able to collaborate and so they came together and they shared insight, and they shared code! , they shared experiences, they shared insight and that has built over decades and decades and decades….. So this was really an effort here of how do we get the larger mainframe community which is also very passionate, very interconnected and also looking forward to long term sustainability of collaborating in open source. And it really started with work around the Linux usage on the s390x mainframe platform and in time it’s grown also to open-source projects. Open mainframe project sits in there and fills the piece and it hosts the project that is very key to making that connection and as a way, it’s really bringing the mainframe into the mainstream.

Z-architecture

Basically, it is similar to the s390x mainframe platform. It is the architecture of these boxes of how they’re put together. And the degree of processing power, degree of redundancy built into the box, degree of additional hardware bits really make these boxes the most performant resilient scalable.

How beginners can contribute..?

All the projects are completely transparent open projects that anybody can get involved with not only source code available under OSI licenses but getting involved in the community very easy and What I recommend is people get on the mailing list, get into the slack channel because there are tons of communications that are happing on both sides you can start to get a feel for what the community is about what the current topics are things of that nature. Many of these projects will also maintain good first issue, tagged issues so you can go there and check what those look like and maybe use those for some low-hanging fruit that you can get involved with. Each of them has a few different tech stacks to take some of the advantages.

You can find all Open mainframe Projects on Github. Even beginners can contribute.

They are also providing mentorship program:you can apply