DEV Community: OCP

DevOps: The Silver Bullet for Digital Transformation

Yogesh Dipankar — Fri, 10 Mar 2023 05:02:50 +0000

Digital Transformation can be loosely defined as transforming the legacy systems running on old hardware and old programming stack to a highly scalable and highly available platform like Cloud, Big Data Cluster Computing platform using modern tools and technology.

DevOps is a set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality. (Definition from Wiki)
Based on my experience working on multiple Digital Transformation Projects with and without DevOps, I have concluded that to make “Digital Transformation” really successful, DevOps is the critical ingredient… instead I would say it is the “Silver Bullet” that can make a big difference

“Digital Transformation Projects is about massive scaling and faster time to market.”

Agile/Scrum has offered a great way to improve software development practices. DevOps provides the much-needed technology toolset and echo system for churning out iterations that create customer value. Now talking Digital Transformation Projects is all about massive scaling and faster time to market, and it badly needs the ecosystem which DevOps offers.

Right Prioritization is Key for success of DevOps

The waterfall was about piling up features, requirements and releasing all at once. Still, with DevOps, if proper prioritisation is done — in every iteration, value flows directly from development to deployment stage to customers like a pipeline. Hence the concept of build and release pipeline became immensely popular; right prioritisation of work is essential here.

With CICD pipelines (i.e. Continuous Integration and Continuous Delivery ) at your disposal, DevOps enables development teams to deploy applications much faster. With the advent of Cloud computing, code deployment and infrastructure creation ( or infrastructure setup) can be done on a click of a button. Cloud virtually makes infinite infrastructure available to everyone across the globe — So this takes care of the massive scaling and time to market.

We have seen that with CICD and cloud computing, the time to market is super fast… but what about security? DevOps has something called as DevSecOps. DevSecOps bring together the Security and Development teams to ensure the highest security standards are followed right from the project's inception. Examples are adding SonarQube, Blackduck, WhiteSource, Checkmarx, and other security tools in the CICD pipeline for early detection. Along with this unit testing, test automation and monitoring can be incorporated into the CICD pipelines, thus making it a continuous value-producing engine for the end-users

BTW, DevOps is not only about Continuous Integration / Continuous Deployment Pipelines, but it does much more; it creates an ecosystem for setting up high standards, the right design from development, testing, and deployment till monitoring and fixing the issues in production in a quick time.

Now let’s look at different phases of DevOps.

Plan: Tools like Azure DevOps (ADO), Jira, Collab, and Assembla can help to make this phase as effective as possible.
Code: Software design, code development, and maintaining the code in Git, GitLab, GitHub, Bitbucket, or any other popular version control system
Build: Compiling the developed software using CI pipelines and package management ( artifacts, NPM repository, etc.); it also has triggers, gates, and approvals.
Test: Testing and quality assurance by integrating the CI pipelines with automated quality gates
Release: ready for release after successful testing; acceptance criteria can be integrated into CI pipelines
Deploy: ready for seamless and automated deployments in Development, Test, Preproduction, and Production environments.
Operate: Once in production, the operations team will ensure the software is running optimally.
Monitor: Continuous monitoring for performance, response time, and errors; tools like Application Insight, Log Analytics, and Splunk can be used here.

All these phases are interconnected. Suppose the planning and prioritisation are done correctly. In that case, it can impact the design and coding phase resulting in poor standards, bad design, and coding will make software hard to build and test, leading to manual testing. Also, the continuous deployment of such software will require some clarification. So getting right every activity in every iteration can work wonders and help create a great product.

Great products, architectures, features, requirements, and design can be built best by self-organising teams, so empowering teams and enabling them on the DevOps path should be the number#1 focus of leaders

Implementing and incorporating DevOps in the organisation requires a shift in the organisation’s culture. With the proper focus, the process can be gradually and systematically implemented. Primary responsibility lies with the leadership to address the skill gap and help the team unlearn old ways and learn/ relearn new ways of working.

Companies like Amazon, Netflix, and Facebook do thousands of daily deployments. These deployments are fully automated, which would only be possible with DevOps; I have seen deployments running for four days in my previous projects ( without DevOps and cloud ). With DevOps, I have experienced seamless delivery and rolling out of quality products to existing and 100s of new customers with a click of a button, completely automated.

To become a market leader, organisations have to embrace and implement DevOps in a big way, minor adaptions here and there will have limited benefits.

I want to encourage all developers, testers, and leaders to embrace DevOps and Agile as top priorities. All development teams should develop DevOps skills; on top of this organisation should set up a DevOps R&D team and continue to build expertise in DevOps, come up with design patterns for different implementations, come up with new automation techniques and keep on improving the DevOps practice (as there is no end to automation). A quick reality check for organisations doing Digital transformation is maturity level in DevOps; the ones who have got it right have made a cautious decision to excel in DevOps.

DevOps and automation also bring substantial cost savings options; if you don’t need a particular server or pipeline, you can dismantle and regenerate it when required. Example: shutting down Dev/POC/Test environment over the weekend when not in use can bring substantial cost savings

By now, it’s evident that DevOps is one of the most sought-after skills; for the last couple of years and in 2019, 2020 StackOverflow survey has rated DevOps as a top skill.

Here are the StackOverflow survey links mentioned in the article

DevOps is the future and will completely change how we work, create new roles and responsibilities, and we have to be ready for that.

Example Full Stack Development: here developer does everything from requirements gathering, development, deployment, testing through automation; another example is Site Reliability Engineer: profile with a complete understanding of development, deployment, and monitoring.

If you are planning to learn DevOps but unsure where to start, do contact me. I will be more than happy to help you.

For those already using DevOps, let me know if you agree with the bold statements made in the article, do share your experience and success stories; I would love to see your feedback and comments.

Thank you!

Cheers!
Yogesh

Agile way to solve Rubik’s Cube in 10 steps

Yogesh Dipankar — Mon, 13 Feb 2023 09:12:33 +0000

My 2023 New Year Resolution is to continue learning at least one new thing every month :-), Just like in 2020, my now 9-year-old daughter (Shreya) and I started the year by taking on solving the Rubik's cube as our first learning challenge. We still remember how we discovered a helpful YouTube video from WIRED, which taught us how to solve it systematically.

After going through the video multiple times and writing down the steps(or algorithm) on a piece of paper, we realized that to solve the Rubik's Cube, we must break the problem into small steps and solve it one layer at a time. And while doing this, I realized that the approach is very similar to what we follow in Agile!, so here I am writing my understanding in form of a blogpost

The approach of breaking a problem in small, achievable yet challenging tasks or goals is at the heart of Agile.

When breaking down complex problems into manageable steps, it's essential to approach the process with patience and attention to detail. For teams using the Waterfall Model for years, shifting to a more incremental approach can take some adjustment. However, small tasks are the key to success.

I often tell my team to imagine these smaller tasks as steps on a staircase. By carefully and methodically making their way up each step, they increase their chances of reaching their target within the desired timeline. On the other hand, paying attention to even a single step can lead to a misstep and, ultimately, fail to achieve the desired outcome. So, always be mindful of each step along the way.

Research suggests that organizations new to agile find it challenging to cope with; it's basically due to the waterfall conditioning of teams and leaders over the decades; it's difficult for them to see the benefits in small steps. Most of us are only worried about the outcome; instead of understanding how the problem can be broken in small steps, we tend to see this process of iterations as an overhead, and they question if Agile works; due to lack of clarity, they are not motivated to implement agile. Instead, they make resources slog and work unplanned instead of following a clean agile approach.

In waterfall there is a lot of uncertainty which adds artificial pressure, on the other hand in agile, it gives complete visibility of task at hand and bandwidth from day 1 as its considerably a smaller time-frame and smaller target.

Applying Agile methodology to solving the Rubik's cube can be valuable in efficient problem-solving. The process is broken down in Agile projects into smaller iterations known as Sprints. These Sprints can range in length from one week to four weeks, depending on the project and team bandwidth.

Here are the critical rituals involved in an Agile project:

Sprint Planning: This is where the team finalizes the goals and deliverables for Sprint and identifies the Product Backlog Items (PBIs) that need to be tackled.
Sprint Grooming: The Scrum Master and team members go through the PBIs, estimate their effort, and allocate tasks.
Sprint Refinement: Midway through the Sprint, the team takes a step back to assess their progress and make any necessary adjustments for the next Sprint.
Sprint Review: The team demonstrates their progress to stakeholders and business outcome owners and reviews the Sprint goals.
Sprint Retrospective: The team reflects on what went well and didn't go well during the Sprint and identifies opportunities for improvement.
Daily Stand-ups: The team holds a quick 10-minute call every morning to check the status of the PBIs and move them from "To Do" to "In-Progress" to "Done."

By following these Agile rituals, teams can work more efficiently and effectively to solve the Rubik's cube or any other complex problem.

To solve Rubik's cube, we broke the problem into multiple Sprints; first sprint - is a high-level understanding of the approach, knowledge of different cube notations, and other moves like a left trigger, right trigger, left pull, proper pull, to get comfortable with the cube to some extent.

Sprint 0: Get Familiar with the Cube / some intro:

Sprint 1: Make the Daisy ( shown in the image ):

Sprint 2: Create the White Cross

Sprint 3 : (Learn to Hold the Cube): This process can be thought of as further understanding how to efficiently handle cube

Sprint 4: Solve the Bottom Layer & 1st Layer ( 1 on 1 free :-) )

Sprint 5: Solve the Second Layer:

Sprint 6: Create the Yellow Cross:

Sprint 7: (Cube Notation):

Up until this point, the process of solving the Rubik's cube has been relatively straightforward. However, from this point on, the team will need to follow more complex algorithms, which require a good understanding of cube notations, such as F and F', R & R', Left Pull, and Right Pull.

Sprint 8: Solve the Yellow Face / Yellow Top:

Sprint 9: Position the Corners:

Sprint 10: Position the Edges, Final Step and its solved

The approach here is to solve one layer at a time only.

Solving a Rubik's cube can be seen as a metaphor for solving complex problems using an Agile methodology. My daughter and I started our journey by learning the basic steps and algorithms in the first week, spending 40-50 minutes solving the cube. As we progressed to the second week, our understanding of the cube notations and algorithms improved, and we solved the cube within 30 minutes by following the notes.

By the third week, our confidence and understanding had grown to the point where we could solve the cube without relying on the notes, taking only 15-20 minutes. In the fourth week, my daughter had become a pro, solving the cube in just 1.5 minutes.

This example highlights the importance of breaking complex problems into smaller, manageable tasks, just as one would break down the Rubik's cube into smaller steps. The Agile process requires discipline, dedication, and patience to achieve the desired results. Just like solving the Rubik's cube after the seventh step can be challenging, building an Agile mindset can also be difficult, but with determination and discipline, it is possible.

Agile development process is a disciplined process, without discipline you cannot achieve agile mindset!

To conclude, the principles of Agile are not limited to software development alone. Its concept of breaking down complex problems into smaller, manageable tasks can be applied to a wide range of goals and objectives, making it a versatile and powerful tool. By adopting an Agile mindset, you can increase your chances of success and achieve your targets in a more efficient and organized manner.

Thank you for taking the time to read this article. I hope you found it informative and inspiring in regards to the Agile mindset. I would appreciate it if you could share your feedback and opinions to help me gauge my understanding. I particularly invite Agile coaches, product owners, Scrum Masters, and anyone working with Agile to provide their honest input. Your input will be greatly appreciated and contribute to the improvement of the content.

Cheers !
Yogesh

Time Series Database and Analytics using Azure Data Explorer

Yogesh Dipankar — Thu, 15 Dec 2022 12:29:19 +0000

In this article, I will be giving a summary of the Time Series Database and how it can be used for Advanced Analytics; we will also cover Azure Data Explorer: A Powerful time series database; this is one of the first articles in the series of Timeseries: ADX articles that I am planning to publish, so lets start which these basic three questions

What is a Time series Database?
Why Time-series DB?
How to do Advanced Analytics with ADX: Time Series DB

What is a Time Series Database?

A Time series database is a database capable of storing high-frequency, streaming data in time series; it stores information or data part of a “time series,” which is a set of data points associated with timestamps. Due to this, the time series database size grows quickly in a short time. We will discuss its characteristics further; let's look at popular TimeSeries Databases.

Why Time Series?

The data coming from Sensors, Scooters, Cars, and Devices running for long hours is usually coming in milliseconds, or nanoseconds, so to store and fetch this data from relational databases on no SQL databases is not a perfect match, storing and optimising this on timestamp till nanoseconds level and ability to fetch, partition the data on timestamp is inbuilt to time series database.

If you see the characteristic of the data here, it is a stream of data coming at a very high frequency, like every second. Time dimension takes a crucial role here. A time series database helps to partition this data on time.

The best way to analyze the things which are constantly fluctuating i.e., Non-Stationary Data over time is by using Time Series

Partitioning data on time helps sort the out-of-order data and ordered data separately. This immensely improves the analytical queries, data engineering jobs, and data scientist models running on it. Also, the data is indexed on time, so fetching time based on time is fast. It also supports time-based joins across multiple tables for effective analytics.

A few of the things that a time series database is good at for the live streaming data is

a) Low latency for data ingestion from the sensor to the database

b) High write throughput as the time series data is live, write-heavy, and highly concurrent

c) High read throughput as the time series data is to be read and instantly presented on BI Tools

d) High data compression within a day, millions of records are added; these should be compressed while ingestion or while storage to avoid substantial disk size consumption

Azure Data Explorer is a fully managed PaaS service from Azure; it comes as a very good option for Time Series Database it fits in all the above requirements, and it also has great computing power to process the live streaming data and historical data stored in ADX cluster

Time Series can be created and analysed within a few seconds in Azure Data Explorer

ADX can generate real-time alerts based on the data that is ingested into ADX; it can run rules against the data in flight and generate alerts, emails

ADX also has very good integration with all Azure Services like Logic App, Event Hub, Blob Storage, and Application insights the journey of ADX ( Kusto ) started at Microsoft as a log aggregator, it is being used as a log engine of all Azure services for last 10 years, in 2021 it was publicly launched to be used as Time Series Database

It has some great machine learning features, trends, correlations, and models can be executed on the fly, and visualisations can be generated using UI/UX and BI tools

Other striking features of ADX

It is used as a logging solution for all Microsoft Services in the background for the last 10 years
So it's a proven solution, as per Microsoft documentation daily 100 Petabytes of data is ingested in ADX, a very powerful data ingestion system
Use intuitive and powerful KQL ( Kusto Query Language ), Azure Log Analytics queries are actually KQL queries so upskilling for Azure resources should not be a big challenge
For SQL users there will be a bit of learning but should be easily manageable
Direct integration point with all Azure services and multiple other third-party services
Plotting trends and applying toner machine learning algorithms becomes easy using ADX features

How to do Advanced Analytics with ADX: Time Series DB?

ADX for Time Series Analytics

Inbuilt support for TimeSeries creation
Inbuilt functions for analogy detection and forecasting
Inbuilt support for ML algorithm integration

In the next few articles, I am going to cover different aspects of the ADX Timeseries database and do a deep dive into it

Let me know what role you play: Are you a Data Engineer, Data Scientist, or Data Analyst, and are you already using any Timeseries database, also let me know what aspect of the Timeseries database you would like to know more - about so that I cover that in my next article, also if you are interested in learning ADX: Time Series database architecture and implementation, I will be happy to share

Hope you enjoyed the article, thanks for reading

Cheers,

Yogesh

CI/CD in Power BI to simplify Release Management in five simple steps

Yogesh Dipankar — Wed, 09 Nov 2022 11:37:18 +0000

Before I dive deep into the main topic of CI/CD ( Continuous Integration and Continuous Delivery) of Power BI, I would like to give a very quick overview of Power BI.

Power BI is a popular and powerful BI ( Business Intelligence ) tool. For four consecutive years, Power BI has been reported as a leader in the Gartner Magic Quadrant ( link to garnet in 1st comment ). Other popular BI tools are Tableau, Microstrategy, Qlik Sence, Superset, and many more; the list is long.

The primary function of any BI Tool is to:

Collect and transform data from multiple sources
Help uncover the data trends and outliers
Help create easy and impactful data visualization
Action-oriented insights

"Gartner has recognized Microsoft Power BI as a Magic Quadrant Leader in analytics and business intelligence platforms."

As the business grows, it needs more analytics, more accurate data in a single report, and quick access to trends; as a developer, the challenge is the get out the dashboards and reports in the fastest possible way along with reliable data, which is more recent, more accurate, more realtime. Manual deployments are error-proper and time-consuming, so CICD comes to the developers' rescue.

"Doing deployments and release management manually can be error-prone and time-consuming."

CI/CD Process can help the BI Developers achieve these five things

Simply the deployment process: improve team productivity
Reduces manual work and errors: fewer bugs
Reduces deployment time: quick turnaround time
Ease of use and improved collaboration: teamwork
Automate the process and make it reusable: automation

"Seamless CICD process reduces errors, deployment time and increases team collaboration"

By now, we understand the benefits of CICD, so let's look at the different ways; there are two approaches to implementing CICD in Power BI.

Power BI Service Deployment Pipelines ( within Power BI Service )
Azure DevOps and Power BI CICD using Service Principal ( with Azure DevOps )

In this article, I will cover the 1st Option within Power BI Service.

Without CICD, the BI creator has to publish the Dashboards manually in Dev, Test, and Prod. There are some difficulties in publishing a PowerBI Report directly from PowerBI Desktop, which includes

a. The developer must be the admin or have access to the Workspace; otherwise, can't able to do any procedures.

b. PowerBI Desktop has to be connected to all the environments ( Dev, Test, and Prod ) to do the deployment which is not very safe

c. The developer needs to manually change the data source settings for every environment

Power BI Deployment Pipelines is a simple five Step process

Create a deployment pipeline with DEV, QA, and PROD environments
Assign workspace to each environment
Develop and Publish the contents to the DEV workspace
Deploy it in the QA and PROD workspace with a click of a button
Share it with users through the Workspace or the App

Detail Steps

[Step 1] Create Deployment Pipeline

Go to app.powerbi.com and look for the rocket-like icon, click on the icon, it will open an option to create a pipeline, hit that button and it will prompt for the pipeline name, enter CICD Demo Pipeline, once you hit the create button it will present you three environments ( DEV, TEST, and PROD ) and an option to assign workspaces

[Step 2] Assign workspaces

Here we can see three environments ( DEV, TEST, and PROD ) and an option to assign workspaces; from the drop-down, give Development-Workspace to the DEV environment, QA to Test Environment, and PROD to the production environment. It should look something like this before and after the assignment.

[Step 3] Develop and Publish Dashboard to Development Workspace

For this article, I will be using a sample dashboard from the Microsoft portal, the link to the portal is in the first comment

Since I am reusing an already developed Sales & Returns Dashboard, let's open it into Power BI Desktop and hit the publish button in the right corner, choose Development-workspace while deployment

[Step 4] Deploy to (test) QA >> Deploy to Prod

After the development of the dashboard is done and it's published in a development environment, the Data Flow, Data Set, and Dashboard will visible in the Development environment to publish, hit the "Deploy to test" button in the Dev environment, the Power BI Deployment Pipeline will copy all the contents from Development to QA in a couple of seconds if not minutes

Ready for Test/Stage Deployment (empty QA and Prod Stages)

Hit the Deploy Button and the changes will be available in QA

The deployment might take a few seconds if not minutes based on the size of the reports, Do a round of testing and take the sign-off from QA and Customers, once done hit the "Deploy to Prod" button, and the contents will be copied to the Production workspace.

[Step 5] Share it with the users through App or through the workspace

Go to the workspace where the deployment is done ( In all three stages of the pipeline, a link to the workspace is available, navigate to the link and view the report

Other Features

Comparing stages

If there is any difference in DEV, test(Stage), or Production environment, that will be easily visible by an indicator in-between two stages, it does a comparison and then the indicator shows two states:

Green indicator: When the content item in both stages is exactly the same.

Orange indicator: When the content from both stages appears different or if there is a difference in the number of items.

Deployment History

A sequential, one-click release pipeline, with comparison and release history, is ready for your daily use, I hope you find this article informative and helpful to implement CICD in Power BI. Power BI Developers and Release managers let me know your thoughts on this article through comments

""Deployment Pipeline, Camparison Tool, Deployment Histroy, 3 top features in five steps, what else do you need ??""

Yes, let me know what else you need, it can be a) how to trigger the pipelines automatically b) approval flow in the pipeline c) selective deployment. Let me know your wishlist in the comment section and I will cover that in the 2nd part of this Power BI Deployment Article series

Cheers

Yogesh!