DEV Community: Najib Radzuan

DevOps Adoption in Geographically Dispersed Organizations

Najib Radzuan — Sun, 03 May 2020 09:16:37 +0000

What Does it Mean to Be Geographically Distributed?

If the developers, IT operations and primary stakeholders are all situated in the same workspace, the team is considered to be co-located. Otherwise, the team is either dispersed geographically or divided in any way. They can work in different buildings within the same geographical area (maybe the team is spread over different office buildings in the same city or some people work from home. For simplicity’s sake, this covers teams that are either geographically dispersed or that have team members that are geographically dispersed. We can divide into 3 categories below;

Co-located. Teams where the developers are working side by side in a single room also called a “war room,” “box” or “tiger team office.”
Near-located. All team members are within acceptable driving distance (e.g. if it would be feasible for all to meet at a single location each day if necessary).
Far-located. Any or more people are far enough away to potentially need to take a flight to physically meet with other team members.

In this article, we will focus on the Far-located team doing DevOps adoption which I’m currently experiencing and this article also gives you insight on how to succeed in DevOps adoption in a dispersed organization.

In A Nutshell

The DevOps concept emerged to bridge the disconnect between the development and IT Operation teams, it also helps the deployment of product into production within distributed organizations. DevOps is not about the destination, it’s about a journey, or some said it’s an odyssey (Don MacVittie), hence DevOps adoption in a geographically dispersed organization is possible if we do it properly. With new technology and ways of working introduced by DevOps, the destination is not a big issue when the organization wants to do DevOps adoption within different time-zone or regional teams.

Advantages vs Challenges

There are several benefits when the geographically dispersed organization implement DevOps culture and method:

ADVANTAGES

1. Access to a larger pool of skilled people
Many organizations struggle to employ their IT departments exclusively with local people, particularly when unique skills are required like DevOps Engineer, Leader, Manager, etc. Looking out from another side of the country is probably the best solution to the organization. The organization also will gain vast experience workers in DevOps since their distributed workers come from different backgrounds and experience, it’s different when only hiring local workers, they cannot gain this type of experience and skills.

2. Lower development costs
On the surface, savings are the result of wage differentials between countries and often even cities within the same country. However, such savings can be easily lost due to increased overhead associated with geographic delivery — you must understand the total cost of ownership (TCO) and not just hourly system costs when estimating cost savings. Cost reductions can still be made, but you must be sufficiently diligent to achieve them.

3. Quicker time to market/production
When the team is diligent enough to obey the “follow the sun” approach, it is given for faster delivery times. The basic idea is that globally dispersed teams will do its work throughout the day, and then turn off the job to a team that is a few time zones away. This team will do some research, then turn it over to the next team.

CHALLENGES

1. Communication challenges
The most effective means of communication between two or more people are face-to-face around a shared sketch space, such as a whiteboard or a piece of paper. Naturally, you have to be together in the same place. With the distribution of the data, you begin to rely, as you can see in Figure 5, on less successful communication techniques that provide more consistency. You can not identify body language that includes several important signs of communication if you are not personally aware of them.

2. Culture challenges
As the team is more dispersed, cultural challenges between sites are often increasing. Different cultures have a different work ethic, treat intellectual property differently, have different ideas about commitment, may be less likely to embrace self-organization, have different holidays, different approaches to things, and so on. Something as “simple” as it means when someone says “yes” can be very challenging in practice.

3. Temporal or Time challenges
When people live in various time zones, it is more difficult to locate similar working hours, increasing connectivity difficulties. To address these challenges, you will find that you need to create more documentation than you would normally.

How-To Steps

Agile Practise
A successful journey starts with the right people in the right DevOps practice and role with the right skills — and a willingness to collaborate. The Agile method is promoting close collaboration between the team and across team members. Agile is all about quick execution and quick releases. There is no scope of perfection. The basic concept here is that communication and collaborate frequency depends on what stage of development your team at:
High at the initial or requirement and implementation level
Low at the production stage and design stage.
Below the Agile stages and communication frequency for Far-located teams;

The Agile process includes Daily Scrum meetings within the dispersed or offshore team and onsite team, hence they have a chance to update or virtually meet frequently between them every day. Sometimes, In-person meetings, visiting the location of the offshore team, or welcoming them to your Headquarter (HQ) office are perfect for building confidence, creating bonding and efficiency and learning.

2. Have Buddy Program or Ambassadors Travelling Between Far-Located Site and Onsite Office.

Normally the organization has an introduction day or induction day for the newcomer in the company to introduce all about the organization’s product, environments, vision and etc. With geographically, time zone, and culture differences, the best to promote collaboration between the onsite and far-located team introduces the Ambassador and Buddy program. Ambassadors are people who regularly travel between sites, often technically senior people or senior business experts, who share information like culture, organization management style and also the technicality of their product between the sub-teams. Ambassadors have short engagements away from their home site, typically a week or two in length and usually, it happens on a quarterly basis, because of the pressures it puts on the people doing the actual traveling. As a result, you’ll have several people flying between sites at any given time on a reasonable rotation schedule. The Ambassador plays an important role to get the team together at the beginning of the on-board far-located team and ensure everyone knows the company objective and to maintain effective collaboration between teams. The Buddy program is each onsite and far-located workers have their pair to ensure the culture barrier between onsite and far-located can slowly take away from teams. They have a specific or fixed time to talk about culture and how their daily routine, hence getting a relationship close between them can make them focus and motivate them to work more frequently.

3: Sprint Release Plan

The first and foremost requirement for building a successful geographically dispersed DevOps team is to lay a proper change or release management process. People are reluctant to change and thus, laying complete guidelines to change with DevOps is considered very useful. It is a checklist that represents standards as to which apps should be introduced and when they are done. It also acts as a base for tracking progress within the project. Releases can be intermediate deliveries made during the project or final delivery at the end of the project.

4. CI/CD Pipeline

With a shift in mindset, a successful DevOps team often needs important improvements to be made to technology, such as pre-production automation, testing, release and integration. The CI/CD pipeline is at the core of DevOps as they encourage collaborative and cooperative work. In other words, the value of CI/CD is no longer a question. The benefit of CI/CD pipeline approach;

Developers can stay focused on writing code and monitoring the behavior of the application in production.
QA workers and product stakeholders have easy access to the latest, or any, version of the system in defined CI/CD stages.
Product updates are not stressful.
Logs of all code changes, test, and deployments are available for inspection at any time.
Rolling back to a previous version in the event of a problem is a routine push-button action.
A fast feedback loop helps build an organizational culture of learning and responsibility.

5. Automation

The idea of an autonomous team is that developers, as well as members of the operational teams, should be able to make choices without a complex decision-making process. Teams are able to operate in a supportive environment where there is no scope of discrepancies. Instead of waiting for manual approval for the code to be deployed in the Testing environment, you can rely on a version control system that can be quickly audited. Without the need for a manual sign-off, the team can automate the deployments and speed up the testing cycle. This would also ensure that the team builds quality products in the development process. The concept of self-testing code also allows regular and low-risk deployments. The following Toolchain that can help you to archive Automation approach in geographical DevOps team;

Team Collaboration — Atlassian Jira, Microsoft Team, Microsoft Office 365, Slack.
Continuous Development — Azure DevOps, AWS CodeDeploy.
Continuous Testing — Selenium, Appium, Travis
Continuous Integration — Jenkins, TeamCity, Bamboo, GitLab, CircleCi.
Continuous Deployment — Chef, Puppet, Octopus, Azure DevOps
Continuous Monitoring — DataDog, New Relic, Nagios, AppDynamic.

6. Monitor
Continuous Monitoring also plays an important factor in geographically DevOps team, since the onsite and far-located team has different region and remotely meet, the monitoring tools will help them to figure it out when something happens either in infrastructure metric and application abnormally. It also reduces the time needed to address customer feedback. The following is the other benefit from continuous monitoring;

An effective monitoring tool improves system performance and productivity and helps you reduce (or even eliminate) downtime
You can adequately plan for upgrades and new projects, and better allocate your time and resources.
You can detect problems — and solve them — before they impact users.

Your monitoring tool can integrate with the Alerting tool which helps lay the foundation for your alerting policies, so you can determine who to notify, how to track issues and outcomes, and how to prioritize remediation.

Conclusion
The organization will face numerous obstacles when developing a far-located or geographically dispersed team for DevOps. There may be problems relating to teamwork and coordination or constructing materials that can be replicated through various projects within the organization. One solution proposed by the industry or DevOps experts in the establishment of the Centre of Enablement the coordination of teams and processes. A DevOps strategy and team structure have something to achieve from implementations in the long run. Any organization will have a strong start if they succeed in bringing more collaboration in different positions and a sense of shared responsibility in order to derive more quality and productivity in their deliverables.

8 Key Challenges in Adopting DevOps: Part 2 — Solutions

Najib Radzuan — Tue, 28 Apr 2020 05:27:43 +0000

In part one (here) of this two-part blog series, I looked at the 8 key challenges I see organizations facing when they are adopting DevOps. Now, let’s look at some solutions.

Change is hardest at the beginning, messiest in the middle and the best at the end. — Robin S. Sharma

1.Making Culture Function According to DevOps Principles
DevOps leaders and change agents will need to find ways to continuously educate teams and people within the organization about what DevOps culture looks like and why it accelerates the flow of value. Here are some things I have tried that have worked:

Learning and Communities of Practice
The DevOps personnel can arrange internal training or introduction presentations for DevOps adoption and give exposure to DevOps culture. This way they can promote and meet face-to-face with everyone in the organization. Face-to-face collaboration is preferred over using email or video conferencing as humans build trust faster this way so it’s recommended to undertake a roadshow if teams are dispersed, aiming to meet everyone at least one in order to build relationships.

DevOps Knowledgebase & FAQ
DevOps teams can create a knowledgebase or Frequently Asked Questions (FAQ) and share with all people within the organization, hence everyone knows where to get information to relate to DevOps where they need it. The visibility and easy access to the information that motivates them to search and read it by themselves and even contribute. This kind of information can be held in collaborative platforms such as Atlassian Confluence or Microsoft Teams.

Applying Westrum Organization Culture Practices
We can use Westrum Organization culture practice to make a generative culture that cultivates data stream and trust by looking at the six parts of Westrum’s model of hierarchical culture, concentrating on those practices found in the generative culture;

Here how you can build a Generative Culture;

2.Addressing Resistance to Change
Leaders have to expect that people will resist change. According to DevOpsologist, Philippa Hale , in her article about stakeholder mapping tools and she discussed how we can address certain group people’s moods and emotions toward changes then we can apply different engagement strategies to approach them toward DevOps initiatives. There are 6 “behaviour profile” and how we can engage with them as below;

Unengaged

Spectators

Cynics

Critics

Enthusiasts

Advocates(Champions/Experts)

To summarise based on the above, we can see all engagement approaches involved communication and stay informed about the DevOps adoption. We need to get closely work with all groups and be visible to them and give help whenever they need one.

3.Bringing Clarity to the DevOps Vision
Introducing the DevOps CALMS framework can help define the DevOps roadmap and goals. CALMS is a conceptual framework for the integration of development and operations (DevOps) teams, functions and systems within an organization.

DevOps leaders need to develop a clear roadmap for DevOps evolution with clear improvement phases. They must share it and make it visible to everyone within the organization;

4.Creating Cross-Team Collaboration
The development and IT operation teams need to learn to collaborate. This may mean creating cross-functional teams including both dev and ops, but this doesn’t work in many organizations. It’s often too dramatic an organizational change, or there is simply aren’t enough people to go around. Traditional technology department set-ups also tend to include deep subject matter expertise in IT operations around security and networking for example, so it’s difficult to see how to share these types of people across development or product teams.
What does help is having both the development and IT operations teams meeting regularly? If development teams are conducting daily scrums as part of their agile practices, inviting IT operations to participate can help with impediment removal. Inviting them to sprint planning can ensure those non-functional requirements are considered in the sprint, thus streamlining the value delivery process.
Cross-team communication tools such as Slack or Microsoft Teams really help here by making it possible for collaboration to be continuous. The “Alert/Notification” chat group or channel also needs to be properly managed, so that issues can be directed to the correct team and escalated fast using the correct action to take to solve the issue/bug.
Here are some collaboration tools that you may use and start collaborate within your organization;

5.Standardizing Environments
An environment is a collection of resources or targeted places that you want to convert from code into a real product via the pipeline. The environment may include Virtual Machines (VM), Database Servers, 3rd-Party services etc. Below is an example of environment stages with its usage, user/persona, and person-in-charge of maintaining the environment;

The advantages of having a well-defined environment include the following;
Deployment record/history — All pipeline run details are recorded in CI/CD tools for its resources.
Traceability — It allows one to track whether a code change (commit) or feature/bug-fix (work items) reached an environment.
Permission/Control — a secure environment by specifying which user is allowed and which target environment to deploy.

Automation environment provisioning is a major factor of success in a continuous delivery process. Can the Dev team request a new environment ad-hoc and are your environment being provisioned on-demand as application deploy? The application environment can divide into 3 main areas:

Infrastructure
Configuration
Dependencies

Infrastructure is the place where the application or service being deployed and the application will run specific configuration needs. It also includes how dependencies need to integrate with the application. As of today, infrastructure can be provisioned by script or they call it “Infrastructure as Code” or in short IaC. IaC is made more accessible today through a comprehensive range of tools available for automating the entire environment provisioning process.
Configuration is the next most consequential aspect of the application environment. Configuration dictates both how the application comports in a given infrastructure and how the infrastructure comports in cognation to the underlying application.
Dependencies are all the different modules or systems an application depends on, from libraries to services or other applications.
The benefit of using automate environment provision as follows;

Reduced complexity by allowing DevOps teams to work at a higher level of abstraction.
Increased stability by enabling applications to dynamically respond to their deployment.
Increased flexibility by decoupling configuration management from an application’s specific hosting environment.

We have many tools available in the market whether it’s open-source or enterprise that we can use to do automate provision for all 3 areas mentioned above such below;

6.Standardizing the DevOps Toolchain & Providing it as Self-Service
Once the DevOps adoption goals and processes are defined, we can determine the toolset required to meet the processes. Make sure the development and IT operation teams work together to decide on the tools suitable for the organization. With any new tool that is introduced, existing workers must be trained. It’s also essential to ensure that the tools meet security requirements and are well-integrated with existing resources and services.

Just name a few toolchains available in the market for the above sections.

7. Accelerating Release Management
Once we have properly defined our environment, the DevOps leaders need to create a proper release pipeline which when we need an auto-trigger for deployment, when need pre-deployment approval gate and when the QA/testing stage need be placed. The below picture has shown a basic release pipeline with combine auto and manual deployment;
Once you have a proper release pipeline, the automation of building, integration, testing, & delivery, and other processes, it will reduce the human activities in every release as well as the amount of management and coordination required.
As development acceleration has become a competitive advantage, the DevOps team has sought to enable Continuous Integration and Continuous distribution (CI/CD). CI/CD helps developers and IT operations overcome a huge hassle on the software development and testing process. Over the years, software development has migrated from the enterprise level, where there are broad resources, to smaller development teams that race to keep pace with demand generated from billions of smartphones and other mobile consumer devices and platforms. Below is an example of CI/CD pipeline with the combination of toolchain available;

In our case, we opt to utilize a combination of tools as it seems to provide the best solution for our complicated needs. Most teams developing enterprise products would benefit from such a ground-up approach. Our toolchain stack consists of,
Atlassian JIRA — a tool for team collaboration from the Product backlog, Sprint Planning and Report of release and how well the Agile team doing in each sprint.
Github — a Distributed Version Control System (DVCS) where the developer communicates with each other and collaborates to make product feature code better and visibility on changes and code versions. Any changes have to be reviewed by other developers or the Code Reviewer which made code more cleaner and less error/bug.
Azure DevOps — it’s a tool we use to orchestrate our CI/CD pipeline and it’s also the place where more collaboration between DevOps Engineer, Developer, Release Manager and QA team. It’s also the place where integrations happen in order to deliver a product with good quality which is why we have security analysis and QA testing before we deploy to the production environment.
Datadog — It’s a monitoring tool which with Datadog, you can monitor your servers, your clouds, your metrics, your apps, your team together. It’s like a one-stop for all kinds of monitors for your environment and products.

An efficient CI/CD pipeline can significantly improve the time to market and avail maintain stability and quality of the software are distributed.

8. Automating Testing
DevOps promotes automation and it’s goals to do as many automation as possible of all daily basis tasks that don’t require human interventions. Add QA experts into DevOps team composition will help the team decide the best approach or testing tools can be automated. Automation tools generally work when it comes to testing for application or system errors, but QA testing does a much better job of testing for usability and release readiness.
Integrating automated continuous testing to your CI/CD pipeline requires an application testing implement that’s easy to integrate with the build, test automation, and CI/CD implements you already use and that has extensive web API support. The benefit of using automated continuous testing as follows:
Stability. It helps you to apply quality and security requisites more consistently. If you record a manual security test and then automate it, it becomes a security requirement you can enforce on every build.
Speed. With automated continuous testing powered by scalable tools, developers can find and fine-tune issues in genuine time throughout the SDLC. Doing so expedites application development and avoids errors common to manual testing.
Scale. To scale manual testing, you require more manual testers. To scale automated testing, you only need more apps and builds to test.

Conclusion
DevOps Adoption is a journey that should start with an analysis of the organization’s architecture and goals. Solving these common challenges in DevOps Adoption will make transformation become much smoother. Over a period, every team or person in the organization will get used to the DevOps changes and adapt to the continuous learning processes. Once the Development, Operation, and Management teams learn how to cooperate, they will automatically be helping each other out and collaborating closely to archive DevOps Adoption goals.

8 Everyday Challenges in DevOps Adoption and How to Solve Them (Challenges)— PART 1

Najib Radzuan — Tue, 28 Apr 2020 04:33:53 +0000

DevOps is no longer a new word in the IT industry; it’s 2020 and nearly every organization either wants or is trying to evolve its culture and methods to these ways of working. However, adopting such revolutionary change is not easy and numerous challenges are faced by both the leaders and the ‘doers’; DevOps is very much a bottom-up AND top-down movement. Here are some common challenges I have observed in my fieldwork:

1. Dysfunctional Culture
Cultures that heavily command and control and bureaucratic can be very demotivating places to work. Where people are punished for raising problems or knowledge and information is hidden will stop people from sharing and collaborating. Where failure is seen to be something to be avoided, people will try and cover up mistakes rather than learn from them and they will be afraid of trying new things. Culture is very hard to define and seems very hard to change. It’s the values and behaviors of the people in an organization and needs to be addressed at this human level; it requires we talk about emotions and feelings, not just bits and bytes.

2. Resistance to Change
Not many people like to change their way of working; they have become used to their method of doing things and are frequently unwilling to adjust their usual routine or processes. Many people prefer working in silos and might not want to communicate with others and so DevOps leaders frequently face resistance in the transformation process. Some neuroscientists, like Britt Andreatta, believe that humans are fundamentally wired to resist change for evolutionary reasons.
Here’s an example of where a change agent or an organization may experience resistance: a tester that has traditionally performed manual testing may be asked to automate the test scenario or steps in a continuous integration pipeline. They might resist changing their style of working because they need to learn how to automate their previous manual test scenarios and feel it’s beyond their capability. Another example is a QA manager who has been used to having a large team of testers to oversee: they may see that their testers are being distributed across multiple teams as part of an organizational redesign to cross-functional squads, which they may feel threatens their status as a leader and manager.

3. Lack of Clarity of Vision
Many organizations want the improvements that DevOps promises but often insufficient time has been invested in properly planning on how DevOps will be adopted. Leaders may not have explored what DevOps may require in terms of organizational redesign or they may be resistant to this. Organizations frequently leave DevOps leaders and advocates to plan alone without leadership support to nurture and practice the true DevOps culture.
DevOps transformations without proper plans and strategy make it very difficult to accomplish its goals. Lack of vision makes it challenging for DevOps leaders to create a clear-cut plan when it comes to deciding on estimation, roadmaps, and deliverables. And it also makes it very hard to communicate and share the vision across the wider organization when it lacks clarity.
Sometimes DevOps change agents have a proper plan and strategy for DevOps adoption that they have proposed to our management or C-suite, but if these leaders are not openly supporting and evangelizing the execution plan for DevOps implementation it can stymie the DevOps adoption processes.

4. Teams Do Not Collaborate
The main goal in DevOps is to encourage and enable Dev, Ops and other teams to work together thus breaking down barriers between silos. Teams will have their own goals; the development team is focused on change and the IT operations team on stability. Ensuring teams share goals is a fundamental challenge for those leading DevOps adoption.
In addition, when an organization is geographically dispersed, cross-team collaboration can be even more challenging. Whilst DevOps and agile models encourage co-location, this frequently isn’t practical or even possible. Additionally, many organizations outsource work such as testing or IT operations, potentially to significantly different geographical regions exacerbating the challenge due to time-zone and language complications.

5. Environments are not Standardized
Dealing with multiple applications or service versions within the DevOps environment can result in slower production releases and increase the incidence of bugs and issues in our products. Not having standard environments, or production-like test environments frequently results in incidents as inconsistency leads to unpredictability. Having insufficient application environments can cause additional pain and delays in the release of new value to customers as the teams may be waiting for access to shared test environments for example.

6. Toolsets are in Contention
Development and IT operation teams frequently use different toolsets, but often they are trying to do the same thing, or manage the same piece of work. The challenge is to make sure the right tool is being implemented and that it is aligned with all teams’ goals and the company goals.
Tool selection is a contentious area as people have (often emotional) preferences. Once a tool is chosen, people have learned to use it and it contains significant amounts of data or customized workflows so it’s hard to change a tool — even when it’s identified that there’s a different one that now suits the teams’ and company goals better.

7. Release Management Causes Delays
Traditionally, organizations have released large batches of features using a project-driven approach. These releases are frequently coordinated via a centralized release team using a release calendar where teams book slots (often requiring work at weekends). These releases are often done manually, or with a bunch of scripts, only the release manager understands since they are the ones that built them. All of this means that releases happen infrequently and are at a high risk of having problems that need to be remediated. Teams are often also more focused on getting problems fixed than learning from why the problem happened and ensuring the learning is embedded back in the organization. Much like project teams are frequently disbanded after a go-live date and no post-implementation review inspects whether the effort resulted in the intended outcomes.

8. Manual Testing is Onerous and Time-Consuming
Testing manually is very time-consuming and is frequently performed by separate teams in separate phases creating delays in the flow of value to the customer. Additionally, it’s impossible to do true continuous integration without automating at minimum the unit test — ideally the integration and user acceptance tests will;l be automated too. Automating testing can appear to be a huge and overwhelming task, particularly when there are few resources capable of doing this work in an organization. People frequently perceive testing to be expensive when it requires a good deal of manual work and try to restrict investment — which leads to poor quality and downtime later down the line.

Conclusion
DevOps adoption is a journey that takes significant time. It requires organizations to embrace dynamic and continuous learning. Awareness of these eight key challenges at a very early stage will help anticipate issues and prioritize efforts. In part 2 of this blog series.I’ll explore how to overcome these challenges in Part 2.