DEV Community: Seetharam Param

Building E2E DevOps Pipelines

Seetharam Param — Tue, 30 Nov 2021 00:49:20 +0000

This blog describes some of the best practices we have seen over the years in developing, building, testing and deploying software. Most engineering organizations have at least a basic understanding of the value proposition of a DevOps pipeline. However, engineering leadership may focus too heavily on simplicity, opting to go for the “one stop shop” approach, instead of picking and choosing the best of different tools. I emphasize that engineering teams should be free to pick and choose the best tools for a given use case, rather than going all-in on a one-stop-shop solution.

Release velocity is one of the most crucial metrics to focus on for a high-performing DevOps organization. Companies that can iterate quickly, releasing more features and new products, are more competitive and more successful in their respective markets. Engineering leaders who want to improve their team's release velocity and overall release quality should pay careful attention to the tools and services being utilized. Empowered DevOps teams choose the tools best suited to support delivering critical workloads. Tools like Jenkins may already be in place to provide a solid CI/CD foundation. Augmenting Jenkins with additional capabilities can still provide a sizable ROI.

Teams that are looking for a place to start should follow the path of the software development lifecycle: going "left to right".

Planning and Design

At the far left of the SDLC is the planning and design phase. While teams may overlook this in the context of the "pipeline"(think CI/CD), planning and designing is the foundational piece of a finished product or feature.

DevOps is the marriage of development and operations teams, but the planning and design phase is the inflection point where product and engineering teams meet. Feature requests and product requirements transform into living, breathing code. There are still powerful tools and services that automation teams can employ in this phase to improve the overall pipeline. Tools like Trello and Jira help manage ticketing and work items, offering Kanban boards and other Agile tooling. For architecture and system design, services like Draw.io and LucidChart provide a capable set of design tools. Key is the current emphasis on cloud-native and cloud-first architecture.

Improved planning and design will lead to better requirements, which are the primary inputs into the next phase of the pipeline.

Committing Code and Testing

When development teams start writing code is where the "rubber meets the road". The requirements and feature requests that were developed during the planning and design phase start to take rough shape. The overall software quality, and critically, security, are heavily influenced by the quality of work in this phase. Untested, inefficient, and insecure code will lead to a "garbage in, garbage out" scenario: production environments will be more susceptible to outages and compromise, regardless of operational tooling and monitoring.

At a minimum, development teams should use some form of version control system(VCS). Version control provides a centralized repository that tracks changes and authorship for code. A VCS like Git or Subversion enables teams of developers to contribute to the same codebase in parallel, contributing changes without impacting or overwriting prior or ongoing work. A version control enabled codebase is so critical to modern DevOps workflows that it seems almost redundant to mention. There are still organizations that do not make use of it. It's not surprising that "Codebase '' is the first principle listed in the Twelve-Factor App philosophy.

Before committing code to a VCS like Git, tooling team should provide developers with the tools to help enforce and guide coding standards. A VCS is simply a repository of code, good or bad. Simple workflows are enabled at the individual developer level that will improve the overall quality of organizational application code. Most integrated development environments(IDE) like VSCode and PyCharm include linters. These linters are specialized tools that highlight basic logic and syntax errors in the code, and in some cases can suggest and correct fixes. Pre-commit hooks can also be utilized. These simple scripts and automation perform further linting and testing around code and code quality prior to submission for review.

In the course of software development, more comprehensive, in-depth tooling is typically required to fully test code for functionality, potential bugs, and susceptibility to security issues and compromise. Static analysis tools can analyze and evaluate code without requiring it to be running as a live application. Tools like SonarQube can be integrated into the developer IDE, or deployed as part of a CI/CD pipeline with tools like Jenkins.

Code written in languages like Java or C++ are compiled into a binary before being integrated and deployed into live environments. In legacy environments, software was often compiled on individual developer machines before being uploaded. In larger, modern environments that model no longer scales. A centralized build system provides homogenous configuration, and ensures build artifacts adhere to standards before being pushed into a CI/CD pipeline. Build tools like Maven, Gradle, and Bower are popular choices, and integrate well with most CI/CD infrastructure.

Integration and Deployment

Once a feature or application is finished, the completed build artifact is then integrated and deployed to the live environment. Once it reaches production, the "value" is realized. Value realization is a result of customer interaction with the new feature or service. Consequently it is critically important to make sure that production workloads are tested, deployed, and monitored with the right tooling.

The core piece of infrastructure for integration and deployment is the Continuous Integration/Continuous Delivery(CI/CD) pipeline. Replacing legacy software deployment methods like FTP, CI/CD pipelines provide a holistic automation platform, encompassing build/compilation, testing, integration, and deployment in a single interface. CI/CD pipelines form the backbone of almost any environment that adheres to DevOps principles. There is a broad selection of CI/CD software available: SaaS tools like TravisCI, CircleCI, and AWS CodeDeploy, as well as self-hosted solutions like Jenkins and Spinnaker. Container solutions like Docker and Kubernetes can provide immutable build artifacts, further enhancing the functionality of CI/CD architecture.

CI/CD pipeline capabilities extend beyond software deployment; the underlying infrastructure can be defined and deployed using code as well. Tools like Ansible, Chef, and Puppet enable DevOps engineers to define the configuration of applications and services in code, automatically applying them during deployments, and maintaining minimal configuration drift. For infrastructure, tools like Terraform, Cloudformation, and recently Pulumi can be employed to define and control the provisioning of resources like compute nodes, databases, and even entire networking zones. Teams that integrate configuration management and infrastructure-as-code tools in their CI/CD workflows have end-to-end deployment and release automation systems, which allow for faster iteration and feature delivery.

Once production workloads are live, choosing robust operational tooling is key to ensure that the customer experience remains positive, and that any issue or performance degradation provides immediate, actionable feedback. The modern ecosystem of highly available, highly performant customer-facing applications has given rise to a landscape of cloud and web focused monitoring and operational services. Tools like DataDog, AppDynamics and New Relic can provide a granular look into the health of application infrastructure. Llog aggregation and searching platforms like Elasticsearch enables critical application data to be found from the vast sea of information generated by modern applications.
Utilize Diverse Tooling to Build a Robust DevOps Pipeline
The modern DevOps landscape offers engineering teams a broad selection of tools at every stage of the SDLC. Rather than going all-in on one vendor or toolchain, teams should be empowered to pick and choose the best functioning and best fitting tool or service for their use case.

Each step in the SDLC is important, even before the first line of code is written, thus the importance placed upon picking the best tools at each stage. Once teams have settled on their pipeline
tooling, the next key focus should be a unified way to manage and monitor the complexity of a diverse toolchain.

Learn more about Enterprise DevOps platforms with this white paper.

Commit-Based Visibility

Seetharam Param — Wed, 06 Oct 2021 04:26:40 +0000

The complexity of modern DevOps environments requires tooling that enables engineering teams to have maximum visibility into the health and behavior of their workloads. Having an acute understanding of the "moving parts" of an application means issues are resolved faster. However, it's often the case that outages or unexpected behavior are a result of internal change, like code commits. Engineering organizations struggle to understand and account for the sum total of change in their environments.

Utilizing commit-based visibility means that each unit of change (commit) can be identified and tracked across their entire lifecycle, as well as multiple environments. Being able to pinpoint individual commits across an entire engineering organization has the potential to provide meaningful gains in critical metrics like Mean Time To Resolution(MTTR). Organizations that can drive down MTTR will improve the ever crucial customer experience.

Implementing commit-based visibility means first understanding the underlying instrument of change: the commit. Afterwards, understanding the lifecycle of a commit and how it's deployed will reveal an opportunity to improve visibility even further.

What is a Commit?

A commit in this context refers to a specific process in which a change in the source code is sent to a Version Control System(VCS) repository, where it is hashed and stored as part of a continuous log of changes. In the course of a single day, development teams may push thousands of individual commits to multiple repositories.

Any one of these commits, at a basic level, represent a net change to the destination environment. The code could be part of application source code, or perhaps represent configuration and orchestration code for infrastructure. The key point is that any change has the potential to bring about unexpected behavior, including an outage or interruption in service. In the past, operations teams were often forced to spend critical minutes tracking down the at-fault change through a variety of version control systems and deployment automation.

A single commit contains only a limited amount of metadata about itself, which has limited usefulness in understanding whether it could be implicated in issues with a wider scope. While most commits make it possible to identify the original author, they provide very little actual information around the context of the change, and under what circumstances it was requested. Sparse commit messages can result in an even greater mystery around the actual intent of a given commit.

Implementing a system with Continuous Integration/Continuous Delivery(CI/CD) provides a wealth of features to test and integrate code before it is deployed to critical production workloads, helping teams avoid issues resulting from untested code changes. Successful implementation of a DevOps initiative typically depends on a well-functioning CI/CD infrastructure.

Commits in CI/CD

CI/CD systems are a critical pillar in DevOps. They allow individual commits to be tested and deployed in an appropriate level of isolation. CI/CD pipelines can be modeled to fit a variety of development and deployment scenarios, and provide fast feedback on interoperability and overall code quality. Earlier stages of a pipeline typically focus on smaller scale unit tests, with some checks occurring on the local development machine before code is pushed into the pipeline.

The example pipeline is from Jenkins, a widely used CI/CD platform. In this instance, several stages and pipeline components have been configured, weaving together different testing and isolated deployment steps into a cohesive suite of release automation. Individual commits make their way through each appropriate stage, providing commit-based visibility throughout the pipeline. Developers and other engineering teams can receive immediate, visual feedback on the health and state of a commit at any part of the pipeline.

Unfortunately, a single pipeline or CI/CD instance may only represent an individual team or workgroup, or may only be a specific application or production service. Even within a single application stack, infrastructure and application code may be deployed through separate CI/CD pipelines. In a distributed system, outages may not present an immediately obvious cause when a variety of changes may be flowing into the environment simultaneously. Multiple engineering teams may maintain their own decentralized CI/CD infrastructure, making holistic visibility across all pipelines and platforms difficult to achieve.

Commit-based Visibility Everywhere

Fully-functional CI/CD infrastructure brings massive gains to previous logistical paradigms in software development and deployment. However, modern distributed systems typically require multiple CI/CD instances to fully cover every corner of the stack. In those types of environments, ReleaseIQ tooling can tie disparate CI/CD infrastructure into a single, unified "pane of glass" view.

When a breaking change is introduced to an environment, MTTR is crucial: it's very likely that customer experience is at stake. Long interruptions to critical services may leave customers dissatisfied and heading for the exit. With a single source of truth for identifying and remediating breaking changes, operations and development teams can work in close concert to quickly resolve issues resulting from malformed changes.

One of the main objectives of DevOps is to remove the logical barriers that have traditionally existed between development and operations teams. In legacy environments, development teams often kicked their completed code "over the wall", leaving operations teams scrambling to integrate and test new features, often incurring long hours debugging unexpected behavior and performance issues. CI/CD helped bring down that wall to some degree, but offering commit-based visibility across the entire pipeline environment provides a new level of shared ownership between operations and development.

Being able to identify broken changes across engineering organizations can also provide longer-term statistical data for identifying trends. If certain teams or application stacks are more prone to introducing negative change into the environment, additional engineering resources can be focused to help improve testing and reliability. Without this level of holistic visibility, it will be much harder for leadership to know where to focus a limited pool of resources. In the case of commit-based visibility, problem areas surface themselves in an easy to understand system of visualization and metrics.

Commit-based Visibility Provides Value

With commit-based visibility, stakeholders and engineering teams alike have better visibility into completed, ongoing, and planned changes. When a change-induced outage occurs, identifying and rolling back the "at-fault" change ASAP is key for customer experience. As with any new initiative, getting buy-in across technical and non-technical stakeholders is critical: engineer teams should be able to demonstrate clear business value in justifying additional resource commitments. With commit-base visibility, there is a clear line of value drawn from reduced outage minutes to customer experience to greater realized value.

Over time, improved commit-based visibility will improve overall code quality and reduce operational firefighting, resulting in happier and more productive engineering teams. Another benefit is increased release velocity. Organizations that stay successful and profitable long term focus on delivering more features to customers faster, safer, and more stable.

Want to learn more about our solution ReleaseIQ Essentials for Jenkins? All you need is a more than one Jenkin’s instance and you can sign up here.