Evolution of Testing in the Age of DevOps [Testμ 2023]

Software engineering teams have always embraced DevOps to ship intelligently, faster, and, most importantly, daily. However, the question remains: How did they ensure they “ship confidently”? Continuous testing offers both solutions and challenges as testing transforms in the era of DevOps.

Our esteemed panel of industry leaders came together to delve into this evolution and shared their valuable insights into how they helped customers transform their testing strategies within the landscape of DevOps.

About the Speakers

Before starting the session, Maneesh Sharma, Chief Operating Officer at LambdatTest, briefly introduced all the panelists.

Vikul Gupta is heading the NextGen CoE for Qualitest. He brings over two decades of experience from top-tier companies. His expertise spans AI/ML, DevOps, Agile, and more, making him a dynamic leader in the world of quality engineering.

Mallika Fernandes is an IT leader with a remarkable 24 years of experience. She is a driving force behind Quality Engineering innovation at Accenture’s Cloud First group. Her passion for AI/ML and Cloud Quality transformation is truly inspiring.

Harleen Bedi is a senior IT consultant specializing in AI, Cloud, Big Data, and other emerging technologies. She’s dedicated to developing and implementing quality engineering strategies that align with business goals.

Bhushan Baggi has twenty years of business and technology leadership and leads the Quality Engineering business at Wipro. He’s a transformation consultant at heart and has played various pivotal roles in Development, Assurance, and Business Transformation.

Asmita Parab is the Head of Testing at UST Product Engineering with nearly two decades of experience in software testing. A passionate advocate for delivering top-notch software products, her leadership has been instrumental in driving excellence in testing practices.

If you couldn’t catch all the sessions live, don’t worry! You can access the recordings at your convenience by visiting the LambdaTest YouTube Channel.

The Evolution of Testing

Kicking off the panel discussion, Maneesh asked Harleen: How have you seen DevOps evolve in the context of testing over the years?

Answering the question, Harleen kept forward her views and discussed the evolution of testing in the age of agile and DevOps. She talked about the three major changes it brought to the testing landscape.

Firstly, there is a stronger emphasis on collaboration among development, testing, and operations teams. Silos that once separated these teams have been dismantled, leading to closer and more effective cooperation.

Secondly, she highlighted that testers have had to expand their skill sets. Previously, they were primarily responsible for understanding business knowledge and using specific testing tools. Today, testers are integrated into development teams, requiring a broader understanding that includes coding.

Lastly, she talked about how automation has become a pivotal aspect of testing. It has evolved beyond merely automating regression testing. Now, it includes the entire development pipeline, ensuring faster and more efficient software delivery.

Agile and DevOps

Moving on, Maneesh asked the panelist about a common confusion in the industry: Is DevOps the same as Agile or vice versa?

Bhushan emphasized the relationship between Agile and DevOps, comparing Agile to a methodology that brings speed and DevOps to a toolchain. He used the analogy of an F1 race, where Agile testing teams act as pit stops, and DevOps serves as the racetrack. The quality and efficiency of the pit stop team (Agile testing) can significantly impact the overall race (software delivery). He highlighted that methodology and machinery (DevOps) must work harmoniously to achieve the necessary speed in Agile development.

Asmita, agreeing with Bhushan’s views, added to the discussion by stating that Agile and DevOps share the primary goal of delivering products to customers as quickly as possible, in line with the Agile Manifesto’s principles. She viewed them as complementary forces that accelerate the entire software delivery lifecycle. As testing teams adapt and integrate into both Agile and DevOps ecosystems, they play a crucial role in enhancing and enabling software delivery acceleration.

Quality Engineering and Modernizing Applications

Since DevOps and Agile methodologies are the main topics for this discussion, Maneesh asked the panel to share their views on Considering that DevOps and Agile are methodologies, how significant is the role of tools in integrating these approaches effectively?

Vikul talked about the four main pillars of DevOps, which were People, Process, Technology, and Culture. He emphasized the importance of tools in technology and mentioned different functional areas like requirement management, continuous integration, and testing. He noted that tools should integrate well with other ecosystem tools, enabling automation, quality gates, and end-to-end traceability in DevOps.

Mallika defined DevOps as a practice that brought together development and operations teams. She highlighted the importance of feedback between these teams, especially in improving the testing process. Mallika recommended selecting the right tools that facilitated collaboration among these teams. She also mentioned the significance of monitoring and observability to provide instant feedback and minimize downtime.

Asmita emphasized that customers had embraced continuous integration, testing, and deployment in DevOps. She mentioned the shared responsibility for quality across teams, including development and testing. Asmita stressed the importance of efficiently automating testing processes and using technology like containerization and orchestration for testing. She also mentioned the role of observability in gaining insights for testing.

Vikul again discussed how DevOps automation had transformed testing, making it an integral part of the process. He noted that testing was no longer an afterthought but had become a first-class citizen in DevOps. Vikul also talked about the shift from quality assurance to quality engineering, where quality started by defining requirements and preventing defects rather than just finding defects after coding.

Clogged PipeLines in Testing and QA

**Talking about DevOps and QA, customers today sometimes talk about a term called Clogged Pipelines, but what does that mean? **Maneesh’s next question for the panelists revealed this answer.

Taking on this question, Harleen explained that a clogged testing pipeline is like a blockage in your water supply, but in software development, issues are preventing your code from progressing to the next stage. She mentioned reasons for this, such as late feedback, inadequate testing, too much manual testing, and lack of automation. These issues can lead to delays in the development process.

Vikul discussed Continuous Integration (CI), Continuous Deployment (CD), and automated testing. He mentioned that the purpose of these processes is to provide fast feedback so that problems can be fixed quickly. He explained that pipelines can be seen differently depending on your perspective. From a developer’s point of view, a clogged pipeline occurs when quality checks cause delays. However, from a DevOps perspective, it’s effective in identifying issues early. He emphasized the importance of preventing “garbage” (flawed code) from entering the pipeline through practices like requirement validation and proper testing.

**Mallika **added to the discussion by discussing the fixed time available for moving code from commit to deployment in production. She explained that a clogged testing pipeline takes up valuable time. She suggested using technology-driven strategies like distributed execution and parallel execution to speed testing. Mallika also highlighted the importance of having the right test environments to execute tests effectively. These improvements can help clear the blockages in the pipeline and ensure a smoother DevOps process.

Changes in the Role of a Tester

As the next point of discussion, Maneesh raised a crucial question: What’s the change in role for a tester in the DevOps world?

Answering the question, Vikul shared his experience with a customer in Chicago who questioned the need for testers and suggested that developers could handle all testing. Vikul explained that the role of testers has evolved significantly in recent years. He mentioned that manual testing is no longer a prominent concept, and testers now focus on quality engineering. Quality engineering involves considering quality from the beginning, not just writing test cases. Testers collaborate closely with development teams, and their role is to ensure the software works as expected.

Vikul introduced the term SDET, a software engineer with a testing mindset. He emphasized that testers have a unique perspective of negative test cases and how to break things, which differs from developers’ focus on design and coding. Testers often have deep domain knowledge and are considered experts because they understand the system thoroughly.

Vikul also mentioned the use of advanced technologies like generative AI in testing. Testers use AI to automate various aspects of the testing process, from requirements to test scenarios and scripts. He highlighted the importance of test optimization and intelligent testing, where AI helps identify the most relevant test cases to run, especially when there’s limited time for testing. Additionally, AI can help track defect yield and test case effectiveness to improve testing efficiency.

Developing Prototypes for Clients

Since the discussion revolved around DevOps and its relationship with automation, Maneesh pointed out that there is an increase in tools for code generation that extend to generating test cases. The scope of code coverage is expanding. What observations have you made in this domain while developing prototypes or production-ready code for your clients?

Harleen replied by mentioning that despite the hype around AI and generative models in software development, these technologies are not fully ready for enterprise use. She explained that many models are trained on open domains and may not provide the expected results in specific enterprise contexts. Harleen emphasized that significant effort and training are required to make these models effective, and quick ROI should not be expected.

Starting the DevOps Journey

Maneesh discussed the importance of AI in accelerating processes like testing. He mentioned the need to identify relevant test cases based on code changes to save time and improve efficiency. Maneesh then shifted the discussion back to DevOps and asked How should organizations and practitioners begin their DevOps journey, structuring the process, considering people and technology, and setting clear goals?

Asmita answered this question by shifting the focus to onboarding and the importance of understanding goals and the desired path when starting a DevOps journey. From a testing perspective, she stressed the significance of “shift-left testing,” which involves activities like static code analysis and automated unit tests before code is committed. She also discussed the importance of choosing the right tools and optimizing test automation for quicker execution and feedback. Additionally, Asmita mentioned the need to integrate performance engineering and monitoring into the testing process and highlighted the importance of observability and experimentation in the shift-right approach.

Future of Manual Testing

Since everyone in the panel was talking about automation in testing and its impact on software delivery, Maneesh asked the panel: Does that mean that manual testing is no longer required?

Bhushan started by addressing whether manual testing is still needed in the age of DevOps and automation. He suggested that while the traditional concept of manual testing may not be required going forward, there is a shift in how automation is applied. Automation is no longer limited to just test execution; it now includes activities like test design and even automating requirements gathering. Bhushan emphasized the importance of how automation is applied and optimized before it is implemented.

Vikul provided a different perspective. He acknowledged that not everything can be automated, and manual testing still plays a role. However, he agreed that the goal should be to minimize manual testing. Vikul highlighted the importance of estimating testing timelines considering the manual testing component. He mentioned that even in industries like banking and insurance, with legacy systems, some manual testing is still needed. Vikul also emphasized reducing manual testing efforts using tools and techniques to optimize test cases. Overall, he believed that while manual testing isn’t obsolete, it should be kept to a minimum.

Automation for Accessibility Testing

As the next topic of discussion for the panel, Maneesh talked about accessibility testing and asked the panel: Although there are automated tools available for conducting accessibility testing, does this imply that individuals with specific abilities and disabilities are no longer required to perform such testing? Can automation entirely replace the need for human testers in this context?

Harleen mentioned that while automation is valuable for functional testing (checking if features work), there are certain aspects like accessibility and usability testing where complete automation might not be feasible. These non-functional aspects are essential, and they should not be overlooked.

Bhushan added that manual testing doesn’t necessarily have to be done exclusively by dedicated testers. He suggested that individuals with various skills, domain expertise, can contribute to manual testing. He also highlighted the importance of combining testing with automation skills for more effective testing.

Mallika believes that almost anything involving code can be automated to a significant extent. She emphasized that automation can cover various aspects, including infrastructure, performance, resilience, and not just functional testing. However, she noted that challenges may arise when code availability or exposure is limited.

Vikul agreed with the points made and cited the example of accessibility and real user actions, which can be challenging to automate entirely. He emphasized that the general consensus is that anything that can be automated should be automated. Further, Vikul talked about the reality of DevOps pipelines, stating that fully automated, touchless pipelines are not common in most organizations.

He mentioned that while companies like Amazon and Spotify have such pipelines, they are exceptions. Most organizations are satisfied with continuous delivery, which involves automated building and promotion of code to pre-production. However, there are still manual triggers for promotion. He emphasized that the level of automation in a DevOps pipeline is closely tied to the application’s architecture. Microservices architecture and a testing pyramid approach, which includes unit and integration testing early in the pipeline, can be successful strategies.

Challenges in Testing

Maneesh asked the panel about: What are the certain challenges that customers whom you work with have faced when integrating the testing practices with DevOps?

Harleen highlighted several key challenges that testing teams encounter within DevOps and agile development methodologies.

One significant issue she addressed is the prevailing belief among some teams and stakeholders that testing is a bottleneck, impeding the swift progression of code to production. In these fast-paced environments, where the aim is to rapidly release new features and updates, the quality assurance processes can sometimes be viewed as hindrances. This perception arises from the fundamental differences in objectives: while developers strive for speed and functionality, testers prioritize quality and identifying potential issues. This discord in priorities can lead to a belief that testing is slowing the development pipeline.

Another major challenge Harleen discussed is the cultural shift necessary for successful DevOps implementation. While organizations can establish strategies, methodologies, and guidelines, the ultimate success of DevOps hinges on fostering a collaborative culture. Team members, including testers, must align their mindsets and embrace the cultural shift toward collaboration, continuous improvement, and shared responsibility. Without this cultural transformation, even the most well-defined DevOps practices may not deliver the expected benefits.

Furthermore, Harleen emphasized the importance of authority and influence within the testing teams. Testers must often challenge decisions or processes that might compromise quality or introduce risks. However, in some instances, testers may not possess the necessary authority or face difficulties asserting their concerns. This can create a situation where testers struggle to uphold testing standards and practices that align with the principles of DevOps.

Best Practices to Embrace DevOps and Testing

As the session concluded, Maneesh requested each panelist share their insights and suggestions for the audience on managing this merge or sync between DevOps and testing.

Firstly, Mallika emphasized adaptability, quoting Charles Darwin, and stressed the importance of building a DevOps tooling chain adaptable to an organization’s unique challenges and technologies.

Harleen highlighted the significance of investing in automation, continuous feedback, and collaboration. She stressed the need for continuous process improvement and greater collaboration across teams.

Bhushan discussed the cultural aspect of quality, emphasizing that quality is everyone’s responsibility and collaboration is key. He advocated for the democratization of quality.

Moving on, **Asmita **emphasized the importance of continuous learning in the testing and quality field. She encouraged staying updated with industry trends and technology to enhance the quality structure.

Lastly, Vikul urged those in the quality world to take pride in their roles as custodians of software quality. He highlighted the critical role of quality in ensuring customer satisfaction and emphasized the importance of digitally delivering high-quality products.

Maneesh concluded the session by emphasizing the significance of DevOps and its enduring presence. He highlighted that culture and collaboration are key aspects of DevOps, as they are crucial in achieving the shared objective of delivering software to customers efficiently. He also mentioned the importance of refining processes to create a smooth supply chain for faster software delivery.

Got more questions? Please put it on the LambdaTest Community.