Panel Discussion: Managing Testing Landscapes, Frameworks, and Tools for an Enterprise [Testμ 2023]

With digital transformation for enterprises comes an acceleration of applications, tools, and technologies. We have varied approaches for testing different COTS, cloud-native, and mobile applications. Enterprises are now looking at strategies to simplify and streamline their testing landscape. Our panel of industry leaders has been advising customers across industries and geographies to define their QE/Testing strategies.

In this session, they have shared strategies for harmonizing testing efforts, optimizing toolsets, and scaling frameworks. This session will help attendees learn how to manage testing in their enterprise environment better to reduce costs, improve efficiency, and increase quality. Don’t miss this opportunity to learn from industry leaders and improve your testing practices.

Adrian O’Leary, Head of Quality Engineering and Intelligent Automation at Apexon, brings a wealth of experience from Coforge, Comcast, and Cognizant. His passion for quality has led him to advocate for embedding quality in every development aspect.

Hemalatha, a visionary leader at LTIMindtree, oversees Automation, Performance Engineering and practice, and alliances. Having incubated 16+ Next Gen Engineering and QA services at Infosys, she’s a global expert in multi-functional testing strategies.

Mohan Bachchu, a leader in Accenture’s Cloud First group, champions application and infrastructure automation. With 24 years of experience, his solutions-driven approach addresses customer challenges using cutting-edge technology trends.

Raghuram Krovvidy, Chief Delivery Officer at Cigniti Technologies, brings over 25 years of expertise in Digital Engineering and Quality Assurance. His leadership has empowered Fortune 500 companies, and his captivating speaking engagements have left an impact.

Rajesh Kannan, Head of Quality Engineering and Testing Services at Deloitte India, is known for his strategic leadership and deep experience in Test Consulting and Digital Assurance.

If you couldn’t catch all the sessions live, don’t worry! You can access the recordings at your convenience by visiting the LambdaTest YouTube Channel.

Evolution of Enterprise Testing Tools and Frameworks

After introducing the panelists, Maneesh quickly moved on to the first question of the panel discussion. He asked this question particularly to Hemalatha-*** How are you seeing landscapes changing for enterprises with so many choices of frameworks and tools in the testing world?***

Answering this question, Hemalatha stressed the importance of carefully considering tool choices, given the number of options available. According to her, there’s no one-size-fits-all solution, and each framework has unique strengths and limitations.

She also highlighted the need to understand the potential impact of selecting specific frameworks. Rushing into tool adoption without considering their effects on testing environments can lead to unforeseen challenges.

In increasingly tight release cycles, she pointed out that not all frameworks are suitable for meeting aggressive deadlines. Some may introduce complexities that hinder timely project delivery, making it crucial to align tool choices with project timelines.

Furthermore, Hemalatha mentioned that customization and integration of various tools can introduce redundancy, negatively affecting regression testing and release schedules. According to her, this requires a holistic understanding of the long-term consequences of tool and framework choices.

Though effective testing and coverage are vital for ensuring software quality, Hemalatha mentioned that the complexity introduced by multiple tools and frameworks can compromise these efforts. To navigate this rapidly evolving landscape, she recommended proactive planning, considering how to maintain scripts and frameworks in the long run in alignment with the broader enterprise landscape.

In addition, she underscored the importance of continuous learning in an industry that’s constantly evolving. Staying updated on emerging tools and trends is essential for meeting industry demands and delivering high-quality testing services.

Unified Strategy for Enterprise Testing

Talking about tools, applications, and testing, Maneesh asked a crucial question to Adrian- How should an enterprise decide on a unified strategy for testing in this evolving landscape with various tools and applications.? What’s the first step that they should take?

Adrian began his answer by talking about certain points enterprises should consider. Firstly, he emphasized the importance of thoroughly assessing an organization’s specific testing needs and existing technology stack. This evaluation should include the potential for seamless integration of chosen tools into the CI/CD pipeline.

Adrian also discussed the need to evaluate the skill sets available within the organization. It’s essential to ensure that the team possesses the expertise to use the selected tools effectively.

Furthermore, Adrian pointed out that tools marketed as low-code or no-code solutions may carry hidden costs. While they claim to be accessible to non-technical users, creating reusable components often requires the support of experienced automation engineers, leading to additional expenses.

Additionally, scalability should be a vital consideration. Organizations should assess whether the chosen tools can accommodate their future growth and changing needs.

Lastly, Adrian emphasized the significance of a robust support mechanism for the selected tools. This could be a user community or customer service to address any issues arising during implementation.

Challenges Faced while Managing Testing Landscape

Moving on, Maneesh had an interesting question for Rajesh, which was more customer-centric- When you are talking to customers. What challenges are they facing in managing their testing landscape?

Rajesh highlighted the critical challenge of aligning testing strategies with the actual testing frameworks, tools, and landscape. This misalignment often stems from inheriting legacy systems or choosing tools for commercial reasons rather than their functionality.

Secondly, he pointed out the complexity within organizations, akin to joint families with various generations using different technologies and systems. Managing multiple tech stacks simultaneously presents challenges and requires a significant learning curve for testing teams.

Furthermore, rapid technological changes, such as introducing new testing tools like Selenium and Karate, demand continuous learning and adaptation. However, the pursuit of speed can sometimes overshadow the core purpose of testing.

Lastly, Rajesh discussed integration issues, where tools are not seamlessly integrated, resulting in siloed teams and the repurposing of tools across different parts of the organization. He emphasized the need for a forward-looking approach to address these challenges collectively as a community.

Developing a Tool Chain Strategy

Since Rajesh mentioned that speed can sometimes be a real culprit behind the immense pressure on the testing teams, Maneesh pointed out that various types of testing, like API testing, performance testing, functional testing, and integration testing, also add to their workload. He asked this question to Mohan- How should a senior leader in an enterprise, such as a CIO or CTO, approach the development of a comprehensive testing toolchain strategy?

While replying to this question, Mohan discussed a comprehensive approach for senior enterprise leaders to develop a testing toolchain strategy. He emphasized the need for tools to manage various test cases, execute tests, and generate reports. Additionally, he pointed out that automation tools play a crucial role, and they should integrate seamlessly with test management tools. He highlighted four supporting pillars: test data management, service virtualization, DevOps integration, and source code management.

Mohan highlighted the absence of automation architects in many organizations and the challenges that arise when multiple teams acquire their automation tools. Achieving 100% automation, especially for DevOps, is crucial. Therefore, he emphasized the importance of selecting an enterprise-wide tool or a combination of tools to address diverse automation needs across different application architectures.

Decision-Making in an Enterprise

Maneesh highlighted a crucial point while going ahead in the discussion that there are no centralized testing teams in the market today. These teams are embedded into different projects, and deciding such cases on what should be the architecture of testing becomes difficult. He redirected a question to Raghu and Adrian- How do you determine the decision-making process for individual teams? If the decision-making authority is left to individual teams, how can an enterprise effectively consolidate its decisions?

Raghu addressed the challenge of centralization and decentralization in testing teams within enterprises. He noted that both approaches exist, with some organizations centralizing while others decentralizing. The issue lies in how centralization and decentralization are implemented. Raghu emphasized that organizations often miss the point of considering solutions proposed by different parts of the organization in isolation. He stressed the importance of establishing a horizontal architecture practice within the testing organization, which understands technology holistically and integrates various tools.

Additionally, Raghu highlighted the significance of data and metrics in convincing stakeholders and driving informed decisions. A unified approach to measuring and managing SDLC metrics can help bridge tool debates and demonstrate value across the enterprise.

Adrian added to the discussion by highlighting that organizations are sacrificing quality for speed, resulting in production outages. CIOs and CTOs recognize the impact on quality and use metrics to address this issue. He mentioned a federated model where a federated TCO manages metrics, frameworks, tools, and testing practices, fostering consistency across the organization. Adrian noted that while organizations initially pushed DevOps and Agile from the top down, they now realize the need for value-added groups within a federated structure. These groups, including architects and metrics experts, can help restore quality and reduce production issues.

Introducing DevOps in an Enterprise

Maneesh mentioned that Adrian’s point regarding the top-down implementation of DevOps resonates with his hypothesis. When introducing DevOps in an enterprise, it can effectively unite various siloed projects and teams. By establishing a centralized collaboration platform through DevOps across the company it has the potential to bring testing tools and frameworks together. He asked the panel to express their views on the question: Do you observe this consolidation occurring in practice?

Hemalatha continued this discussion by expressing her views and insights first. She shared an example of a large client organization that initially decentralized its testing tools and frameworks due to leadership changes and reorganization. Different business units within the organization chose their tools and partners for DevOps and other testing practices. However, this approach led to budget inflation, complexity, and a lack of coordination. Eventually, the client realized the need to centralize their tools, environments, and skills, resulting in centralized training, knowledge management, and processes.

Hemalatha emphasized that even though organizations want to align with ESG (Environmental, Social, and Governance) goals, governance, maturity, culture, constant feedback, and executive sponsorship are crucial. She highlighted the importance of strong leadership and empowerment of teams to achieve high reusability and optimize resources, ultimately contributing to ESG goals.

Mohan shared a recent client experience in which they transitioned from a decentralized model to a federated model to balance autonomy and standardization. This allowed teams to make project-specific decisions while sharing common processes and tools like test case management and metrics. The transition not only improved consistency and reusability but also reduced costs. Mohan’s example illustrates how organizations can benefit from a federated model by combining autonomy with standardized practices and tools.

Managing Runtime Execution Environments in Enterprises

Maneesh focused on Mohan’s point regarding “just in time” and Hemalatha’s mention of ESG, which was supposed to be the third topic in the panel discussion on the testing landscape. Maneesh added that the customers he interacts with frequently complain about the cost implications of managing their runtime execution environments in the cloud. He asked Rajesh a question- How are customers currently effectively managing their runtime execution environment setups, or is there a trend towards standardization in this regard?

Rajesh mentioned that organizations are becoming more aware of their runtime execution environment’s efficiency due to DevOps and monitoring tools. They are now realizing the need for precise timing in environment management to avoid unnecessary spending.

Furthermore, Rajesh highlighted a significant shift in testing investments. While optimizing software testing landscapes has progressed, organizations are increasingly directing their resources toward system testing environments, including IoT, mobile, and 5G testing. This shift reflects a broader trend where testing extends beyond traditional software into real-world applications. Rajesh emphasized the importance of the testing community catching up with these evolving testing requirements, especially for systems with significant real-world impacts.

Adopting Best Practices in a Complex Testing Landscape

Taking the conversation forward, Maneesh asked Raghu a question on the best practices that should be adopted by the organizations- So, what do you see as best practices that enterprises are now embracing in this complicated landscape setups for testing?

Raghu discussed how many enterprises are still grappling with fragmented structures in their software development processes. While they recognize the need for new-age test organization structures, Raghu emphasized that most are still in the early stages of implementation. He also highlighted the increasing significance of risk-based testing and the difficulties in achieving mature impact analysis. Furthermore, Raghu mentioned that enterprises are continuing to prioritize automation in testing but are now exploring more advanced automation solutions. Lastly, he noted the ongoing relevance of exploratory testing in a rapidly evolving testing landscape. These trends reflect the changing dynamics of testing practices within enterprises.

AI in Enterprise Testing

The next question of the panel discussion was a question everyone is curious about today. Maneesh picked up that question on the audience’s behalf- Do you think AI in testing will solve some of these challenges that we talked about today, about landscape and tooling?

Adrian discussed the cautious approach required when implementing AI in testing. He mentioned that clients experimented with AI, such as using ChatGPT to analyze defects and optimize defect handling in sprints. One of his clients used ChatGPT to analyze use cases for consistency and to identify dependencies, aiding testing. However, Adrian stressed the importance of human oversight, as AI could sometimes generate incorrect test cases. He advised careful consideration when adopting AI tools for testing and highlighted opportunities in test data management and environment management for AI applications.

Hemalatha added her views to this conversation by emphasizing the need for a gradual, secure implementation process. She mentioned a case where a company unknowingly exposed its code through experimentation. Hemalatha highlighted the importance of guarding client secrets and the need for careful observation and testing before the widespread deployment of AI in testing scenarios.

Conclusion

As the session concluded, Maneesh requested each panelist to share their insights on What could be their advice to the audience as they think about setting up a landscape for an enterprise for testing?

Hemalatha shared her closing remarks by emphasizing the need for adaptability and flexibility in the testing industry. She stressed that testing jobs won’t disappear due to AI but will become more complex as AI needs testing. She encouraged professionals to adapt to changes, suggesting that rigid organizations risk losing their way in the evolving landscape.

Moving on, Rajesh highlighted the importance of having a clear purpose at the center of testing strategies. He urged teams to align their efforts with the organization’s business objectives, ensuring they are testing for the right reasons. Rajesh emphasized that having a clear purpose would guide selecting the right strategies, tools, and landscapes, leading to more effective and efficient testing.

Adrian recommended shifting the focus from defect detection to defect prevention. He stressed the importance of collaboration across organizational and geographical boundaries, using metrics and data to facilitate these conversations. Adrian explained that by preventing issues upfront and collaborating effectively, testing can become a smoother and more valuable process within organizations.

In his concluding remarks, Mohan highlighted the need for testing teams to work closely with engineering partners and demonstrate engineering excellence in testing. He emphasized the importance of automation in achieving this excellence, noting that learning and adapting to new technologies and techniques is essential in the evolving landscape.

Lastly, Raghu emphasized three key points: continuous collaboration and communication, implementing a holistic measurement framework and governance, and early involvement of all stakeholders. He stressed that continuous collaboration and communication should not be reserved for escalation points but should be an ongoing practice. Further, he said a holistic measurement framework and governance are crucial for visibility and convincing stakeholders. Also, involving stakeholders early helps avoid surprises and resistance to change.

The key takeaway from this session was that flexibility, collaboration, and continuous learning are essential to navigate and manage the diverse and complex testing landscape effectively. These principles help professionals and organizations adapt to evolving technologies and industry demands.

This panel discussion didn’t answer your questions? Feel free to drop them on the LambdaTest Community.