DEV Community: Lucas@QAComet

What is bug surface area?

Lucas@QAComet — Thu, 08 Aug 2024 14:00:00 +0000

One of the tricky problems while building software is finding the constant balance between releasing new features and dealing with technical debt. One common form of technical debt is bug surface area. Simply put, it's the phenomena where as you develop new features the number of bugs increases at a much faster rate.

Why should you care? A larger bug surface area can lead to increased maintenance costs, reduced user satisfaction, and longer development cycles. These problems happen because in any non-trivial project features interact with each other, which creates more opportunities for bugs to creep in. Moreover, if new features have to work around a bug (call it Bug A) introduced in another part of the codebase, when you fix Bug A, you may have to go back and fix these workarounds.

It is essential to mitigate around the problems caused by bug surface area. Some techniques like implementing a modular design, avoiding rewriting core functionality between modules, thoroughly testing features with automated tests, and investing time in paying down this type of technical debt will help your project succeed. Remember, sometimes less is more – a lean, well-executed feature set often outperforms a bloated bug-filled one.

Why Net Promotor Score (NPS) is ubiquitous in tech

Lucas@QAComet — Sun, 04 Aug 2024 16:28:02 +0000

Net Promoter Score (NPS) is a ubiquitous metric for gauging customer satisfaction and product quality. But why has this simple survey question become so prevalent? And how can it effectively measure customer perception of product quality?

NPS gained popularity because of its simplicity and accessible insights. You start by surveying customers by asking, "How likely are you to recommend our product to a friend or colleague?" and letting them select a value from 0-10. When calculating NPS, responses are categorized into three categories: Promoters (score 9-10), Passives (7-8), and Detractors (0-6). The final NPS is calculated by subtracting the percentage of Detractors from Promoters.

NPS = %Promoters - %Detractors

This straightforward calculation gives an easy benchmark for customer sentiment about your product.

But how does NPS measure perception of product quality? If a customer is in the Promoters category, they are likely bragging to their friends about how awesome your product is. If your survey yields a high NPS, many of your users love your product and think it's valuable enough to stake their reputation on it. This indirect measure often correlates strongly with actual product quality.

However, NPS isn't without limitations. It doesn't provide specific feedback on what aspects of the product drive the score. That's why many startups augment NPS with follow-up questions interviews. For example, your Detractors likely have useful feedback for how you can improve your product. This information can help you steer your development team to new features and improvements. Ignoring negative feedback can be perilous to your startup's success.

Curious how you can improve your product without frustrating customers? Check out QAComet, we offer fractional QA services for startups with plans starting at $999/month.

CrowdStrike is overhauling their QA process

Lucas@QAComet — Sun, 28 Jul 2024 19:56:24 +0000

Because of the recent outage from CrowdStrike's software update, affecting over 8.5 million users, they've had to overhaul their QA process into one with many layers of quality checks. CrowdStrike has released a preliminary report detailing their plans to revamp their quality assurance processes. The incident, which resulted from an erroneous update passing through their automated validation system, caused widespread BSODs. In fact, it's estimated this deployment failure cost fortune 500 companies over $5.4 billion dollars.

The root cause was traced back to a change in a configuration file that led to an out-of-bounds memory read. CrowdStrike admitted that their testing process for configuration file updates was relatively lax, relying solely on a single automated content validation system, which seemed more akin to a JSONSchema validator than a comprehensive testing suite.

How CrowdStrike will implement their QA

To prevent such incidents in the future, CrowdStrike is implementing several new QA processes:

Local developer testing: Surprisingly, their previous development process didn't include a final manual check on a developer machine before releasing updates. They will now incorporate human oversight in the process.
Content update and rollback testing: CrowdStrike plans to rigorously test their content updates and rollback procedures, simulating various fault scenarios to ensure their systems can handle broken updates effectively.
Stress testing: The company will expand its stress testing to include configuration file updates, not just new components of their Sensor system. This comes after realizing that updates may have overwhelmed parts of the computers being updated, causing fail-safes to break.
Fuzzing: CrowdStrike will enhance their fuzzing efforts, particularly for the "Interpreter" system that reads configuration files, which may have internal logic.
Fault Injection: The company will adopt this common testing technique, deliberately introducing errors into different parts of their system to evaluate how other components respond to failure.
Stability testing: CrowdStrike will implement stability testing, gradually removing portions of the functioning system to study how different components break – a crucial technique for mission-critical systems.
Content interface testing: Finally, they will bolster testing for their underlying content interface, focusing on making their interpreter more resilient to unforeseen issues.

By implementing these comprehensive QA processes, CrowdStrike aims to significantly reduce the risk of future outages and ensure a more robust and reliable service for users all around the world.

DEV Community: Lucas@QAComet

What is bug surface area?

Why Net Promotor Score (NPS) is ubiquitous in tech

CrowdStrike is overhauling their QA process

How CrowdStrike will implement their QA

Links