Strategies for Choosing The Right Mobile Devices for Testing

Did you know how much time people spend on their mobile phones today? Well, as per a survey conducted in February 2021, nearly half of the respondents mentioned that they spend 5 to 6 hours on their phones daily, excluding work-related use. Further, 22 percent of the respondents said spending 3 to 4 hours on average on their phones. Of this, consumers spend the majority of their time on mobile apps. Hence, as mobile apps are a fundamental part of our lives and play a key role in how we work, communicate, and entertain ourselves, being downloaded in millions each day across a myriad of devices, assuring the quality of these apps and experience delivered is a top priority. As a result, there’s been an increased focus on mobile device testing, and testing for apps across mobile devices is quintessential now.

What is mobile app testing?

Mobile app testing refers to the process by which apps for the devices are tested for their functionality, usability, performance, and much more.

Mobile application testing can be manual and automated, which helps ensure that the app caters to all the business requirements and user expectations.

Challenges of mobile app testing

Testing apps on mobile devices can often be complex. Following are some of the reasons for this:

• A wide range of mobile devices — Devices now have different screen sizes and hardware configurations such as a hard keypad, virtual keypad, trackball, etc.
• Different types of models of mobile devices like Samsung, Apple, Google, HTC, Nokia, OnePlus and much more.
• Various operating systems, including Android, iOS, and many more
• Different versions of OS like iOS 5, iOS 15.5, Android 6 etc.
• Multiple network types including both WiFi & SIM(5G,4G,2G etc.) Regular and frequent updates, where with each update, a new testing cycle is required to ensure that no application is impacted.

Testing on real devices vs. virtual devices

Whether to use a real or virtual device for testing mobile apps mostly depends on what you wish to test.

Real devices

If testing your app’s performance is a priority, real devices are always the most suitable option. Real devices help provide accurate results and allow testing in the same condition as end-users.

Benefits of testing on real devices

• Test applications across a variety of real user conditions such as network conditions, screen size, battery consumption and much more
• Assists in recreating the customer user journeys
• 3rd party transactions can be effectively monitored
• Ability to assess geolocation, push notification and security features(2FA) on an app
• Faster test execution as compared to simulators/emulators

Several enterprises garner the benefit of real device cloud to enhance the efficiency of the testing processes and deliver the best quality apps. It offers a mobile testing environment that has a diverse collection of iOS and Android devices.

Virtual devices

Besides the real devices, QA teams utilize virtual devices for mobile testing, including emulators and simulators.

Emulators are often preferred while testing the external behavior of the mobile device, such as calculations, transactions, etc. On the other hand, simulators are less reliable than emulators and inefficient for debugging in most cases.

Device fragmentation and Android versions

As Android is supported by a large variety of mobile device manufacturers, the fragmentation or diversity of versions in use at any given time is more than in iOS.

How does it affect the users?

• A large section of customers judges a company’s credibility based on the range of devices and OSs supported
• More than half of the users will uninstall or not revisit an app due to issues on mobile and visit a competitor
• The overall experience of the end-users

Strategies for selecting the right mobile device for testing

As the dependence on mobile devices and apps has increased exponentially, it is an absolute mandate that the app meets user expectations. Under this premise, selecting the right device is a fundamental part of the entire process as the performance and functionality can be varied depending on the mobile device being tested. This is due to the fact that there is a wide range of devices with diverse screen resolutions, processing capacities, or even hardware pieces that are integrated into the apps.

Let us take a look at specific strategies that can help you determine the right test devices for testing mobile apps.

1. Narrowing the scope

• Narrowing the scope and identifying the list of test devices that are a 100 percent fit for the test requires the following steps:
• Collecting the requirements — It is essential to collect all the requirements for the product and check if there is any device or devices that you want to include compulsorily.

• Identifying the market share for different platforms — It is crucial to analyze the market and your potential customers to identify the market share for different platforms easily. Every region/country might have a different market share for different devices and platforms, and from hereon, it is important to leverage the available statistics from trusted sources and identify the devices that the target audience is most interested in. With this, you’ll also be able to target the users more efficiently and smoothly.

• Establishing testing goals and constraints — Prior to starting the testing process, it is essential to clearly define and understand the testing goals and probable constraints that might hinder the process. Testing goals may include checking the usability, performance, security, network connectivity, functionality, compatibility, and many other aspects.

Once the goals are defined, you should prioritize them and assess the number of test devices.

For instance, if your app is a mobile banking app, security is one of the biggest priorities. Therefore, you will need to test the app on only a few devices since the app’s security doesn’t depend much on the firmware or hardware. On the other hand, if performance is the core priority of an app, testing across multiple hardware and firmware is an important aspect.

When it comes to the constraints, the budget remains one of the most significant ones. If you wish to test the app on several devices, it will require more time and result in many working hours for the QAs. Therefore, any budget limitations will directly affect the number of devices that you can test on. In this case, one can use the most popular devices for the target audience and get back to the second picks later if required.

2. Identifying parameters for choosing the devices —

OS versions:

In the case of the Apple products and their unified hardware and software, it is often simpler as you can consider only the latest and second latest versions to support. While for Android device testing, several OS versions are used simultaneously. For instance, Android 9.0, Pie, Android 10.0, Android 11.0, and Android 12.0 are some of the popular versions existing parallelly, and hence all of these need to be considered. Certain factors to consider while testing on various Android versions include:

• Determining the precise target audience and their preferred version
• The countries of residence of the target audience
• The share of different OS versions across these countries
• Screen sizes and resolutions

It is essential to ensure that the UI of the app is compatible with different screen sizes and resolutions. Due to a wide range of devices with multiple screen sizes and resolutions available in the Android market, it is advisable to check the statistics to observe users’ devices’ DPIs. These data are crucial when choosing the devices for testing as it helps show the preferences of real users who will be using the app.

- Hardware

Certain applications might require specific hardware to function properly. In these cases, choosing particular devices that can provide the necessary hardware is essential.

Further, it is also vital to focus on the processor architecture. While testing, one must ensure that the app works equally on the primary architectures.

- Software

Certain apps depend on the specific built-in features of the devices, such as biometric support, 5G, and many others. If the app is built around these features, it is a mandate to choose devices that possess the required functionality; else, the app might not be able to deliver as intended.

- Considering the target users

Selecting devices based on certain factors critical to your target audience is essential to ensure broad testing coverage. Key data points to consider include:

- Market share

• Traffic volume
• Popularity
• Region

Mobile Vendor Market Share North America

Mobile Vendor Market Share Asia

Mobile Vendor Market Share Europe

• Industry
• Different device configurations (as discussed priorly)
• Company size

3. Reviewing the statistics of the most popular devices—

It is crucial to review the statistics of the most popular and widely used devices in the global and local market that can assist in narrowing down the list of devices for testing.

Websites like StatCounter, Statista, and much more constantly update statistics related to operating systems, processors, screen resolutions, and device models that can help choose the devices for testing.

‍4. Selecting the right amount and variety of devices

The type of tests you’re running often determines the number of test devices required for your app. As manual testing is time-consuming, testers can select a small number of devices from the range of devices popular among the target audience. On the other hand, automated testing allows selecting from a broader range of devices.

- Manual testing

A high-end model, a low-end model, and a tablet (depending on your app) can make the right combination for manual testing. For Android device testing, a suitable practice is to include a popular device brand like Samsung and begin with a high-end and low-end device. For better brand coverage, add other popular brands like Xiaomi, Oppo, Motorola, and many more. Google Pixel devices can also assist in commencing the Android beta version testing.

- Automated testing

It is advisable to add a larger mix of OS versions for automated testing to achieve optimal coverage and ensure faster test execution.

- Test speed v/s granular device selection

Based on the requirement of speed, it might be more effective to focus on choosing the right device than the granular elements of the screen size, OS, and others. However, in this case, choosing the Android and iOS devices that the target users most extensively use is vital to ensure better test coverage.

- ‍Parallel testing

‍Parallel testing allows testing multiple configurations of an app at the same time. One can seamlessly run parallel testing on a single app or several components on several platforms and allows to test the app, subcomponents such as OS, browsers, or even both. Customers subscribe for multiple device types to ensure they get the speed in testing and align well with their testing goals and metrics.

5. Reassessing the test devices periodically

With the evolving mobile devices market, new devices are emerging every now and then. Therefore, it is more crucial than ever today to focus on the current market conditions, new models, device configurations, and OS versions while planning the testing coverage.

For choosing the device versions, evaluating the technical specifications of similar models is helpful. Again, Google Pixel devices are often the first to support the latest version of Android OS for Android devices.

Top selling smartphones 2022

• Apple iPhone 13
• Samsung Galaxy S21
• Honor 50
• Xiaomi Mi 11
• Realme GT
• Infinix NOTE 11 NFC
• Redmi Note 10 Pro
• One Plus 9 Pro
• Huawei nova 9
• Samsung Galaxy A72

6. Creating a device matrix

With the information acquired till this stage, one should attempt to create a device matrix that contains information related to OS version, model, resolution, CPU, memory, and the associated market distribution data.

‍7. Look out for similar devices

As the device matrix is ready, the next step involves finding the mobile devices that have similar characteristics and further narrowing down the possibilities of discarding devices that have characteristics already covered by a similar device.

Conclusion

However, the selection of mobile devices is not limited to a fixed list. It depends widely on the specifications and requirements of each individual project. Keeping in mind a particular strategy and the tips mentioned above can nevertheless help refine the final list of devices selected for your app testing.

Given the extreme importance of mobile devices and their apps today, it is pivotal to ensure that these are thoroughly tested before the market release. Doing so is a major step forward toward improving the brand’s reputation as a developer and building a positive relationship.

Source: https://www.headspin.io/blog/strategies-for-choosing-the-right-mobile-devices-for-testing