DEV Community: Kalei White

Mobile Game Testing: Industry Leaders’ Challenges and Strategies

Kalei White — Mon, 10 May 2021 22:16:38 +0000

In 2020, the mobile gaming industry saw a massive spike in engagement and revenues, growing more than 13% year over year. With game quality and performance being of critical importance to stakeholders in the space, improving it has become top priority for mobile game development and QA teams globally.

With this topic in mind, we invited Aliaksandr Kavaliou, Performance Test Manager at Playtika, Robson Siebel, QA Lead at Infinity Games, and Melissa Eaden, Quality Engineering Manager at Unity Technologies for a panel discussion with Apptim CEO, Sofia Palamarchuk, about mobile game testing!

Watch the webinar panel discussion here: https://www.youtube.com/watch?v=UF3lmsUVVAY

Below is a recap of the discussion, including questions provided by the audience:

What Are Some of the Biggest Mobile Game Testing Challenges You’ve Faced?

Melissa Eaden shared that one of the biggest challenges teams find themselves tackling first is testing on the myriad of different devices and operating systems that are a direct result of the growing fragmentation within the Android ecosystem. For example, the emergence of newer devices every year, makes testing more complex than ever before.

Mobile game testing and quality isn’t only focused on functionality and performance, Eaden shared. She and her colleagues are focused on helping Unity developers to make games that are engaging and interactive enough that keep players coming back again and again.

Some other areas in game quality that present challenges, she added, were:

Ensuring compliance with new privacy laws and regulations such as GDPR,
Testing ads and third party integrations, and
How to create games with high-quality, using advanced features such as inter-player chat, matching, awards, etc.

Robson shared that at Infinity Games, they create games that are designed to be immersive and relaxing, so there’s nothing worse than a user finding a bug, thus ruining their experience. Worse, if that user that experienced a buggy game leaves a negative review on the app store, it could lead to a negative financial impact for the business.

Another challenge for his team in particular is that they don’t have a dedicated tester, which, in the past, had resulted in devs being handed over imprecise or less helpful bug reports than desired.

For Aliaksandr, his greatest challenges at Playtika revolve around capacity testing / capacity planning on a huge setup.

He explained, “Once you have more than one data center, hundreds of virtual machines, a lot of Kubernetes setups and so on… data warehouses, databases, and hundreds of thousands or millions of users, and you’re awaiting a new feature or a marketing campaign that is expected to generate a user traffic increase… Planning for this enormous setup is a headache, really.”

The second biggest challenge he mentioned in mobile performance testing for games is the high cost of running end-to-end performance tests. The closer to production, the more expensive it is for any team to run these tests. He elaborated, the cheaper the tests are, the less realistic numbers you can obtain, and you can’t somehow approximate those results and understand what exactly you need and how much to scale.

What Are Some Ways You’ve Overcome These Challenges?

For Robson, he has found ways to help his team overcome the challenge of not having dedicated testers. He assumed the role of the QA lead and also adopted Apptim, which has helped enable anyone on the team to run functional and performance tests on their games.

He shared that, before Apptim, “The bug reports were either vague or they were sometimes incorrect and that led to a lot of wasted time for the developer trying to hunt or to reproduce a bug.” He continued,

“Being able to have a tool (Apptim) that recorded the play session of the person and automatically got all of the logs and generated a performance report in the end, is something that for us, as a small company, was pretty huge. We’ve improved the quality of the games a lot because of the crashes or exceptions that we couldn’t catch before, and now we have that access.”

Aliaksandr shared that, to deal with the high cost of end-to-end mobile game performance tests, he found you can reuse existing automation tests. He commented, “It will also save you time in such cases because almost every time you have automation already created for your functionality, why not add some markers or some [performance] measurements, and it will help you to also understand what’s going on where you are.”

But, he warns, although that’ll help, it can’t protect you from bad code and a bad configuration, so a lot of effort and attention should be placed on the configuration of your performance test, not to saturate earlier than expected due to some numbers conflicting somewhere. “That’s why it’s hard, step-by-step analytical work,” he asserted.

What is Your Advice for Reducing Testing as a Bottleneck?

For Melissa, the short answer may seem controversial to testers, but she shared that the best thing she’s learned to do is to get out of the developers’ way. She said,

“Removing myself has been the biggest, reducing that bottleneck, putting automation in place, letting the developers really collaborate on those ideas and moving the production cycles and everything faster because I am not stationed on a team, I’m not embedded, but I’m there as somebody that can coach them and help them.”

She explained that she’s a big advocate of modern testing principles, which give a lot of credence and leverage to the teams themselves. “Developers know the code the best, the more they interact with the code and the more they understand the code, the more they can test it, and they can test it on a level that oftentimes as someone who was traditionally in the role of a tester might not be able to get to.”

She continued, this is the collaborative part; helping developers understand where their knowledge gaps are and that’s where she comes in. She’ll often be the person that asks questions like, “Hey, you’ve developed this wonderful piece of functionality, but did you think about how to scale it?”

She remarked that there are a lot of moving pieces and developers are really well positioned to handle that. It’s about helping developers uncover their blind spots.

Melissa added, “So I’m not perfect, they’re not perfect, but getting out of the collective folks’ way that deal with the code on a day-to-day basis has really improved removing that bottleneck from the development process, along with adding in DevOps and CI/CD practices, which have just made things a lot easier.”

“Automation always makes things easier as much as it makes it harder, but you have to balance the positive with the negative, right?”

The first answer that came to Robson’s mind about reducing testing as a bottleneck was, “You can always ask the developer to stop creating bugs. That works very well.”

Kidding aside, he shared that in his experience, having a well-defined test pipeline in a reliable software that can automate some of the steps goes really a long way.

Another key area of improvement has been reducing the friction between developers and whomever is testing the games:

“Using a tool like Apptim for that can also help catch some underlying bugs and reduce the friction between the tester and the developer because sometimes there is a lot of back and forth.”

To illustrate the point, he said, “Today, I had an issue where someone reported a bug. ‘Hey, this is happening at the end of level six of the game.’ I checked the video from Apptim because he was using it for that session, and it was actually at level 10. So if I didn’t have the video to back it up, I might have lost quite a lot of time chasing a bug that didn’t really exist.”

How is Testing a Mobile Game Different Than Testing Other Mobile Apps?

The consensus among the panel was the added complexity that games have at different levels and understanding how they interact. Melissa stated,

“Mobile apps are complex as it is, but you have multiple platforms you’re looking at, and then you multiply that by the game complexity and the algorithms and the performance you have to maintain.”

Melissa shared several other factors that add more complexity in mobile game testing:

Handoffs between ads, online chatting, or any other auxiliary things that your app is doing besides just being a game.
Dealing with game performance, device performance, and network performance.
Testing various ad layouts and formats: For example, you can have a full page static ads, banner ads, video ads, etc.
Non-linear usage and behavior patterns from users open a wide array of possibilities. Depending on the order that the user does something, you might find a different bug.

What Aspects of Non-Functional Testing Like Performance Do You Prioritize in Your Games and Where?

For Aliaksandr, when choosing where to prioritize performance, he stated areas like the game lobby and anything a brand new player will see first are top priority.

He explained that, for a fresh player, he or she should see the functionality of your game as soon as possible and enter the game fast. This aspect should be the most prioritized because players will not want to wait a long time to get started.

But, he added, it depends on what experience the user has during the waiting; “If it’s something interesting, he interacts with something, even during the game, during the waiting lobby or whatever inside the game, it’s maybe a good thing. But if he sees just a blank page and a spinner, it’ll make him crazy. Of course, for fresh users, I suppose getting to the lobby as fast as possible is a critical thing, but then other features should work also fine.”

How Do You Set Mobile Game Performance Benchmarks?

Aliaksandr volunteered to expand upon this follow up to the previous question, saying that it can be measured from the number of users. But first, it’s important to understand business critical transactions. Then step by step, you can do benchmarking, load testing, and so on. He also shared, “Use such a tool as Apptim to set markers and see what’s wrong with these events, with this transaction, this feature, deep dive, and use code profiling… and so on.”

He stated that self-written solutions will be the most accurate because you inject something in your code and this code will show the real duration, but it takes time to do. Therefore, if you have a good profiler on hand, which can easily show you the bottlenecks, then it’s better to use that at first.

Don’t forget to watch the full webinar for more insights from our guests in this panel discussion and check out similar videos on Apptim's Youtube channel!

This article was originally published here

How to Enhance User Experience by Improving Mobile App Performance

Kalei White — Fri, 22 May 2020 21:52:58 +0000

What’s one of the biggest predictors of a mobile app's success?

Performance (how fast it loads and how reliable it is).

The findings of a mobile app user survey conducted by Dimensional Research showed that 80% of app users will only attempt to use a problematic app three times or less and 36% said that an app with slow performance issues made them have a lower opinion of the company.

In this article, we'll cover how to whip your mobile app's performance into shape so that crashes and slow loading times don’t cost you downloads, revenue, and brand perception.

What Is Mobile App Performance?

Keep in mind that, when we discuss mobile, it includes not only smartphones, but also their cousins: the legions of tablets, smartwatches, and other smart devices on the market today.

Performance is one of the factors of app quality that has to do with how it behaves when experiencing certain load sizes and in various other situations. For example, does the app load slowly or crash every time it experiences a peak in user activity, like when you launch a special promotion?

Performance is a product of multiple factors: the server, mobile device, network, and programming of the app itself.

Unfortunately, a user may have a poor experience with your app’s performance, even if the cause of the issue has nothing to do with the app itself.

In the previously mentioned survey by Dimensional Research, 55% of respondents have held an app responsible for performance issues although many admitted they didn’t know what the cause of those issues were.

Why Mobile App Performance Matters

In the same Dimensional Research survey, nearly all users (96%) stated that performance is important, while over three quarters (76%) shared that it is “very important” or “critically important.” The same study even showed that users actually care more about the performance of an app than its price.

What’s the target for mobile app performance?
While there is no magic number, several surveys and studies have shown that two seconds or less is the optimal amount of time it should take for an app to load in order for users to be satisfied.

In accordance with that, according to a survey by Akamai, 47% of consumers expect web pages to load in two seconds or less and 20% expect mobile apps to load instantly.

Users are fickle.

When a mobile app’s performance doesn’t meet user expectations, there is a very high chance they will abandon it, resulting in a loss of potential revenue.

Ratings can also suffer, which leads to a decline in downloads. One app maker reported that even a 0.1 drop in an online app rating caused a 5% decline in downloads, while a 0.3 decrease resulted in a 60% drop. What a difference!

Although not terribly common, 7% of users will write a bad review if an app has errors, crashes regularly, or stops responding, and 8% will give it a poor star rating for the same reasons.

On the flip side: When your app performs well, it can be a huge money maker.

On average, mobile shoppers are more engaged, spend more, and shop more often.

Tammy Everts, a mobile performance expert from SOASTA, conducted a case study showing that a one-second decrease in page load time resulted in a 27% increase in its conversion rate.

In the same study, mobile pages that loaded in an average of 2.4 seconds enjoyed the peak conversion rate (1.9%) over a span of 30 days.

This has been reflected in many studies of web page performance as well. For instance, according to a report by SOASTA and O’Reilly, Intuit cut its load time in half and saw a 14% increase in conversions.

In short, any company that produces web or mobile apps should make performance a primary concern, integrating performance testing as a mandatory part of the QA process.

Tips and Tricks to Improve Mobile App Performance

Now that we’ve covered the need for superior mobile app performance, let’s look at some of the ways to enhance it. Some are easier to implement than others. But if made, these changes will be worthwhile.

Cache images

Caching is a part of a computer’s memory that stores recently used information. It can store web pages, images, files, etc. on its local hard drive to provide rapid access to them while improving performance and decreasing traffic on the network.

Downloading external resources takes time, so make sure to cache your images.

Compress and resize images

When you compress an image, you minimize its size in bytes without greatly degrading its quality. There are several sites that do this, but the one I typically use is compressor.io.

Also make sure that the image size is right because it’s a waste of bandwidth to rely on the browser to scale a high-resolution image into a smaller width and height. So, you want to make sure images are adequately sized and ready to go off the bat.

Re-use data templates

When the app has to load multiple templates, that slows it down. To make the app faster, load fewer templates by reusing them.

Reduce HTTP Requests

Use fewer HTTP requests required to fetch the resources for each page in order to simplify things for your app.

Use loading validations

People perceive load times slower than they truly are. So, even perception of performance matters.

Make your app load in a way that makes it seem to be going faster, even if it isn’t!

One way to do so is to have it provide some visible, instant feedback as the application waits for the background response. Putting something up during the loading process will give users the perception that it’s loading quicker.

Load data as you need it

If your app requires a large amount of data, it’s useful to only load it as you need it. Don’t make the user wait longer than necessary to start viewing parts of the app.

Splitting up the assemblies will help accomplish this. You could also pre-load/pre-fetch data, if possible, so that when the data is needed, the user doesn’t have to wait.

Create an offline mode

In the event that the network connection drops while the user is in the middle of completing an action, it’s smart to alert them as to why their action cannot be completed. Then give them the option to save the data for later, so they can resume the activity once the connection has been re-established.

This will at least give them some peace of mind — certainly more than I had when I experienced my own problems with the travel app!

Test for performance early and often

Apps must undergo all sorts of testing to ensure that they work as intended, are compatible across devices and networks, etc. They should also be rigorously tested for performance and monitored frequently.

One misconception that many developers have is thinking you have to wait until the end of development to start the testing process. It’s actually a time and cost saver to test as you go, when bugs and performance issues are smaller and easier to fix.

Use the right tool for performance tuning

There are several tools on the market to execute mobile performance tests, such as Apptim.

Basically, you can download Apptim and connect a device and start exploratory testing your app right away, pushing bugs to JIRA as you find them while at the same time, the tool assesses performance for you, finding crashes, exceptions, measuring resource usage, power consumption, etc. It then shows you a neat report of everything that it finds.

It’s a great performance tuner because it runs a deep performance analysis that includes checking response time and resource consumption (cpu, memory, data transfer, render, database, etc).

It then shows you a report with everything you should know about your app, its weaknesses, and how you can fix them.

Put an APM (application performance monitoring) system in place

APM refers to the monitoring and management of performance and availability of software applications. APM helps detect and diagnose deep-level application performance problems to maintain an expected level of service across the board.

It monitors two sets of performance metrics: those that have to do with the performance the user experiences (like average load times) and those that measure computational resources that the app uses for the load.

There are several companies that provide APM Management tools such as New Relic and AppDynamics.

Make your company’s culture performance-centric

As TechBeacon editor, Todd DeCapua, has stressed many times, performance is now everyone’s concern from the engineering department to the marketing department.

Organizations should get behind the attitude of testing early and testing often.

If all stakeholders support the vision of having high-performing apps, it will be easier to devise a plan and stick to it. It’s also important for everyone to be aware of the causes of slow performance so that issues can be detected and remedied sooner.

Performance Matters

Mobile app performance directly affects user retention rate, ratings, number of downloads, conversions, and ultimately, revenue.

If you commit to creating a high-quality app that meets or exceeds user performance expectations, your users will want to use it over and over again, elevating their perception of your brand and strengthening the relationship they have with it.

Have some more tips? Please leave me a comment!