DEV Community: Sofia Palamarchuk

How to Add Performance Metrics to Your Appium Tests

Sofia Palamarchuk — Tue, 08 Sep 2020 16:31:41 +0000

In this post, I will show how you can get mobile performance data using Apptim while running Appium automated functional tests.

For anyone involved in the development of a mobile app, being able to test it thoroughly and even automate some of the process is a huge relief and time saver. This is where mobile app automation becomes important.

There are many ways that testers and developers can find out about bugs and potential issues in their app. Apptim is a mobile performance testing tool that can be used during manual app testing, exploratory testing, and debugging. And lately, many Apptim users have been wondering if there’s a way to use it along with their automated tests.

The answer is, yes!

Appium is an open source tool that enables you to write automated UI tests for Android and iOS apps. In order to use Appium, you will need to install the Appium server and create some tests for your app.

In this post, I will show you how you can use both tools together for more complete mobile app testing in just three simple steps:

1- Run Appium Java tests on an iOS app

2- Install Apptim and run a test

3- Review the Apptim report

Example of How to Use Apptim with Appium

1- Run Appium Java tests on an iOS app

For this example, we will be using IntelliJ to write and run our tests in Java. We created a class called DemoApptimTest, where we set up our desired capabilities.

We then created three tests for our iOS retail app: filter a category, select a product and add a product to cart. We’ll use Java to write our tests and TestNG as the test runner.

Example of a test to filter products by category, “Phone”:

We can debug and run the Appium tests from inside IntelliJ.

2- Install Apptim and run a test

We then download the Apptim Mac desktop application from here. Apptim requires installing some dependencies, which it will list for you, and after clicking “Install Dependencies,” it will begin to install them for you if they are not installed already.

We already have our retail app installed in an iPhone. To start a test, we will simply connect the device to our Mac through a USB cable and open Apptim.

Then, Apptim will ask what type of test we’d like to run. In this example, we will be running our Appium tests and we want Apptim to capture the app performance data. So, we’ll choose the exploratory tests option when prompted, and put a name to our test.

Apptim will start recording everything that happens in the app. We now go back to our IntelliJ IDE and click on “Run” to run the Appium tests. Make sure you have the Appium server running before starting the tests. At this point, the automated tests will exercise the app and Apptim will record everything and generate a report at the end.

Once the Appium tests are done, we click on “End Session” in Apptim and wait for the test session report (which will be generated automatically).

3- Review the Apptim report

Once we’ve finished our test session, Apptim will generate a report with a summary of all of the performance data that it has automatically captured for us while the Appium tests were running. We can open this report locally or publish it to our Apptim cloud workspace.

Here are some screenshots that the Apptim Report generated for our retail app:

And here we can see how the memory usage was increasing over time:

You can check the full report here.

With just a click of a button, Apptim captured a lot of insights for us!

More Resources

To learn more about the different mobile performance metrics that Apptim covers in its reports, check out this article.

Check out this blog post where I cover how to run automated tests with Apptim using the open source tool, Taurus.

Get More Out of Mobile App Automation

These days, mobile performance is more crucial than ever, but it’s also a challenge to gain visibility into all of the factors that impact it.

Apptim provides easy-to-read reports to check mobile app performance, review logs, and crash details while going about your manual or automated functional tests. Teams can now release new versions of their app with confidence by getting full visibility on app performance trend with minimal effort.

I hope you found this article useful and feel better prepared to add a performance test to our current mobile app automation efforts!

16 Best Podcasts on Software Testing

Sofia Palamarchuk — Fri, 14 Aug 2020 05:07:40 +0000

Podcasting has been around since 2004, but now in 2020, it’s safe to say that we’ve entered the golden age for podcasting.

According to Podcast Insights, there are over one million active podcast shows with over 30 million episodes.

As with the growth of vlogging and youtube, podcasting has exploded because it’s one of the easiest ways for anyone to cultivate or reach out to an audience, no big-time contracts or middleman required!

There’s a podcast out there for absolutely everyone on just about any topic, whether it’s broad or the nichest of niches.

If any devs in this community want to learn more about testing, then boy do I have a good list of podcasts for you!

Software Testing Podcasts

1. AB Testing

Known for his popular weekly blog posts, “5 for Fridays”, Alan Page, aka “Angry Weasel” and friend, Brent, created this podcast to talk about modern testing including topics like Agile, data, leadership and more. There is a great community around this podcast and it’s a very fun one to listen to. Recently, they started live streaming the podcast on Youtube, so you can catch them chatting there as well!

Website: https://www.angryweasel.com/ABTesting/

2. The Evil Tester Show

The Evil Tester Show is a software testing show produced by test expert, Alan Richardson. He discusses software testing and related topics from software development like test automation.

His reason for creating the podcast? “TLDR; The world needs another Software Testing Podcast. So I created one.” Alan is insanely knowledgeable about testing and views it from a programmer’s perspective, which makes it all the more interesting.

Website: https://www.eviltester.com/show/

3. The Guilty Tester

The Guilty Tester stated the purpose of their podcast very bluntly, “Do you ever feel guilty for not meeting the standards set by others in the Software Testing community? You’re in the right place then.”

Website: https://theguiltytester.libsyn.com/

4. Let’s Talk About Tests, Baby

The show with perhaps the best name on this list, “Let’s Talk About Tests” is a podcast with over 100 episodes on software testing, QA and Agile. Created by Gem Hill, she decided to start the podcast to help herself learn and become a better tester.

Website: http://letstalkabouttests.xyz/

5. TestGuild Podcasts

There are three TestGuild weekly podcasts, one for automation, security, and performance hosted by Joe Colantonio, which geek out on all things software testing. In each episode, he has a different guest who brings their perspective to the table. You might recognize Joe as the creator of the online conferences: TestingGuild, PerformanceGuid, AutomationGuild, and SecureGuild.

Website: https://testguild.com/podcast/

6. The Ministry of Testing Podcast

The Ministry of Testing is a very special community online that is completely dedicated to building up the testing profession and helping testers learn. Its podcast is made of interviews of software testing experts that discusses specific software testing topics. The podcast also delves into non-technical topics such as office culture improvement and diversity and inclusion.

Website: https://dojo.ministryoftesting.com/dojo/series/podcast-series-ministry-of-testing

7. Perfbytes

Perfbytes is a podcast managed by Leandro Melendez, Brian Wilson, James Pulley, and Mark Tomlinson. All four of whom are experts with impeccable experience in performance and load testing, having worked for large organizations. The podcasts are mainly centered around software testing tools and conferences. A great thing about it is that there is also Perfbytes Español for Spanish speaking listeners.

Website: http://www.perfbytes.com/

8. Quality Coaching Roadshow

Tune into this monthly podcast hosted by Anne-Marie Charrett and Margaret Dineen to listen to the unique challenges and the context of Quality Coaches from around the world. Anne-Marie Charrett is an internationally recognized expert in quality engineering and Margaret Dineen is Director of 3WestStreet and the founder of Encompass Testing.

Listen here: https://www.spreaker.com/show/quality-coaching

9. Quality Remarks

This is a podcast about software testing and quality management produced by Keith Klain, which seems to be becoming active after a hiatus. In the latest episodes, you can find guests like Michael Bolton, Rob Lambert or Anne Marie Charrett. Klain is a professional with over 20 years of experience managing enterprise-wide quality programs for financial services and global IT consulting firms, so there is much to learn from his show!

Website: https://www.listennotes.com/podcasts/quality-remarks-the-podcast-keith-klain-df-EmybMrei/

10. Quality Sense

Quality Sense was created by Abstracta co-founder and COO, Federico Toledo, with the aim of helping to share knowledge about topics related to software testing through weekly one-on-one chats with different guests and personalities in the field of testing. Listen for insights on today’s most pressing topics and good practices from the creators and collaborators of different software testing tools, methodologies, conferences, etc.

Website: qualitysensepodcast.com

11. Test and Code

Test and Code is a well-established podcast produced by Brian Okken covering a wide array of topics related to software engineering and testing, especially around the Python language. It focuses on testing and process questions like “How do I know it works?”, “How do I effectively test?”, etc. The podcast even has a Slack community!

Website: http://testandcode.com/

12. Tester’s Island Discs

What would you do if you were stuck on an island with only five songs and one book to keep with you? That is the idea behind tester’s island, another podcast by the Ministry of Testing community. In each episode, Neil Studd interviews one of the members of the global QA community and asks them that very question. This is a great, light-hearted show which is of course, not purely about testing, yet very insightful.

Website: https://www.ministryoftesting.com/dojo/series/testers-island-discs-podcast

13. Testing One-on-One

This podcast was created by Rob Lambert and Joel Montvelisky wherein each episode, they answer a number of interesting questions from testers and that focus around a number of different areas of the software testing profession. Rob and Joel have decades of combined experience in testing and management and are very well known for their high quality software testing content.

Website: https://qablog.practitest.com/podcast/

14. Testing Peers

The Testing Peers is a newer software testing podcast produced by four like-minded software testers. Initially, they formed a group chat wherein they would discuss certain issues, rant, and ask for advice. The group chat then became a Slack community, but they still wanted another forum for discussion. So, they launched this podcast wherein they come together to support and challenge each other on their journeys as software testers and leaders.

Website: https://testingpeers.com/

15. That’s a Bug!

In each monthly episode of “That’s a Bug!” the hosts discuss one specific bug. For example, in one episode, they discussed a particular Boeing 787 bug that required the aircraft to be restarted every 51 days or else it could cause a loss of control. If you ever needed a podcast to convince others that there’s tremendous value in software testing, this would surely be it!

Website: https://thatsabug.podbean.com/

16. Women Who Change Tech

The Women Who Change Tech podcast from Women Who Test highlights women who are contributing, trailblazing, and disrupting in the business of technology. In each episode, hosts Alison Wade, president of TechWell and founder of Women Who Test, and Jessie Shternshus, owner of Improv Effect offer a dose of inspiration from some of the most talented and creative women who are changing the face of technology. Many of the episodes center around leadership, creativity, and the soft skills needed to succeed.

Website: https://www.womenwhotest.com/

Let me know, are there any other software testing podcasts worth adding to this list? Leave a comment! Thanks to my friend, Kalei, for helping me write this.

How to Test Mobile App Performance with Apptim

Sofia Palamarchuk — Fri, 14 Aug 2020 04:45:21 +0000

Hey guys! I'm co-founder and CEO of Apptim. In this post, I want to share with the world why we built this free tool and how you can use it for mobile performance testing.

Testing mobile performance can seem tricky and time consuming, but it’s an aspect of the mobile user experience that developers and testers can’t overlook in 2020.

According to Google, 70 percent of users abandon an app because it takes too long to load.

Similarly, App Samurai has found that 70 percent of app uninstalls are due to app crashes.

These are both negative consequences related to poor mobile app performance that brands must avoid if they want their apps that they’ve poured so much time and energy into making to be a success.

Mobile performance involves measuring, analyzing and optimizing an application under different circumstances, to increase its robustness and reliability.

Apptim can be used during development and testing to catch critical bugs faster and identify performance issues in an app without the need to install an SDK or change the source code.

This makes it easy for manual testers, developers, and even product owners to run a test.

It automatically measures app render times, power consumption, and resource usage while capturing crashes and more on Android and iOS devices, providing developers with the visibility they need to improve performance and enhance the user experience.

Once you’ve installed Apptim, you can run your first test and obtain performance results in as little as five minutes.

In this post, I will share how to use Apptim to capture client side performance data using the Android demo app developed by AWS.

These are the steps we’ll follow to test mobile performance with Apptim:

Install Apptim Desktop
Install the demo app
Run a test
Review and share your results

1 – Install Apptim Desktop

There’s a simple two-step process to configure Apptim Desktop.

First, you can create an account on the Apptim website and then download the Windows or iOS desktop application. Apptim requires installing some dependencies, which it will list for you, and after clicking “Install Dependencies,” it will begin to install them for you if they are not already installed on your computer.

After the installation process is completed, you will see Apptim’s home screen.

2 – Install the demo app

You can download the Android demo app from here. Then, connect your Android phone to your PC or Mac using a USB cable. Make sure to follow these instructions to correctly allow Apptim to access the device later in the test.

If you were to test an iOS app, you would follow all these same steps, except instead of connecting an Android device, you would use an iPhone or iPad and your Mac.

You can install the demo app in the Android device by executing the following command from the Terminal in Mac or Command line in Windows: adb install

Make sure to run this command from inside the ../android-sdk/platform-tools/ folder or add the Android SDK directory to your path.

3 – Run a test

Once you have the demo app installed in your device and the device is connected to your PC or Mac through USB, you are ready to start a test with Apptim!

Go to Apptim and click “Start a new test,” it will ask you to select your mobile device and then you can choose the app on the device that you want to test or search for it by name. Here you will search for “ReferenceApp” and then choose the app from the list.

Then, it will ask what type of test you’d like to run. Currently, only exploratory tests are supported, but in the future, there will be the option to run a bot test.

Next, it will ask you to add a name to your test and press “Start” to begin your exploratory testing session.

During the test

At any moment during your testing session, you can choose to report a bug or take a note. You can end your testing session anytime by clicking on “End Session.”

Report a bug

When you click on “Report bug,” you will be prompted with a new screen where you can give the bug a title and description, add a tag, and attach a video and screenshot that were automatically captured. You also have the option to edit the screenshot.

Take a note

When you click on Take a note, you will be prompted with a new screen where you can write a Note, a Risk identified, an Idea, or just a To do as a reminder for later. You can also attach a screenshot.

Review session

Once you end your testing session, you will see a summary with bugs, ideas, risks, and notes taken.

Here you will also have a list of all bugs captured and within each bug, the option to publish a bug to your workspace and publish a bug to Jira.

4 – Review and share your results

Once you’ve finished your test session, Apptim generates a report with a summary of all of the performance data that it has automatically captured for you while you were testing the app’s functionality.

You can publish the test session report to your workspace in the Apptim Cloud and then share it with other team members. You have the option to share it privately, by adding team members to the workspace, or publicly, by generating a link.

In each Apptim report, you’ll find:

A screen recording of the device during the execution of the exploratory session
Alerts and warnings related to the user experience (nested layouts, screen controls, rendering times, etc )
Alerts and warnings related to the usage of device resources (CPU, Memory, Threads, Power Usage, Network Data), with reference thresholds, shown in yellow or red depending on their severity
Test environment data (device, app, version info, etc)
Crashes/exceptions details with associated screenshots, and the exact minute of the session in which it happened in order to locate the specific moment in the video
Detail of resource usage with associated graphs
Logs and downloadable artifacts collected during the test

Here are some screenshots of what an Apptim Report includes:

Here you can also see an example of an iOS Apptim report.

To learn more about these different mobile performance metrics, check out this article. To understand more specifically about understanding Apptim reports, check out these docs.

I'd love to know what you think! Give Apptim a try? :)

How to Test Mobile App Performance (iOS & Android): 3 Key Factors

Sofia Palamarchuk — Thu, 18 Jun 2020 21:58:34 +0000

In this post, I’ll go over the different factors and scenarios that impact a mobile app’s performance. I’ll also explain how to test mobile app performance in the different ways that matter including which tools and metrics should be involved.

You’ve probably interacted with an app on your phone or tablet that’s slow, takes a long time to load, freezes or even crashes on you altogether.

Frustrating, right?

On the flip side, we’ve all experienced apps that, from day one, have never given us any trouble and we love them for working so fast and effectively.

Or, we haven’t even thought about it all because isn’t that how they should be?

What causes one app to be crash-prone and another, fast and reliable?

Whether an app has good or bad performance depends on three factors combined: the backend, the network, and the app itself running on the device.

A developer or mobile tester can measure the performance of an application in different scenarios.

For example, they can test for when there’s a concurrency of users on the app at the same time, on different devices (which vary in hardware resources and screen sizes), and multiple networks such as 3G, 4G, Wifi, and more.

The reality is that there are many variables that affect the performance of a mobile application. Moreover, a user may have a very bad experience with your app and the cause might not even have anything to do with the code or its implementation.

But, by running performance tests for each of these three factors, you’ll be able to identify problems and optimize your app for the best user experience possible.

Keep reading as we’ll cover the different types of tests for each factor, what to measure, and what tools are available to help you along the way.

1st mobile performance factor: the backend

A mobile app’s backend architecture is generally based on an application server, a web server and a database.

When it comes to the backend, the things related to performance that are important to know when an app is under load are the server’s response times, database queries times, and the server’s resource usage.

Using this information, it’s easier to detect issues such as:

High server response times
Bottlenecks or breakpoints in the database and application server resources
Poor implementation of escalation policies

So what kind of tests are normally run to check the app’s backend performance? Load tests.

This is when you simulate load on the backend in different ways, whether it be through stress testing, peak testing, endurance testing, load testing, etc.

In general, the objective of these tests is to understand how the backend systems of an app behave and handle a certain volume of concurrent users.

There are several tools that allow you to load test your mobile app. The most commonly used ones include:

Apache JMeter – the number one open source load testing tool
Gatling – a developer-friendly, open source load testing tool with scripts written in Scala
BlazeMeter – a cloud performance testing platform that scales your JMeter or Gatling tests for a greater amount of concurrent users

2nd mobile performance factor: the network

With regards to the network that the device is connected to, there are two key things to measure: latency and bandwidth.

Latency is the time that elapses when information is sent on the network (measured in milliseconds).
Bandwidth is the maximum capacity (the amount of data) that can be transmitted through the network (measured in bits per second).

For mobile performance, the lower the latency and the higher the bandwidth, the better.

An app’s performance can vary depending on, for example, whether it’s connected to a 3G network or a 4G network, and unfortunately, this is beyond an app developer or tester’s control.

But, it is possible to incorporate the network during the mobile app performance testing process, simulating the different types of networks and measuring their impact on the response times, both on the server side and the client side.

Luckily, the third factor is something that’s been overlooked for years and we now have ways to incorporate it into our test cycles and gain greater control over it.

3rd mobile performance factor: the client

For a mobile app, the client is the software that runs on the device, the app itself.

With client-side testing, the variety of devices, operating systems and screen sizes of a device that an app is running on comes into play.

Today there are thousands of mobile devices on the market which vary by brand, hardware and operating system.

And each year, it becomes more and more difficult to ensure that an app works well on most of the devices on the market because the amount only continues to grow.

This image below shows the different Android versions in use over the past seven years:

And here are the different versions that were used in just 2019:

In 2019, you needed to test on at least 6 different devices with OS’s going from Lollipop 5.0 to Pie 9.

Fortunately, unlike Android developers, iOS developers’ lives are a bit easier as the iOS community consists of fewer devices and operating systems.

Testing client-side performance allows you to understand how the app behaves on a device and how it uses the resources that it shares with other apps.

Running tests for client side performance, you’ll also be able to find areas to improve that you do have control over (unlike the network), resulting in a better user experience.

It's critical that any mobile app development team integrate client-side performance testing into its test cycle.

There are several areas with different metrics you can gather to understand client side mobile app performance:

Device resource usage:

% CPU
Memory (Mb)
Data sent and received by the application (Kb)
% Battery usage

Rendering:

Render time (ms)
Frames per second (fps)
Render lag (janks)

Errors:

Exceptions
ANR (Application Not Responding) in Android

Transaction response time, which can also be measured from the user perspective but involves:

Server response times
Database response time
Network latency
Screen drawing time

Once you’ve got all of this information, you can uncover critical problems such as:

Memory leaks: when an app doesn’t free up memory well, it increases until it runs out of available memory and then crashes
Screen freezes: due to very high rendering times
Crashes: exceptions that jump in the app and cause it to suddenly close… bummer!

For native apps, there are profiling tools such as Android Studio and Instruments (for Android and iOS respectively).

These tools are of great help when debugging an app and finding the root of certain kinds of performance problems.

In addition to Android Studio and Instruments, you can also use Apptim, the tool we've launched at my startup to give entire development teams access to mobile app performance information and KPIs, where anyone can quickly and easily run tests.

With Apptim, you can test all of your native apps’ performance with just one tool, instead of one for Android and one for iOS and also compare performance over time.

Test mobile app performance as early as possible

Now, you may be wondering, “At what point in the development cycle should I start running client-side performance testing?”

If you ask me… start as soon as possible.

If you start testing early on, from the first version of your app, you’ll give yourself more time to understand how it behaves, get familiarized with client side metrics, and run these tests alongside functional ones in each sprint.

Another advantage of testing early is you can test on different devices at your disposal as well as understand how these metrics evolve over time.

It’s important to understand how the metrics change over time so you can identify what changes to the app alter its performance (find performance regressions) and have more time to remedy degradations.

This helps a developer understand how the decisions they make affect performance and ultimately, avoid disappointing users when they find that the app is running slower than usual.

I hope this post provided you an overview on how to test mobile app performance!

Have any questions? Leave a comment!