TL;DR
This article will give you an overview on how to build and run a Flutter cross-platform mobile application for the Android OS platform that is deployed and tested on several devices in the AWS Device Farm.
Setting the scene
When building and testing mobile applications and in case you want to assure that your mobile application is running correctly on different devices and operating systems, this can be very cumbersome and expensive over time. The efforts for various tasks of the device management like maintaining the physical infrastructure, patching and updating the devices can be very cost intensive for a company in case these tasks must be conducted by a developer or someone else in a business organization. Just imagine you have 15 or 20 different devices, that need to get patched manually. This will take a few hours in case everything works fine. The effort for maintenance will increase linearly, depending on the number of managed devices. And not only the different devices are important but also the combinations of devices and installed operating versions could let explode the test efforts dramatically.
AWS Device Farm to the rescue
But there is a solution. AWS Device Farm to the rescue. AWS offers with AWS Device Farm a solution that makes mobile testing a breeze. AWS Device Farm is a managed service on AWS that allows using a fleet of mobile devices and mobile environments from AWS via an on-demand model to either provision devices for manual or automated testing.
The good news, you can test AWS Device Farm and get the first 1000 minutes for free.
Prerequisites
What you will need for the sample is the following:
- Android Studio
- Android SDK
- Flutter (Version 3.0.5 or higher)
- Git
- AWS Account
- Some free space on your SSD ;-)
Flutter sample app
The Flutter sample app for the test is a simplistic login page with username and password that forwards to a welcome screen page. From there you can start a browser application with a search engine web page.
Testing with Appium
Appium is a common open-source test automation framework for use with native, hybrid and mobile web apps. It uses the Webdriver protocol (similar like Selenium) and can be used for testing iOS, Android and Windows apps.
Appium is one of the test frameworks which is directly supported by the AWS Device Farm.
For more information:
https://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-appium.html
With Appium you can use specific annotations like “AndroidFindBy” to identify elements on a screen of a mobile app, interact with these elements (clicking, entering text, ...) and checking the state of these elements (e. g. is shown or is enabled). This can be done with several programming languages, like Java, Kotlin, Python, JavaScript, etc.
Here some examples with Java.
// identifying elements with annotations
@AndroidFindBy(accessibility = "LoginText")
private MobileElement loginText;
// identifying elements programmatically
MobileElement loginText = driver.findElementByAccessibilityId("LoginText");
// perform a click
loginText.click()
// enter text
loginText.sendKeys("Hello");
// asking if element is shown or not
loginText.isDisplayed()
For writing Appium tests in Java you can also choose between different testing frameworks, like JUnit or TestNG.
Write Appium tests for AWS Device Farm
There is a sample app for the AWS Device Farm which illustrates what you need for you Appium tests on the AWS Device Farm.
See https://github.com/aws-samples/aws-device-farm-appium-tests-for-sample-app
One important step is to create a ZIP file with all compiled Appium tests, and their dependencies (Appium jars) needed to execute the tests. This ZIP file will be later uploaded in the AWS Device Farm together with the APP binary.
The recommended way to implement the tests is to make use of the page pattern, which creates a “shadowed” object of the mobile frontend page. The test can make use of the Appium features to identify Android application elements like the input fields and the button by using the @AndroidFindBy annotation with an XPath expression in the object tree and will provide that object as a MobileElement. In addition, you can now configure the possible actions of the page by using public methods to modify the contents of the MobileElements and trigger send actions on the button.
package com.alex.flutter_sample.appium.pages;
import io.appium.java_client.AppiumDriver;
import io.appium.java_client.MobileElement;
import io.appium.java_client.pagefactory.AndroidFindBy;
public class LoginPage extends BasePage {
@AndroidFindBy(xpath = "/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.view.View/android.view.View/android.view.View/android.view.View/android.widget.EditText[1]")
private MobileElement username;
@AndroidFindBy(xpath = "/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.view.View/android.view.View/android.view.View/android.view.View/android.widget.EditText[2]")
private MobileElement password;
@AndroidFindBy(accessibility = "Login")
private MobileElement loginButton;
public LoginPage(AppiumDriver<MobileElement> driver) {
super(driver);
}
@Override
public boolean isDisplayed() {
return loginButton.isDisplayed();
}
public void setUsername(String username) {
this.username.click();
wait(2000);
this.username.clear();
this.username.sendKeys(username);
}
public void setPassword(String password) {
this.password.click();
wait(2000);
this.password.clear();
this.password.sendKeys(password);
}
public void clickLogin() {
this.loginButton.click();
}
}
A sample Appium test could check if a user can go from the LoginPage to the SecondPage when authenticating with the correct credentials. To spin up this scenario, we would create an instance of the LoginPage and check, that the page is loaded successfully. Then the user input (username and password) is simulated and a click on the login button is triggered. This will cause a change of the page and we can assert that the SecondPage is displayed. In the end we could take a screenshot of the test run, that will be bundled with the test result of this test run and can be used for manual checks at a later point in time.
@Test
public void login_withCorrectCredentials_gotoSecondPage() throws InterruptedException {
// given
LoginPage loginPage = new LoginPage(driver);
assertThat(loginPage.isDisplayed(), is(true));
// when
loginPage.setUsername("Alex");
loginPage.setPassword("alex");
loginPage.clickLogin();
// then
assertThat(new SecondPage(driver).isDisplayed(), is(true));
takeScreenshot("secondPage.png");
}
Run automated Appium tests on AWS Device Farm
The following steps are needed to setup a project in AWS Device Farm to run Appium tests.
You need to create an AWS Device Farm project which is just an “organizational unit” containing a configuration of your application. So, you can simply start by creating a mobile application project.
You can then see the project in the overview of your mobile projects.
Within the project you can start the configuration of automated tests and start a new run. In this setup we want AWS Device Farm to spin up an Android application and run the Appium tests automatically once the device is ready. The other option here would be to provision a device for “remote access”. That would give you a “bare” device that you can use for various manual testing purposes.
In the project you have two options, what kind of project you want to test on the device. That could be a mobile application or a web application. In this guide we will focus on an Android mobile application that has been implemented with the Flutter framework. You need to upload an apk file (Android Package) and upload it AWS Device Farm. You can find your app-release.apk file in project-name/module-name/build/outputs/apk/ folder.
In the configuration section you have the option to choose between a standard environment setup and a custom environment setup. A custom setup can be very useful in case you have some special steps to perform before test execution. A typical use case for a custom environment could be the setting of a specific Appium version.
The next step is to select some devices that can be used to run the mobile application. You have the possibility to choose between more than 100 devices and you can group device to a device pool, that can be reused. I selected a list of different devices from different vendors like Samsung and Google. In general, the availability of Samsung devices is very good while other vendors like Huawei are not available. I recommend testing the application also on different versions of the Android operating systems in combination with different devices.
In case you want to load specific data on to the device, there is an upload possibility. In addition, you can specify the network settings of the device, the geo location for the test run. This can be useful in case you have scenarios where a geo location is relevant as you have a location-based service using a map or in case you want to test your application with a bad internet connection with a high latency or low bandwidth. For this case you could also create a network profile. When running a multi-market or multi-language application it could also be useful to have some variations on the locale settings. So, AWS Device Farm offers you a wide bandwidth of parameters ton change the desired state of the device.
After setting the device state you can start the automated test runs and will then see an overview of the automated test for your application. Some minutes later you can see the result of the test as a circle diagram, indicating the success rate of the executed tests.
When the test has been performed you can see the jobs, that have been run and can drill down into the details of the test run. Typically, it takes about 2-3 minutes until the execution of the tests starts, as this time is needed to select and prepare a device in the AWS Device Farm for you. After this preparation time your automated tests are executed on the specified device and after the execution, the device is shutdown. The result of the automated test run contains different artifacts like screenshots of the test workflow on the device, a short video of the workflow, the test specification log, the log output, and a very simplistic performance dashboard of the automated test run, giving a good indication of the CPU and memory usage during the test on the device.
Setup AWS Device Farm project from CLI
In case you do not want to do all the steps manually, it may be more efficient to make use of the AWS CLI which allows you to perform all the necessary steps from a shell script. Especially, when you want to use a CI/CD tool like Jenkins, Gitlab, and others, these scripted steps might be useful for reuse within some pipeline jobs or steps.
#! /bin/bash
echo "Starting automatic deployment of Alex Sample App with Tests"
# 1. Retrieve PROJECT_ARN
PROJECT_NAME = "Alex Sample Project"
PROFILE = "alex.dev"
REGION = "us-west-2"
PROJECT_ARN = aws devicefarm create-project --name $PROJECT_NAME --profile=PROFILE --region=$REGION | jq '.project.arn'
# 2. Get Upload ARN and URL for Android App
APP_FILE = "app-release.apk"
APP_TYPE = "ANDROID_APP"
APP_UPLOAD_JSON = aws devicefarm create-upload –project-arn $PROJECT_ARN –name $APP_FILE –type $APP_TYPE --profile=$PROFILE --region=$REGION | jq '.upload'
APP_UPLOAD_ARN = jq '.arn' $APP_UPLOAD_JSON
APP_UPLOAD_URL = jq '.url' $APP_UPLOAD_JSON
# 3. Upload the app
curl -T app-release.apk $APP_UPLOAD_URL
# 4. Get Upload ARN and URL for Test App
TEST_FILE = "tests.zip"
TEST_TYPE = "TEST_TYPE"
TEST_UPLOAD_JSON = aws devicefarm create-upload –project-arn $PROJECT_ARN –name $TEST_FILE –type $TEST_TYPE --profile=$PROFILE --region=$REGION | jq '.upload'
TEST_UPLOAD_ARN = jq '.arn' $TEST_UPLOAD_JSON
TEST_UPLOAD_URL = jq '.url' $TEST_UPLOAD_JSON
# 5. Upload the tests
curl -T tests.zip $TEST_UPLOAD_URL
# 6. Create device pool
DEVICE_POOL_NAME = "SampleDevicePool"
DEVICE_POOL_ARN = aws devicefarm create-device-pool –project-arn $PROJECT_ARN –name $DEVICE_POOL_NAME –rules '[{"attribute": "PLATFORM", "operator": "EQUALS", "value": ""ANDROID""}]' --profile=$PROFILE --region=$REGION | jq '.devicePool'
# 7. Schedule test run
SCHEDULE_RUN_NAME = "Sample-Test-Run"
TEST_RUN_RESULT = aws devicefarm schedule-run –project-arn $PROJECT_ARN –app-arn $APP_UPLOAD_ARN –device-pool-arn $DEVICE_POOL_ARN –name $SCHEDULE_RUN_NAME –test '{"type": "TEST_TYPE","testPackageArn":"' + $TEST_UPLOAD_ARN + '"}' --profile=$PROFILE --region=$REGION
echo $TEST_RUN_RESULT
Summary
This article has come to an end, and you now have a blueprint and hopefully a good feeling on how to test a Flutter application automatically on AWS Device Farm.
My experience with the AWS Device Farm and Android mobile applications was quite nice. The UI of AWS Device Farm is simple but contains the most important features to modify the state of the devices and supports the relevant testing frameworks. Most of the actual devices on the market are available within 2 to 6 weeks after their market release. My observation is that the support for some device manufacturers is really good (Samsung, Apple, Google), while some other manufacturers like Huawei are not widely supported. But in case you need some very specific devices you can always make use of the "private devices" features which is a BYOD model (bring your own device).
Top comments (0)