DEV Community: Rosário P. Fernandes

A GitHub Action to Build Modular Android Projects

Rosário P. Fernandes — Thu, 17 Sep 2020 20:01:36 +0000

My Workflow

As most Android Developers know, an Android Project may have many
different modules.

If you change one single module and submit a Pull Request, most CI workflows will try to build/test all the existing modules, which might take a lot of time, when in reality you only need to build the module you changed.

This GitHub Action aims to solve that problem. When you submit a Pull Request, it will only build the module you changed,or no module at all if you left them untouched (in case of documentation or configuration changes).

Real Project Showcase

Take a look at this project I forked: quickstart-android. It contains many different modules (admob, analytics, database, etc).

I made a change to the database module and sent a Pull Request:

refactor(database): make a demo change for the Github Actions Hackathon #11

rosariopfernandes posted on Sep 17, 2020

I'm hoping that this PR will only build the database module, since that's the only change I made.

View on GitHub

GitHub Actions built the database module only and finished in 4m 40s, as seen in the log.

Building the whole project generally takes around 17min, as seen in this log.

Submission Category:

Phone Friendly

Yaml File or Link to Code

thatfiredev / modular-android-action

GitHub Actions to selectively build PRs on multi-module Android Projects

Github Actions for Modular Android Projects

As most Android Developers know, an Android Project may have many different modules.

This GitHub Action aims to solve that problem. When you submit a Pull Request, it will only build the module you changed or no module at all if you left them untouched (in case of documentation or configuration changes).

Usage

Add it to your workflow YAML like this:

    - name: Build with Modular Action
      uses: rosariopfernandes/modular-android-action@v0.2.0
      with:
        for-each-module: 'assembleDebug'
        for-all-modules: 'check'

As you can see, it takes 2 arguments:

for-each-module - gradle task to run for each module.
for-all-modules - gradle…

View on GitHub

Additional Resources / Info

Inspired by the blog post "Selectively running Android modularized unit tests on your CI server" by Joe Birch.
I've previously written a blog post describing how I adapted Joe's code for GitHub Actions.
The Projects bellow are already using this workflow:
- firebase/quickstart-android
- firebase/snippets-android

Run Android modularized builds selectively on GitHub Actions

Rosário P. Fernandes — Thu, 09 Apr 2020 18:01:52 +0000

Header Photo by Brad Neathery on Unsplash

TL;DR: I've improved Android build times on GitHub Actions for Pull Requests from ~17min down to ~5min using this.

A few months ago I read this great post "Selectively running Android modularized unit tests on your CI server", where Joe Birch describes some of the tips and tricks he used to implement a CI Workflow that selectively runs unit tests on a modularized Android Project. This week I had some free time and decided to try implementing a similar workflow on a project that I've been contributing to (firebase/quickstart-android). So in this post I've decided to share the challenges I faced while doing it and some of the workarounds.

Before we get started, let me walk you through some of the basics of it all to make sure you're familiarized with the concepts described here. If this post's title was 100% clear for you, you can go ahead and skip to Current Scenario section.

The Basics

1. Modularized Android?

If you're an Android Developer, you (probably) already know that an Android Studio Project can be divided into modules. This allows developers to create different apps for the same Project. For example, if I'm building a "Health Tracking" app, I could split it into different modules:

app - this is usually created by default when you select "new Android App" on Android Studio. This module is the Android app for phones/tablets;
wear - let's suppose we also want our app to work on Android Wear devices. We can build that app in this separate module;
lite - let's say we have a different version of the app which uses less resources and is meant for Android devices with low-processing power.

If you're not yet familiar with Modules, I recommend checking the Android Documentation.

2. GitHub Actions?

Now before we get into this, you should first understand the concept of Continuous Integration. I really like this definition by Atlassian:

Continuous integration (CI) is the practice of automating the integration of code changes from multiple contributors into a single software project. The CI process is comprised of automatic tools that assert the new code’s correctness before integration.

Putting it in simple words: let's say we have a project stored on GitHub and many contributors. Before accepting any contribution, we want to make sure that this contribution will not break our code, and for that we can use Automatic CI Tools which will try to build the new code and then tell us if it breaks our current code or not.

Particularly, I've been working with tools such as Travis CI and CircleCI. But recently, GitHub introduced GitHub Actions - another automated CI tool. This came as great news for some developers, since this means they can have their source code and their CI on one and only platform: GitHub. No need to share the code with any other 3rd party CI tools.

Current Scenario

As I've mentioned in the beginning of the post, I've been contributing to firebase/quickstart-android. This Open Source project is basically a collection of Android sample apps that aim to demonstrate developers how to use the Firebase APIs. Because Firebase provides many different services, the project has been divided into modules (one for each Firebase Service).
At the time I wrote this post, it had 18 different modules and building the whole project on GitHub Actions would take around 17min (average).
So whenever someone sent a contribution to the project, they'd have to wait for around 17 minutes before they could find out whether their contribution breaks the project or not.

This may be "unfair" at times when someone has implemented changes in a single module and has to wait for GitHub Actions to build all the other unchanged modules. Or even worse: when someone doesn't change any code at all, because their changes were made on the project's documentation (E.g. README.md, LICENSE, etc), but will still have to wait for the project to build. And Joe's post tries to show how to reduce CI time in such cases.

NEED FOR SPEED!!!

So what Joe proposes is: instead of testing the whole project (done with gradlew testDebugUnitTest), we could simply test the changed modules only (with gradlew :moduleA:testDebugUnitTest :moduleB:testDebugUnitTest) which might be a lot faster.

Now in my scenario, the project doesn't really have any tests and what the CI does is basically check if the project builds successfully and if there are no lint issues(gradlew clean ktlint build).
For Pull Requests I've decided to only build the debug variant of the changed module (gradlew clean ktlint :module:assembleDebug :module:check).

Doing this should be easy, right? I could simply copy Joe's code. But as it turned out, it was not that straight forward. Mainly because he uses a different CI Environment (Bitrise) and some of his code was not running correctly on GitHub Actions.
So let's have a look at what my code looks like and I'll explain the changes I made.

1. Find out which modules were changed

In order to find the changed modules, he uses git diff with a while loop in a pipe to read the command's output. I've tested this on my local terminal (zsh) and it works fine. But GitHub Actions uses the bash terminal, and bash was not so happy with that pipe. So I had to rewrite the while loop in the main shell, as suggested here:

# build_pull_request.sh
# Get all the modules that were changed
while read line; do
  module_name=${line%%/*} # This gets the first word before '/'
  # Now we check if we haven't already added this module
  if [[ ${MODULES} != *"${module_name}"* ]]; then
    MODULES="${MODULES} ${module_name}" # string concat
  fi
done < <(git diff --name-only origin/$GITHUB_BASE_REF)

Notice that we're using an env variable from GitHub Actions: $GITHUB_BASE_REF - this contains the name of the branch where our Pull Request is supposed to be merged into (generally the master branch).

After pushing this, it still wouldn't work. It would show an error, saying it couldn't find the branch I had specified ($GITHUB_BASE_REF) and that's because GitHub Actions pulls our code to their CI machine using a shallow clone, therefore the branches were not being fetched. So in order to fix this, we have to unshallow and fetch the branches before running git diff:

# build_pull_request.sh
git fetch --unshallow
git fetch origin

2. Check if the changed files belong to any module

Now this is not very different from Joe's script, with a few exceptions, since I adapted it for my own scenario:

# build_pull_request.sh
# Get a list of all available gradle tasks
AVAILABLE_TASKS=$(./gradlew tasks --all)

# Check if that list contains the modules that were changed
build_commands=""
for module in $changed_modules
do
  if [[ $AVAILABLE_TASKS =~ $module":app:" ]]; then
    build_commands=${build_commands}" :"${module}":app:assembleDebug :"${module}":app:check"
  fi
done

# Build the Pull Request
eval "./gradlew clean ktlint ${build_commands}"

Notice that if no module was changed, the build_commands variable will be empty and CI will only run gradlew clean ktlint.
Now we can save our file as build_pull_request.sh and change it's permission so that it can be executed: chmod +x build_pull_request.sh.

How will GitHub Actions know whether it's a Pull Request?

That's the easiest part. We can add conditions to our workflow config:

# .github/workflows/android.yml
name: Android CI

on:
  - pull_request
  - push

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: set up JDK 1.8
      uses: actions/setup-java@v1
      with:
        java-version: 1.8
    - name: Build with Gradle (Pull Request)
      run: ./build_pull_request.sh
      if: github.event_name == 'pull_request'
    - name: Build with Gradle (Push)
      run: ./gradlew clean ktlint build
      if: github.event_name != 'pull_request'

And that's it! One git commit and a git push and you should see CI times faster than Lightning McQueen.

Final Considerations

After implementing this change, CI times in quickstart-android PRs dropped from ~17min to ~5min. While this may seem like it's only 12 minutes, it actually adds up to days, weeks or months on the long run. It also means that the CI Server will be freed quicker, allowing other contributions on different repositories of the same organization to use the CI.
The full script and workflow config file is available on Gist.
Please note that this code only works if all modules are independent. So if you have moduleA dependending on moduleB and you make changes to moduleB, then moduleA might break and not be tested.
If you're new to Continuous Integration, I'd also recommend reading about Continuous Delivery.

I hope this helps you improve build times in your CI Environment. If you plan on trying out anything I've mentioned here, leave a comment bellow or reach out to me on Twitter(@_rpfernandes) and I'll be happy to have a chat about it.

Using Kotlin Extension Functions and Coroutines with Firebase

Rosário P. Fernandes — Tue, 09 Apr 2019 08:58:53 +0000

Portuguese Version available on developingwith.firebaseapp.com

"The simpler the code, the faster you'll understand it." -
Paulo Enoque

Kotlin's simplicity and concision are the main reason why developers have been adopting this language for their work.
I must also remind you that, in May 2017, Google announced that Kotlin became an official language for Android App Development. Ever since then, the number of Kotlin developers has been increasing.

A year and a half after Kotlin became the official language for Android Development, Firebase added this language to their official documentation :

@Firebase code samples for Android are starting to get @kotlin samples to go with Java! Thanks to @daggerdwivedi and @_rpfernandes for the assists here. It's a tremendous effort overall, and it's great to have help from the community.https://t.co/PMlAViMf9F #AndroidDev pic.twitter.com/W386VlM3HC
— Doug Stevenson 🔥 (@CodingDoug) October 11, 2018

But I must agree with the popular saying "Better late than never". This change brought many improvements to the platform:

The Java code on the Firebase Android SDK was improved for better interoperability with Kotlin;
"Extension Functions" were created to make the use of the Android SDK more concise;
Some developers created libraries to improve the way Firebase is used with Kotlin, etc.

I decided to create a library to help people use Firebase in Kotlin, it was called fireXtensions (now deprecated) and it used to provide some Extension Functions for the Firebase Android SDK.

Kotlin Extension Functions

Putting it short, Extension Functions are a Kotlin feature that allows you to add new methods/functions to a class, even if this class was not created by you. No need to implement the class nor extend it.

After adding Kotlin to their Official Documentation, the Firebase Team has also added a few Extension Functions to their Android SDK (thus deprecating fireXtensions).

The first extensions were the libraries Common KTX and Firestore KTX.

Common KTX

The firebase-common-ktx module contains extension functions used to obtain the Firebase Instance. If you've ever worked with Firebase (either in Java or Kotlin), you're probably familiarized with the method/function FirebaseApp.getInstance(). If you use the firebase-common-ktx module, calling this method gets easier: Firebase.app.

In order to use this module, make sure you have the Kotlin Plugin 1.3.20 or higher on your build.gradle(project) file:

dependencies {
    classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:1.3.20"
}

And then add the following dependency to your build.gradle(app):

dependencies {
    // ... Other dependencies ...
    implementation 'com.google.firebase:firebase-common-ktx:16.1.0'
}

See the full list of available extension functions.

Firestore KTX

As the name suggests, this module contains extensions that simplify the way you use the Firestore Android SDK. This module can be added to your project the same way we did with Common KTX, but using this dependency instead:

dependencies {
    // ... Other dependencies ...
    implementation 'com.google.firebase:firebase-firestore-ktx:18.2.0'
}

See the full list of available extension functions.

Note that when using the firebase-firestore-ktx dependency, you no longer need to use com.google.firebase:firebase-firestore.

Although these are currently the only available modules, I'm pretty sure more modules will be added to their SDK. I'll update this post every time a new module is added.

Kotlin Coroutines

As you might already know, Firebase APIs are asynchronous, forcing us to use listeners in our code to read data from our database. And programmers often try to (wrongly) read data without listeners, or try to use the data outside those listeners, like this:

Example 1: Read all users stored in Firestore.

// What we would like to do
// (PS: This code doesn't work)
val db = FirebaseFirestore.getInstance()
var users = db.collection("users").get()
updateUI(users)

// (This code doesn't work either)
// What some people do (wrongly)
var users: List<User>()? = null
usersRef.get().addOnSuccessListener { querySnapshot ->
    users = querySnapshot.toObjects(User::class.java)
}
if (users == null) {
    displayError()
    // displayError() will always be called because
    // the users list is loaded asynchronously.
    // The if is executed while the list hasn't been loaded yet.
} else {
    updateUI(users)
}



// What they should do:
usersRef.get()
    .addOnSuccessListener { querySnapshot ->
        val users = querySnapshot.toObjects(Userr::class.java)
        updateUI(users)
        // The method is now being called after
        // loading the users list.
    }.addOnFailureListener { e ->
        displayError()
    }

Looking at the code above, you might think there's nothing wrong with listeners, because everything makes sense and even keeps the code organized. But what if you need to read data from different collections and merge the results before displaying on the app UI? This would force us to use nested listeners, which is not so easy to read, as we can see bellow:

Example 2: Load John's profile and his list of friends (this list is stored under a different collection).

// Allow me to send a shot out to all
// JavaScript Developers who have been to callback hell
usersRef.document("john").get().addOnSuccessListener { querySnapshot ->
            val johnUser = querySnapshot.toObject(User::class.java)

            friendsRef.get().addOnSuccessListener { friendSnapshot ->
                val friends = friendSnapshot.toObjects(Friend::class.java)
                showProfileAndFriends(johnUser, friends)
            }.addOnFailureListener {
                displayError()
            }

        }.addOnFailureListener { e ->
            displayError()
        }

Coroutines have come to change that. They allow you to write asynchronous code as if it was synchronous, making it more concise and easier to read.
In order to use Coroutines in your Android Project, make sure you're using version 1.3.x or higher of the Kotlin Plugin, and then add the following dependencies to your build.gradle(app) file:

dependencies {
    // ... Other Dependencies ...
    implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-android:1.1.1'
    implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-play-services:1.1.1'
}

As of the time this was posted, 1.1.1 was the latest version. You can check the maven repository for the latest version of kotlinx-coroutines-play-services.

Now, if we used Coroutines in our first example, it would become:

try {
    val snapshot = usersRef.get().await()
    val users = snapshot.toObjects(User::class.java)
    updateUI(users)
} catch (e: FirebaseFirestoreException) {
    displayError()
}

It resembles our "What we would like to do" code from example 1, doesn't it?

Now our second example:

try {
    val querySnapshot = usersRef.document("john").get().await()
    val johnUser = querySnapshot.toObject(User::class.java)

    val friendSnapshot = friendsRef.get().await()
    val friends = friendSnapshot.toObjects(Friend::class.java)
    showProfileAndFriends(johnUser, friends)
} catch (e: FirebaseFirestoreException) {
    displayError()
}

The code now looks synchronous and easier to read, right?

And that's all. I hope you found these Kotlin features useful and I hope that they'll improve your productivity when developing firebase apps.

If you have any doubts or suggestions, please leave it on the comments bellow.

If you're trying to use the Kotlin Extensions or Coroutines and ran into a problem, you can post it on StackOverflow explaining what you did and what error you ran into. I'm sure you'll find help (either from me or someone from the Firebase Community).

Exploring Firebase Bill of Materials (BoM)

Rosário P. Fernandes — Tue, 19 Mar 2019 18:08:27 +0000

Portuguese version available in developingwith.firebaseapp.com

Once upon a time, in a not so distant past, one could easily add the Firebase SDK to their Android Project not worrying about different versions for each module. Every module (core, auth, firestore, storage, etc) would use the same version, so our build.gradle (app) file would look like this:

    implementation 'com.google.firebase:firebase-core:12.0.0'
    implementation 'com.google.firebase:firebase-firestore:12.0.0'
    implementation 'com.google.firebase:firebase-auth:12.0.0'

Or even better, we could declare the firebase version in the build.gradle (project):

buildscript {
    ext.firebase_version = '12.0.0'
    // ... Other configurations
}

and then use it in the build.gradle (app):

    implementation 'com.google.firebase:firebase-core:$firebase_version'
    implementation 'com.google.firebase:firebase-firestore:$firebase_version'
    implementation 'com.google.firebase:firebase-auth:$firebase_version'

But the happiness in this Kingdom didn't last long. After some time, some modules started getting updates more frequently than others, forcing the sdk developers to bump their versions and thus having different versions for each module.

Well, if you wish you had lived in that fairy-tale, I have good news for you! Thanks to the feature Bills of Material (BoM) introduced in Gradle 5.0 you can now use a single version for all the Firebase Android SDK modules.

Bills of Material (BoM) allows the SDK developers to specify (in a bill) which module versions are known to work with each other and then set a version to this bill.

So every time the Firebase Team publishes a new bill, you can be sure that you'll be using the latest versions(at time the BoM was published) of every module.

You can find the latest BoM version in the Firebase Android Release Notes.

Getting Started

Upgrading to Gradle 5.0

Since the BoM feature was introduced in gradle 5.0, you'll probably want to upgrade your gradle version (if you haven't already).

To upgrade to gradle 5.0 and add Firebase BoM to your Android Project, follow these steps:

Upgrade to version 4.10 first and check for any errors deprecation warnings. It's better to fix all errors and warnings before upgrading to a newer version.
Upgrade to version 5.0 using this command in your project root folder:

./gradlew wrapper --gradle-version 5.0

Sync your project and add the dependencies to your gradle file:

dependencies {
    // Import the BoM
    implementation platform('com.google.firebase:firebase-bom:16.0.0')

    // Import the modules not minding their versions. Hooray!
    implementation 'com.google.firebase:firebase-core'
    implementation 'com.google.firebase:firebase-firestore'
    implementation 'com.google.firebase:firebase-auth'

    // ... Other dependencies
}

One more project sync and that's it! You no longer need to worry about different versions for each module.

With Gradle 4.6.x or higher

If you're not planning to upgrade to Gradle 5.0, you can still take advantage of this feature, because it has been in experimental mode since gradle 4.6.x, you just need to enable it:

Open your project's settings.gradle file and add the line enableFeaturePreview('IMPROVED_POM_SUPPORT'):

include ':app'
enableFeaturePreview('IMPROVED_POM_SUPPORT')

Sync your project and add the dependencies to your gradle file:

dependencies {
    // Import the BoM
    implementation 'com.google.firebase:firebase-bom:16.0.0'

    // Import the modules not minding their versions. Hooray!
    implementation 'com.google.firebase:firebase-core'
    implementation 'com.google.firebase:firebase-firestore'
    implementation 'com.google.firebase:firebase-auth'

    // ... Other dependencies
}

One more project sync and that's it!

Notes

Please note that:

If you specify a module version and you use a Firebase BoM, then the specified module version overrides the version contained in the BoM.
Firebase BoM is still an experimental feature, which means there might be some issues. If you want to report an issue or provide feedback, consider doing so in the Firebase Android SDK Issue Tracker.