DEV Community: Ilya Vorobiev

Initializing and scheduling app size collection job

Ilya Vorobiev — Sun, 12 Mar 2023 11:04:25 +0000

This is the third and the last part in the series of articles describing how to calculate the size of an uncompressed app on disk and report it. This part covers how to stitch all the components implemented in the previous parts. Make sure you read the first and the second parts before proceeding to this one.

One of the requirements, I mentioned in the first part was “The solution shouldn’t impair the user experience especially when the app is in the foreground”. I will use jetpack WorkManager to make sure this requirement is fulfilled and to initialise the system.

Before defining the constraint, let’s stitch all the components together. Our Worker will look like that:

class AppSizeWorker(private val appContext: Context, params: WorkerParameters) :
   CoroutineWorker(appContext, params) {
   override suspend fun doWork(): Result {
       // In debug can return LogCatAppsizeTransport
       // In prod will return the transport that sends report to the backend
       val transport: AppSizeTransport = provideTransport()
       val appSizeCollector = AppSizeCollector(appContext, transport)
       // Should fetch the config from backend or cache
       val appSizeCollectorConfig: AppSizeCollectorConfig = fetchAppSizeCollectorConfig()
       // Will try to scan disk and report the result. The function is main-safe
       val result = appSizeCollector.traverseAndCollect(appSizeCollectorConfig)

       return if (result.isSuccess) {
           Result.success()
       } else if (runAttemptCount < 3) {
           Result.retry()
       } else {
           Result.failure()
       }
   }
}

Note: since we are retrying if the result of the collection was not successful, we will need to provide a backoff policy when creating the worker.

This job should always be done in the background and not urgent to execute immediately. That’s why the natural limitations for the worker will be that the job is periodic and device should be idle. A daily snapshot of the disk should be representative enough to analyse, however, the frequency might be lower if on average your users use the app less often.

I would also add the requirement that the device should be charging. Given that phone anyway needs to be charged once a day on average, it is unlikely we will miss reports due to that constraint, but we will keep the battery life longer, by not introducing heavy background jobs.

Overall the worker request will look like this:

val appSizeCalculateRequest = PeriodicWorkRequestBuilder<AppSizeWorker>(Duration.ofDays(1))
   .setConstraints(Constraints(
       requiresCharging = true,
       requiresDeviceIdle = true,
   ))
   .setBackoffCriteria(BackoffPolicy.EXPONENTIAL, Duration.ofHours(1))
   .build()
WorkManager.getInstance(this).enqueue(appSizeCalculateRequest)

In the end

Now, when we have a part that collects and calculates the size of our application, we can use the report it generates, to analyse, protect and reduce it.

There are some additional steps needed to use the system to full benefit, e.g. you need to save results to a file instead of log to analyse the result with the end-to-end tests, or you need to implement a network layer to report the analysis to the backend. I will leave these topics out of the scope of this series but feel free to reach out if you want to know more about how to make it work end to end.

How to calculate directory size on Android?

Ilya Vorobiev — Sun, 12 Mar 2023 11:04:14 +0000

This is the second part in the series of articles describing how to calculate the size of an uncompressed app on disk and report it. This part covers the implementation of various components, mentioned in the first part.

Schema of the report and Transport

I will stick to a simple structure: the report will be a map, where a path will be the key and its size will be the value. The map will be flat, meaning that path to a parent directory and paths to all its children will just be different keys of the same map.

It means that transport will take the map of paths and sizes as input and will log it wherever we need it. So the interface for transport will look like this:

interface AppSizeTransport {
   suspend fun sendAppSizeCollection(appSizeCollection: Map<String, Long>): Result<Unit>
}

Note: I use suspend function since transport will most likely report the metrics to backend, and hence needs to do that asynchronously. For that same reason, it returns results in the end.

If the report fails, the system should retry the attempt, however in that case we may need to run AppSizeCollector again, as the state of the filesystem may change.

Configuration of the AppSizeCollector

Filesystem is not under our control, which means there may be thousands of files created. If we are going to send this information to the backend from hundreds of thousands of devices, we may need to pay a lot for storage. That’s why the system needs a way to control how much data it sends.

There are a few metrics that we 100% need:

Total size of the app
Size of each top-level directory (e.g. caches, files, etc) to understand what impacts size the most

The number of the top-level directories is limited and fixed (apart from the data dir contents, but let’s hope it is not used actively in your app) so it is safe to report those. Deeper in the file tree the number of files is almost unbound and a limit on how deep to report must be introduced.

Sometimes a particular path needs to be investigated, or maybe there is an important directory somewhere deep, that must be monitored. For that, a list of exceptions from the limit is needed.

With all these requirements the configuration of the system should look something like that:

data class AppSizeCollectorConfig(
   val importantDirsLimit: Map<String, Int>, // List of paths that must be reported, and how deep
   val globalLimit:Int = 3, // General limit on how deep we will report
)

Calculating application size

The main component of the system is the AppSizeCollector, which will scan the disk, calculate the size of the app and then pass the result to the transport.

Given that, the dependencies of the class are quite clear: we need a context instance to resolve top-level paths and the transport object. I will also add CoroutineDispatcher to the constructor in order to provide concurrent IO operation and make the overall class Main-safe.

class AppSizeCollector(
   private val context: Context,
   private val transport: AppSizeTransport,
   private val dispatcher: CoroutineDispatcher = Dispatchers.IO
)

The main method of this class should get the config as an input and return the result of the operation. For now, I will not add specific errors support and will only tell if the operation was successful or not, however better type in generic will improve the system’s error handling and debuggability.

suspend fun traverseAndCollect(config: AppSizeCollectorConfig): Result<Unit>

As was described previously, there are 6 top-level directories that we are interested in, however, it is a bit more complex. There may be multiple different external directories associated with the app. Furthermore, the internal data dir, apart from the rest, contains the internal cache and internal files directories. The full set of top-level directories that we would want to traverse will look like that:

val dirsToTraverse = (context.externalCacheDirs.filterNotNull() +
       context.getExternalFilesDirs(null).filterNotNull() +
       context.externalMediaDirs.filterNotNull() +
       context.filesDir +
       context.cacheDir +
       (context.filesDir.parentFile?.listFiles() ?: emptyArray<File>())
       ).toSet()

Once we have a list of the directories we will need to iterate through them and calculate the size, however, there might be a few caveats.

Calculating the size of files and directories

If your app supports the API level 26 and higher the task is much easier, you can use Files.walk. But if you have to support older versions, there are a few edge cases that must be accounted for.

The main difficulty comes from the fact that the calling length() method on a directory will not return the total size of its contents. This forces us to recursively traverse the file tree and calculate the size.

While we are iterating recursively we want to avoid scanning symbolic links, as this may impose a risk of stucking in the loop or calculating the same directory twice. Instead, we will detect if the current file is a symlink and calculate the size of the link itself, instead of its contents.

To detect if the file is a symlink, I will use this utility function:

private fun File.isSymLink(): Boolean {
   val canon: File = if (parent == null) {
       this
   } else {
       val canonDir: File = parentFile.canonicalFile
       File(canonDir, name)
   }
   return canon.canonicalFile != canon.absoluteFile
}

It recalculates the file path from the canonical path of its parent (which is the path when all the symlinks are resolved) and compares it with the current path.

If the current file that we are scanning is neither a symlink nor a directory, this is the end of our recursion and we need to return its size and add it to the hashmap, which we will then pass to our transport.

var fileSize = dir.getSizeOnDisk()
if (dir.isFile) {
   // Check if current level of the recursion is within the limit defined in the config
   if (currentLevel < limit) {
       dirSizes[dir.path] = fileSize
   }
   return fileSize
}

currentLevel and limit will be passed as arguments to the recursive function, and we will talk a bit later about its signature. You might have noticed a weird function getSizeOnDisk which is not a part of the Java File API. What does it do?

How much disk space does the file use?

Filesystems usually don’t operate with bytes directly, but write and read in small chunks of information called blocks. Block has a fixed size and no file in the filesystem can be less than the block size. So if your filesystem has a block size of 4KB, even a file with a size of 1B will take 4KB of actual space. Note that call to file.length() will return 1 in this case, since it is the real length of a file. That is also a reason why empty directory may use 4KB of space since it is the minimal amount of bytes required to store the directory entry. So since we are interested in the disk space, we can’t use file.length() to calculate the size.

Android provides different ways to find the block size: you may look at StatFs class, for example. I will use Os.lstat which is equal to linux lstat command. Call to lstat will give me the block size and I will be able to derive how much disk space will the certain file take

private fun File.getSizeOnDisk(): Long {
   val lstatData = Os.lstat(canonicalPath)
   return (ceil(lstatData.st_size.toDouble() / lstatData.st_blksize) * lstatData.st_blksize).roundToLong()
}

Recursion for directories

Now we are able to get the size of the file and stop recursion if we encountered a symlink. The last part is to calculate the size of the directory, which should be trivial now. However, we haven’t yet defined the signature of the recursive function, so let’s do it!

private suspend fun calculateDirSize(
   dir: File,
   currentLevel: Int,
   limit: Int,
   dirSizes: HashMap<String, Long>,
   config: AppSizeCollectorConfig
): Long

Where dir is the current file to look at, currentLevel is the current depth of the recursion, and limit should verify whether we need to report the dir or use it only to calculate towards the top level directory size, dirSizes is the map that we are going to pass to the transport and config is needed to access exceptional paths.

Here is the chunk responsible for directory traversal:

dir.listFiles()?.forEach { file ->
   val nextLimit = config.importantDirsLimit[dir.path] ?: config.globalLimit
   fileSize += calculateDirSize(file, currentLevel + 1, nextLimit, dirSizes, config)
}
if (currentLevel < limit) {
   dirSizes[dir.path] = fileSize
}
return fileSize

Now as this introduces blocking IO and I defined the function as suspend I must make it Main-safe, so the resulting function will look like this:

private suspend fun calculateDirSize(
   dir: File,
   currentLevel: Int,
   limit: Int,
   dirSizes: HashMap<String, Long>,
   config: AppSizeCollectorConfig
): Long = withContext(dispatcher) {
   if (dir.isSymLink()) {
       return@withContext dir.getSizeOnDisk()
   }
   var fileSize = dir.getSizeOnDisk()
   if (dir.isFile) {
       // Check if current level of the recursion is within the limit defined in the config
       if (currentLevel < limit) {
           dirSizes[dir.path] = fileSize
       }
       return@withContext fileSize
   }
   dir.listFiles()?.forEach { file ->
       val nextLimit = config.importantDirsLimit[dir.path] ?: config.globalLimit
       fileSize += calculateDirSize(file, currentLevel + 1, nextLimit, dirSizes, config)
   }

   if (currentLevel < limit) {
       dirSizes[dir.path] = fileSize
   }
   fileSize
}

And our main function will turn into this:

suspend fun traverseAndCollect(config: AppSizeCollectorConfig): Result<Unit> =
   withContext(dispatcher) {
       val dirsToTraverse = (context.externalCacheDirs.filterNotNull() +
               context.getExternalFilesDirs(null).filterNotNull() +
               context.externalMediaDirs.filterNotNull() +
               context.filesDir +
               context.cacheDir +
               (context.filesDir.parentFile?.listFiles() ?: emptyArray<File>())
               ).toSet()
       val dirSizes = hashMapOf<String, Long>()
       var totalAppSize: AtomicLong = AtomicLong(0L)

       dirsToTraverse.map { dir ->
           launch {
               val limit = config.importantDirsLimit[dir.path] ?: config.globalLimit
               totalAppSize.addAndGet(calculateDirSize(dir, 0, limit, dirSizes, config))
           }
       }.joinAll()
       dirSizes["/"] = totalAppSize.get()
       transport.sendAppSizeCollection(dirSizes)
   }

Note, that I’ve added some concurrency here, which may speed up the overall process. Since coroutines are very light, this may be added to the calculateDirSize as well, but I’ll leave it outside the scope of this article.

I also use “/” as a representation of the overall app size.
Making LogcatTrasport implementation
I will not cover the backend reporting part but will leave the debug transport here that just prints the results to Log for you to analyze and debug the system.

const val TRANSPORT_TAG = "APP_SIZE"

class LogCatAppsizeTransport : AppSizeTransport {
   override suspend fun sendAppSizeCollection(appSizeCollection: Map<String, Long>): Result<Unit> {
       Log.d(TRANSPORT_TAG, "App size collection is ${appSizeCollection.size}")
       appSizeCollection.entries.forEach { entry ->
           Log.d(TRANSPORT_TAG, "Path ${entry.key} is ${entry.value} bytes")
       }
       return Result.success(Unit)
   }
}

This wraps the second part of the Calculating your App’s size article. In the last part, I will show how to stitch everything together.

Calculating Android App Size on disk: Overview

Ilya Vorobiev — Sun, 12 Mar 2023 11:03:59 +0000

This is the first part in the series of articles describing how to calculate the size of an uncompressed app on disk and report it. This part covers motivation and high-level architecture of the system.

Why is it important?

As it was said by Google “App size is one of the biggest factors that can affect your app’s install and uninstall metrics”. While there is a lot of discussion about why the compressed binary size is important and how it affects the app installs, lack of free storage space is often the reason why people uninstall the apps.

That means there are at least two reasons why we should care about the installed app size and how much we save on disk:

To not be at the top list of apps occupying too much space
To not bring the user to the state, where they need to decide what to delete

Now, the Google Play Console gives some insight into the uncompressed app size but it might not be granular enough and doesn’t show everything that is stored on disk. Building your own monitoring of what you store will give you a full insight into where the most impactful directories are and how to reduce the chance that users will uninstall your app.

Data collection requirements

There are a few requirements that we want to impose on the app size monitoring solution

It should calculate how much space the app occupies on the disk. This is the system’s primary goal - we want to see how much space our user will gain by deleting the app, and we want to minimize this metric.
It should calculate how much space important folders take (e.g. caches, databases or unpacked code). The number for the overall app is good, but the goal is to be able to identify where the most impact comes from and to identify opportunities. For that we need to understand which folders contribute the most.
The solution should be scalable with a number of files and nesting growing over time. We could’ve just reported all files related to our app, but with the app’s and users' engagement growth, we might encounter weird edge cases with very deep nesting or just a huge amount of files.
The solution shouldn’t impair the user experience especially when the app is in the foreground. Traversing the file system may impose a lot of IO work and compete with the rest of the app for resources. The job needs to be executed when the user is not actively interacting with the app and preferably when the phone is charging.
The solution shouldn’t violate any user’s privacy preferences. Even though I will not cover this in detail in this series of articles, always make sure your users are aware that you collect the usage data and they should have the option to opt out.

What is counted towards the app size?

Basically, everything that gets deleted together with your app.

Internal cache
Internal files
Internal data
External cache
External files
External media (may not be needed, in case you switched to MediaStore API)

Note: anything saved to external storage, not related to your app (such as other app’s external folders or shared media folder) will not be counted towards your app size and will not be deleted with your app. With this in mind, I don’t think it impacts the user’s decision to remove your app in particular, that’s why I’ll not include it in the calculation. But keeping an eye on how you utilise the shared media space is still a good idea, since by overusing it you still push the user towards the space limit and closer to the decision to delete something from their device.

High-level architecture of the solution

The monitoring system will consist of multiple components so that it can gain some flexibility: e.g. be controlled from the backend or report to various systems.

AppSizeCollector is the class that scans the filesystem and calculates the sizes of all the folders on the device.

AppSizeCollectorConfig is settings for the collector. It may contain a map of how deep to go in the recursion for certain folders.

AppSizeCollectionScheduler will be responsible to schedule the AppSizeCollector work once a day and make sure the job is launched in the background with minimal impact on user experience.

AppSizeReportTransport is a component responsible for sending information to a consumer (us). It may send the collected info to a backend endpoint, but for demo purposes, we will just output this into logcat.

AppSizeReportSchema is a schema of the final report.

I will dive deeper into the implementation of these components in the next part of this series and stitch everything together with a scheduler job in the third part.