Pablo Baxter

Posted on Oct 30, 2023

"Too Many Open Files" - Debugging Leaking Files on the Gradle Daemon

#gradle #java #kotlin

Banner image created with Dall-E using the prompt "elephant surrounded by files"

Preface

The Too many open files issue can be ignored by increasing the file descriptor limit, but there is no magic bullet to fix the error that may have brought you here. It's going to take some debugging and it is tedious. This article isn't to show you how to increase the file descriptor limit on Linux, or workaround the recent MacOS issue with increasing the file descriptor limit (see https://developer.apple.com/forums//thread/735798). There is no snippet of code that will fix this Gradle issue glaring at you. We'll be looking at how to identify file leaks on the Gradle daemon with the hope that it will provide you with the skills and tools to debug (and possibly fix) your build.

Let's Talk About Gradle

The Daemon

The Gradle build tool uses a long-lived daemon process that performs all Gradle builds to cache information in memory about previous builds, making subsequent builds faster. However, this also allows for open file descriptors to persist between builds, with subsequent builds potentially opening new file descriptors. Eventually this can lead to the dreaded Too many open files exception, which provides basically zero information about what file has leaked or where the leak is occurring.

Due to the nature of the daemon being long-lived, detecting file leaks can be problematic, as you'll be looking for an errant open file in a collection of other validly open files. However, there are some techniques that can be used to find these errant open files, so long as we have a basic understanding of the Gradle build lifecycle.

Build Lifecycle

Gradle has three distinct build phases:

Initialization: Evaluates the settings.gradle file.
Configuration: Evaluates the build.gradle files of projects.
Execution: Performs a run of the Gradle tasks.

Every phase requires the previous to occur and each of these phases can be triggered in different ways via IntelliJ or the command line. For the purposes of this article we'll only be using IntelliJ to trigger each of these phases.

The Tools

File Leak Detector

The tool I found effective for detecting these file leaks is the File Leak Detector by Jenkins (jenkinsci/lib-file-leak-detector). It is a Java Agent that can be applied via JVM arguments or attached after the JVM app has started. This library instruments several Java APIs to track all open file descriptors and either provides a simple HTTP server to show all currently open files or dumps them after the JVM has exited (which isn't helpful in our case, since Gradle is long-lived).
The output from this tool is very noisy. Each file descriptor, the thread it was opened in, time/date, and stacktrace are listed by default. For example:

#885 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:48 PDT 2023
 at java.base/java.nio.file.Files.newDirectoryStream(Files.java:482)
 at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:301)
 at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:374)
 at java.base/java.nio.file.Files.walkFileTree(Files.java:2844)
 at org.gradle.internal.snapshot.impl.DirectorySnapshotter.snapshot(DirectorySnapshotter.java:119)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.lambda$snapshot$8(DefaultFileSystemAccess.java:155)
 at org.gradle.internal.vfs.impl.AbstractVirtualFileSystem.store(AbstractVirtualFileSystem.java:84)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.snapshot(DefaultFileSystemAccess.java:145)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.lambda$read$0(DefaultFileSystemAccess.java:82)
 at java.base/java.util.Optional.orElseGet(Optional.java:364)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.lambda$readSnapshotFromLocation$9(DefaultFileSystemAccess.java:187)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess$StripedProducerGuard.guardByKey(DefaultFileSystemAccess.java:221)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.lambda$readSnapshotFromLocation$10(DefaultFileSystemAccess.java:184)
 at java.base/java.util.Optional.orElseGet(Optional.java:364)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.readSnapshotFromLocation(DefaultFileSystemAccess.java:184)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.readSnapshotFromLocation(DefaultFileSystemAccess.java:169)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.read(DefaultFileSystemAccess.java:82)
 (... Stacktrace manually collapsed for brevity...)
 at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:109)
 at org.gradle.internal.operations.DefaultBuildOperationQueue$WorkerRunnable.runBatch(DefaultBuildOperationQueue.java:224)
 at org.gradle.internal.operations.DefaultBuildOperationQueue$WorkerRunnable.run(DefaultBuildOperationQueue.java:192)
 at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
 at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
 at java.base/java.lang.Thread.run(Thread.java:1589)

Although descriptive, a single IDE sync can have upwards of 1k files opened with each file having a log entry as shown above. Thankfully, there is another tool that parses these raw logs and matches file descriptors with similar stacktraces.

File Leak Postprocessor

We can use the File Leak Postprocessor (centic9/file-leak-postprocess) to collect file descriptors with similar stacktraces and batch them as such:

#885 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:48 PDT 2023
#886 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#884 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2 by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:48 PDT 2023
#888 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#889 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main/java by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#892 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main/java/org/gradle/process by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#893 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main/java/org/gradle/process/internal by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#894 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main/java/org/gradle/process/internal/util by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#890 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main/java/org by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#883 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:48 PDT 2023
#887 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#891 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main/java/org/gradle by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
 at java.base/java.nio.file.Files.newDirectoryStream(Files.java:482)
 at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:301)
 at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:374)
 at java.base/java.nio.file.Files.walkFileTree(Files.java:2844)
 ...
 at java.base/java.util.Optional.map(Optional.java:260)
 ...
 at org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess.withFileLock(LockOnDemandCrossProcessCacheAccess.java:90)
 at org.gradle.cache.internal.DefaultCacheCoordinator.withFileLock(DefaultCacheCoordinator.java:219)
 at org.gradle.cache.internal.DefaultPersistentDirectoryStore.withFileLock(DefaultPersistentDirectoryStore.java:188)
 at org.gradle.cache.internal.DefaultCacheFactory$RHunting File LeakseferenceTrackingCache.withFileLock(DefaultCacheFactory.java:214)
 ...
 at org.gradle.cache.Cache.lambda$get$0(Cache.java:31)
 at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
 at org.gradle.cache.ManualEvictionInMemoryCache.get(ManualEvictionInMemoryCache.java:30)
 at org.gradle.cache.internal.DefaultCrossBuildInMemoryCacheFactory$CrossBuildCacheRetainingDataFromPreviousBuild.get(DefaultCrossBuildInMemoryCacheFactory.java:255)
 at org.gradle.cache.Cache.get(Cache.java:31)
 ...

As shown above, the postprocessor tool collapses the stacktraces of many of these file descriptors and batches the ones that have similar traces. This makes it easier to analyze a diff of the processed outputs.

Let's Learn Some Debugging Skills

I have created a simple project (pablobaxter/file-leak-example) which will be used for the rest of this article to showcase how to find file leaks that may occur either in a build script directly or within the plugin of another library. These approaches can be applied to any Gradle project.

Attaching the File Leak Detector

One of the first steps to detecting leaks is by using a... detector. Thankfully, one is available here: https://repo.jenkins-ci.org/releases/org/kohsuke/file-leak-detector/

Regardless of what version you choose, you'll want to download the file-leak-detector-<VERSION>-jar-with-dependencies.jar file, and store it somewhere that is accessible.

Next, you'll need to attach the Java agent from this Jar to the Gradle daemon of the project. This can be done by adding -javaagent: to the org.gradle.jvmargs parameter of the gradle.properties file.

org.gradle.jvmargs=-Xmx1536m -javaagent:/path/to/file-leak-detector-<VERSION>-jar-with-dependencies.jar=http=19999,strong

In this example, we also add in the arguments to start the HTTP server (http=19999) and to not let the JVM garbage collector clean up any open file descriptors (strong). Once this property is added, kill any currently running Gradle daemon and re-sync the project on IntelliJ. This will start a new Gradle daemon and http://localhost:19999 will now start displaying open file descriptors or crashing the web browser if it's too much, but this isn't an issue for us. This output is what we will use for postprocessing. Once the sync has completed, save a copy of this output as a text file, as it will be used to create the baseline of the open file descriptors for the Gradle daemon.

At this point we have accomplished the following:

Instrumented the Gradle daemon with the File Leak Detector.
Started a simple web service that shows all currently open files on the Gradle daemon.
Stored a raw baseline of all open files to compare against.

Now we can start our search for leaking files.

Establishing the Baseline

So far, by syncing Gradle via IntelliJ, we have basically only run the Initialization and Configuration phases of the Gradle build. This means the settings.gradle and build.gradle files have been parsed and evaluated.

Now, let's do some postprocessing of the baseline output. The steps are listed in the centic9/file-leak-postprocess repository, except you will also store the baseline postprocess output somewhere that is easily accessible. With this, we finally have our true baseline file that we will use to diff against. We can now start looking for file leaks.

Hunting File Leaks

With the baseline postprocess file, we can now trigger the Gradle sync on IntelliJ a number of times. I typically start with 5 syncs, but any number is fine, so long as a count of the runs is kept. You'll notice these syncs are slightly faster, which is expected. Wait for the syncs to complete before continuing with the next.

If this is done on the example project, you may notice that the X descriptors are open line on the generated webpage (http://localhost:19999) has increased (probably not by a whole lot), but this may differ if you are testing on a different project. Let's store the new output, and run the postprocess job on it. Now if we look at the diff between the baseline and the new postprocess output, we'll notice a few things here.

On the example project, the first and most obvious difference is that we have multiple mycustom.properties files open:

1139a1140,1261
> #898 /home/pablo/Development/file-leak-example/mycustom.properties by thread:Daemon worker Thread 2 on Tue Oct 17 14:26:01 PDT 2023
> #878 /home/pablo/Development/file-leak-example/mycustom.properties by thread:Daemon worker Thread 2 on Tue Oct 17 14:25:57 PDT 2023
> #888 /home/pablo/Development/file-leak-example/mycustom.properties by thread:Daemon worker Thread 2 on Tue Oct 17 14:25:59 PDT 2023
> #883 /home/pablo/Development/file-leak-example/mycustom.properties by thread:Daemon worker Thread 2 on Tue Oct 17 14:25:58 PDT 2023
> #893 /home/pablo/Development/file-leak-example/mycustom.properties by thread:Daemon worker Thread 2 on Tue Oct 17 14:26:00 PDT 2023
>  at java.base/java.io.FileInputStream.<init>(FileInputStream.java:160)
>  at Build_gradle.<init>(build.gradle.kts:8)
>  at Program.execute(Unknown Source)
>  at org.gradle.kotlin.dsl.execution.Interpreter$ProgramHost.eval(Interpreter.kt:523)
>  at org.gradle.kotlin.dsl.execution.Interpreter$ProgramHost.evaluateSecondStageOf(Interpreter.kt:434)
>  at Program.execute(Unknown Source)
>  at org.gradle.kotlin.dsl.execution.Interpreter$ProgramHost.eval(Interpreter.kt:523)
>  at org.gradle.kotlin.dsl.execution.Interpreter.eval(Interpreter.kt:198)
>  at org.gradle.kotlin.dsl.provider.StandardKotlinScriptEvaluator.evaluate(KotlinScriptEvaluator.kt:121)
>  at org.gradle.kotlin.dsl.provider.KotlinScriptPluginFactory$create$1.invoke(KotlinScriptPluginFactory.kt:51)
>  at org.gradle.kotlin.dsl.provider.KotlinScriptPluginFactory$create$1.invoke(KotlinScriptPluginFactory.kt:48)
>  at org.gradle.kotlin.dsl.provider.KotlinScriptPlugin.apply(KotlinScriptPlugin.kt:35)
>  at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:65)
>  ...
(Rest of stacktrace omitted for brevity)

As can be seen, this file was opened 5 times and has yet to close. The number of times this file has opened coincides with how many syncs you have triggered. This is an obvious file leak, which you can verify by triggering more runs of the IntelliJ sync, downloading the raw logs, running the postprocess job, and diff-ing against the baseline again.

Finding and Fixing the First File Leak!

Now that we have found our file leak, we can begin looking at what may be causing it. If we examine the log closely, we'll see:

>  at java.base/java.io.FileInputStream.<init>(FileInputStream.java:160)
>  at Build_gradle.<init>(build.gradle.kts:8)
>  at Program.execute(Unknown Source)

This tells us that this leak is occurring in a build.gradle.kts file. Unfortunately, it doesn't tell us which project this belongs to, which means we'll have to dig into the build files for all the projects to find any code that opens a FileInputStream. Since I wrote the file leak in the example project, I can tell you that it is in the :someproject project, but it is very likely file leaks you encounter could also be in 3rd party libraries/plugins, buildSrc plugins, and/or included builds. Further investigations will be required on your part to find the location of these leaks.

Coming back to the current leak we found...

someproject/build.gradle.kts

import java.util.Properties

plugins {
    kotlin("jvm")
}

val customProps = Properties().apply {
    load(project.rootProject.file("mycustom.properties").inputStream())
}

afterEvaluate {
    println("SOME_VALUE=${customProps["SOME_VALUE"]}")
}

Looking at this script, we can see that the custom properties file, mycustom.properties, is loaded via the call to inputStream(). This custom properties object is then used to output the value of SOME_VALUE after the project is evaluated. However, on line 8, the inputStream() is never properly closed. This can easily be remedied by utilizing the use {} Kotlin lambda:

someproject/build.gradle.kts

import java.util.Properties

plugins {
    kotlin("jvm")
}

val customProps = Properties().apply {
    project.rootProject.file("mycustom.properties")
        .inputStream().use(::load)
}

afterEvaluate {
    println("SOME_VALUE=${customProps["SOME_VALUE"]}")
}

We can verify this change by killing the Gradle daemon, re-creating our baseline postprocess file, re-running the syncs, capturing the test postprocess results, and doing a diff of these files. As you can see, there are still more files being opened.

5365a5366,5369
> #873 /home/pablo/.gradle/caches/jars-9/851490ff745ba1a7abc0a483e543c397/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:12 PDT 2023
> #885 /home/pablo/.gradle/caches/jars-9/9449ad0b53bb85542ad161a6def1ef0b/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:15 PDT 2023
> #881 /home/pablo/.gradle/caches/jars-9/01fde95ef1bd4e20158029345491b272/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:14 PDT 2023
> #889 /home/pablo/.gradle/caches/jars-9/15e95a482b204e0b70b9ef9625b482b2/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:16 PDT 2023
5366a5371
> #877 /home/pablo/.gradle/caches/jars-9/866081816ac4706c0b764862ea82a630/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:13 PDT 2023
5500a5506
> #875 /home/pablo/.gradle/caches/jars-9/ab8e4f5ceb792f868ec42ef3d07eee30/init.jar by thread:Daemon worker on Tue Oct 17 16:20:12 PDT 2023
5501a5508
> #879 /home/pablo/.gradle/caches/jars-9/bc1404c99cc8606ca09377f830411b42/init.jar by thread:Daemon worker on Tue Oct 17 16:20:13 PDT 2023
5503a5511,5513
> #883 /home/pablo/.gradle/caches/jars-9/b429404e655ec8107d5d074223d8a436/init.jar by thread:Daemon worker on Tue Oct 17 16:20:14 PDT 2023
> #887 /home/pablo/.gradle/caches/jars-9/2c575d0a492aeab5fafd052018371556/init.jar by thread:Daemon worker on Tue Oct 17 16:20:15 PDT 2023
> #891 /home/pablo/.gradle/caches/jars-9/5f04e79805608b2dfa1b7c741b6dd56a/init.jar by thread:Daemon worker on Tue Oct 17 16:20:16 PDT 2023
11426a11437,11439
> #874 /home/pablo/.gradle/caches/jars-9/851490ff745ba1a7abc0a483e543c397/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:12 PDT 2023
> #886 /home/pablo/.gradle/caches/jars-9/9449ad0b53bb85542ad161a6def1ef0b/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:15 PDT 2023
> #882 /home/pablo/.gradle/caches/jars-9/01fde95ef1bd4e20158029345491b272/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:14 PDT 2023
11427a11441,11442
> #878 /home/pablo/.gradle/caches/jars-9/866081816ac4706c0b764862ea82a630/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:13 PDT 2023
> #890 /home/pablo/.gradle/caches/jars-9/15e95a482b204e0b70b9ef9625b482b2/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:16 PDT 2023
11556a11572
> #624 /home/pablo/.gradle/caches/jars-9/a008d0781e8859dd03f02bf926b9fbcf/init.jar by thread:Daemon worker on Tue Oct 17 16:19:11 PDT 2023
11557a11574,11577
> #880 /home/pablo/.gradle/caches/jars-9/bc1404c99cc8606ca09377f830411b42/init.jar by thread:Daemon worker on Tue Oct 17 16:20:13 PDT 2023
> #884 /home/pablo/.gradle/caches/jars-9/b429404e655ec8107d5d074223d8a436/init.jar by thread:Daemon worker on Tue Oct 17 16:20:14 PDT 2023
> #888 /home/pablo/.gradle/caches/jars-9/2c575d0a492aeab5fafd052018371556/init.jar by thread:Daemon worker on Tue Oct 17 16:20:15 PDT 2023
> #892 /home/pablo/.gradle/caches/jars-9/5f04e79805608b2dfa1b7c741b6dd56a/init.jar by thread:Daemon worker on Tue Oct 17 16:20:16 PDT 2023
11559c11579
< #624 /home/pablo/.gradle/caches/jars-9/a008d0781e8859dd03f02bf926b9fbcf/init.jar by thread:Daemon worker on Tue Oct 17 16:19:11 PDT 2023
---
> #876 /home/pablo/.gradle/caches/jars-9/ab8e4f5ceb792f868ec42ef3d07eee30/init.jar by thread:Daemon worker on Tue Oct 17 16:20:12 PDT 2023

These files are not part of the example project (nor a part of any other project you may be evaluating). They are actually init script files injected by IntelliJ into the Gradle daemon. As you can see, the number of these files coincide with the number of times the IntelliJ sync was run. Currently, this issue is filed here: https://youtrack.jetbrains.com/issue/IDEA-335172/Potential-file-leak-with-Gradle-init-scripts. For the purpose of this article, we'll ignore these leaking files.

The File Leak Hunt Continues!

So far, we have looked at the Initialization and Configuration phases of the Gradle build. This would mainly look at plugins and build scripts, but we haven't performed any task runs such as a test or build.

Searching for file leaks in the Execution phase can be tricky due to build caching, configuration caching, etc. Each task could also have their own file leak, meaning you'd have to run through all your tasks to detect them. In the example project, we'll just focus on the :generateProtos task, though I would implore you investigate other tasks.

Why :generateProtos?

Spoiler Alert: In the example project, there is another file leak with this specific task. We'll use the skills and tools learned from this article to find it. We can use IntelliJ to run the tasks to keep using the same daemon, but the command line can also be used, though you'll need to kill the current daemon, and re-establish the baseline again. You can do this by just running the task once.

As with the Gradle sync, if we run :generateProtos a number of times (about 5 times is what I go for), capture the logs, run postprocessing on them, and diff them against the baseline, you may see a number of files opened. However, if we investigate this diff, we'll see a new batch of files opened several times with a common stacktrace:

> #935 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #938 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #932 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #933 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #936 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #934 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #937 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #931 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
>   at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:215)
>   at okio.JvmSystemFileSystem.openReadOnly(JvmSystemFileSystem.kt:83)
>   at okio.ZipFileSystem.metadataOrNull(ZipFileSystem.kt:102)
>   at okio.internal.-FileSystem.commonMetadata(FileSystem.kt:36)
>   at okio.FileSystem.metadata(FileSystem.kt:33)
>   at okio.internal.-FileSystem.symlinkTarget(FileSystem.kt:152)
>   at okio.internal.-FileSystem.collectRecursively(FileSystem.kt:126)
>   at okio.internal.-FileSystem$collectRecursively$1.invokeSuspend(FileSystem.kt)
>   at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
>   at kotlin.sequences.SequenceBuilderIterator.hasNext(SequenceBuilder.kt:129)
>   at kotlin.sequences.FilteringSequence$iterator$1.calcNext(Sequences.kt:169)
>   at kotlin.sequences.FilteringSequence$iterator$1.hasNext(Sequences.kt:194)
>   at kotlin.sequences.TransformingSequence$iterator$1.hasNext(Sequences.kt:214)
>   at kotlin.sequences.SequencesKt___SequencesKt.toCollection(_Sequences.kt:787)
>   at kotlin.sequences.SequencesKt___SequencesKt.toSet(_Sequences.kt:828)
>   at com.squareup.wire.schema.DirectoryRoot.allProtoFiles(Root.kt:128)
>   at com.squareup.wire.schema.internal.CommonSchemaLoader.loadSourcePathFiles$wire_schema(CommonSchemaLoader.kt:117)
>   at com.squareup.wire.schema.internal.CommonSchemaLoader.loadSchema(CommonSchemaLoader.kt:101)
>   at com.squareup.wire.schema.SchemaLoader.loadSchema(SchemaLoader.kt:71)
>   at com.squareup.wire.schema.WireRun.execute$wire_schema(WireRun.kt:234)
>   at com.squareup.wire.schema.WireRun.execute(WireRun.kt:221)
>   at com.squareup.wire.gradle.WireTask.generateWireFiles(WireTask.kt:189)
>   ...
>   at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)
>   at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:337)
>   at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:324)
>   at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:317)
>   at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)
>   at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)
>   at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)
>   ...
>

Fixing a File Leak in the Open

As we saw with mycustom.properties, there are multiple files with a very similar stacktrace being opened. This may not always coincide with how often the task is run (especially if other projects run the same task), but we can see that there is another file leak, as continued runs of the task will open more files. Since this is inside of Okio (square/okio), which is open-sourced, this means we can create a PR fix for this issue! If you happen to find a leak inside closed-source software, these steps should provide enough data to file bugs as appropriate.

To lightly touch on the Okio file leak and the repercussions of it, when a protos Jar is used as the srcJar in Wire (square/wire), Wire will make a call to allProtos, which calls Okio to do a recursive iteration of all the files inside the Jar. However, to determine for symbolic links in Okio, each file's metadata is read, opening a RandomAcessFile, but it is not closed properly. Since this would happen for each project on each run of :generateProtos, this can quickly leak file descriptors to the limit, especially if the Jar file contained a large number of files!

Wrap-up

File leak detection is not easy, although the fixes may be. My goal with this article is to provide enough guidance to developers, so that they may continue finding and fixing these file leak issues, instead of just looking for ways to cover them up. Also, the skills and tools learned in this article can be applied to JVM webservices, long running JVM applications, etc. I chose to write this article with Gradle in mind due to how ubiquitous Gradle is for Android developers, but don't think this is just a Gradle (or even JVM) specific issue.

File Leak Bug Finds

Feel free to post your bug finds below!

DEV Community