DEV Community: Pablo Baxter

Detecting File Leak in the Kotlin Daemon

Pablo Baxter — Wed, 19 Feb 2025 02:54:02 +0000

This was originally posted in https://frybits.com/blog/detecting-file-leak-in-the-kotlin-daemon/

File handle leaks are notoriously difficult to debug, so much so that most of the “fixes” for them are “increase the file descriptor limit”. However, this is not a fix. This is as close as you can get to covering your ears and closing your eyes, then screaming “I CAN’T HEAR YOU” to a bug. The file handle leak will still exist, but now it’s ignored.

Seriously, it's exactly this

In this case study, I decided to take the approach of finding out why the leak was occurring, and finding the best fix for it.

Signs of a File Handle Leak

The hardest part of debugging an issue is noticing it to begin with and file handle leaks are very hard to notice until an error is thrown, and even then it may not be noticed since the error could be handled by some logic that changes the exception cause or swallows it altogether. So how did I notice the one in the Kotlin daemon? Dumb luck. I had been tracking another file handle leak in the Gradle daemon, and just got curious about how many files remained open in the Kotlin daemon after compilation.

There are some signs that your application may have a file handle leak, but even these signs are easily missed. A few common signs are:

Errors relating to files unable to be created
Any error about a file being deleted (especially on Windows)
Unable to open files for any reason
Files having garbage data written or being corrupted

These issues don't always appear on every run and could be intermittent as well.

Noticing the Kotlin Daemon File Handle Leak

Going back to how I found the Kotlin daemon file handle leak, as I had mentioned previously, I found it due to curiosity and dumb luck. When Paul Klauser (https://github.com/PaulKlauser) reported a metaspace leak occuring in the Kotlin daemon, I was curious if this metaspace leak was potentially due to a file handle leak, as I had been tracking one in the Gradle daemon (which I still haven't found).

To go over how I did this at a high-level (more details later on), I attached Jenkin's file-leak-detector to the Kotlin daemon and ran ./gradlew clean assembleDebug --rerun-tasks several times, capturing the output of the file leak tool and using a post-processor (see https://github.com/centic9/file-leak-postprocess) to make the logs easier to read. I ended up with a file containing many stack traces that show where the file was opened. Many of them looked like the following (shortened for easy reading):

// Several hundred other open files up here with the same stacktrace
...
#1295 /home/pablo/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk8/1.9.0/e000bd084353d84c9e888f6fb341dc1f5b79d948/kotlin-stdlib-jdk8-1.9.0.jar by thread:RMI TCP Connection(10)-127.0.0.1 on Wed Oct 09 18:35:43 PDT 2024
#542 /home/pablo/.gradle/caches/modules-2/files-2.1/org.checkerframework/checker-qual/3.41.0/8be6df7f1e9bccb19f8f351b3651f0bac2f5e0c/checker-qual-3.41.0.jar by thread:RMI TCP Connection(21)-127.0.0.1 on Wed Oct 09 18:35:52 PDT 2024
#341 /home/pablo/.gradle/caches/modules-2/files-2.1/com.google.guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/b421526c5f297295adef1c886e5246c39d4ac629/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar by thread:RMI TCP Connection(140)-127.0.0.1 on Wed Oct 09 18:36:13 PDT 2024
    at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:181)
    at java.base/java.util.jar.JarFile.<init>(JarFile.java:346)
    at java.base/jdk.internal.loader.URLClassPath$JarLoader.getJarFile(URLClassPath.java:825)
    at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.run(URLClassPath.java:769)
    at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.run(URLClassPath.java:762)
    ...
    at java.base/java.net.URLClassLoader$3.next(URLClassLoader.java:659)
    at java.base/java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:684)
    at java.base/java.lang.CompoundEnumeration.next(ClassLoader.java:2730)
    at java.base/java.lang.CompoundEnumeration.hasMoreElements(ClassLoader.java:2739)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1210)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1228)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273)
    at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
    at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
    at com.google.common.collect.ImmutableSet.copyOf(ImmutableSet.java:280)
    at com.google.common.collect.ImmutableSet.copyOf(ImmutableSet.java:265)
    at dagger.internal.codegen.ServiceLoaders.loadServices(ServiceLoaders.java:35)
    at dagger.internal.codegen.DelegateComponentProcessor.lambda$initialize$0(DelegateComponentProcessor.java:89)
    ...
    at dagger.internal.codegen.DelegateComponentProcessor.initialize(DelegateComponentProcessor.java:89)
    at dagger.internal.codegen.KspComponentProcessor.initialize(KspComponentProcessor.java:49)
    at dagger.spi.internal.shaded.androidx.room.compiler.processing.ksp.KspBasicAnnotationProcessor.process(KspBasicAnnotationProcessor.kt:57)
    at com.google.devtools.ksp.AbstractKotlinSymbolProcessingExtension$doAnalysis$8$1.invoke(KotlinSymbolProcessingExtension.kt:310)
    at com.google.devtools.ksp.AbstractKotlinSymbolProcessingExtension$doAnalysis$8$1.invoke(KotlinSymbolProcessingExtension.kt:308)
    at com.google.devtools.ksp.AbstractKotlinSymbolProcessingExtension.handleException(KotlinSymbolProcessingExtension.kt:414)
    at com.google.devtools.ksp.AbstractKotlinSymbolProcessingExtension.doAnalysis(KotlinSymbolProcessingExtension.kt:308)
    at org.jetbrains.kotlin.cli.jvm.compiler.TopDownAnalyzerFacadeForJVM.analyzeFilesWithJavaIntegration(TopDownAnalyzerFacadeForJVM.kt:112)
    at org.jetbrains.kotlin.cli.jvm.compiler.TopDownAnalyzerFacadeForJVM.analyzeFilesWithJavaIntegration$default(TopDownAnalyzerFacadeForJVM.kt:75)
    ...

So what's going on here and how does this tell the story of a file leak?

Well, this stacktrace actually represents 417 files that were opened and never closed when the Gradle task was completed, but for brevity, I only listed the final three files left open (lines 3-5). All these files were opened at the ZipFile constructor (line 6), however that is not where the origin of the file was at. It actually originated elsewhere in this stacktrace, which required some investigation. Also, this is just one stack trace for the given set of files. There were 1380 files left open at the end of the Gradle task run, and about another 1k extra files were opened after each run. In the following sections, I'll get into the details of how I debugged this file leak and found the fix for it.

Setting Up the file-leak-detector

To begin with, I downloaded the file-leak-detector jar, and in the root gradle.properties file of the NowInAndroid repo, I appended the following line to the JVM args of the Kotlin daemon:

kotlin.daemon.jvmargs=<other args> -javaagent:/path/to/file-leak-detector-jar-with-dependencies.jar=http=19999

What this does is attach the file-leak-detector Java Agent to each Kotlin daemon that is launched. Since I only want one Kotlin daemon running, I made sure to kill the other Kotlin daemon processes before running the Gradle command.

The flag http=19999 creates a local web server on port 19999 that the file-leak-detector uses to provide a list of all the open files and stack traces to them. This is important as the Kotlin daemon is a long lived process, so it doesn't exit when the build has completed.

Capturing the File Handle Leak

Getting this file handle leak to occur was a bit tricky. I had to determine the task that would trigger the Kotlin compiler, and ensure that consecutive executions were re-run properly. For this reason I chose to run ./gradlew clean assembleDebug --rerun-tasks in order to ensure that the compilation occurred each time.

So what does each part do?

clean ensures that no generated code exists, and gives me a clean build directory.
assembleDebug triggers the compilation of the Android app in NowInAndroid.
--rerun-tasks tells the Gradle daemon to ignore any up-to-date checks from tasks and rerun them all.

After the first successful build of NowInAndroid using the above command, I was able to go to localhost:19999, which outputs all the open files left on the Kotlin daemon. That's where I got my first indication that something may be wrong, as I was left with 1380 file handles left open. This didn't necessarily mean there was a file leak, as these open file handles could be cached. To be sure, I stored this result as a text file, and re-ran the Gradle command while making sure the Kotlin daemon was not restarted or killed.

The next run left me with 2694 open file handles, with each run opening another ~1k files. This was obviously a leak. I stored the results of each run in a text file for post-processing, to make it easier to read.

Analyzing the Results

The file-leak-postprocess tool is useful as it gets all the open file handles that share a similar stacktrace and groups them together (see output above). I ran this post-processor tool for each output I collected, and took a quick look to see where most of these files were being opened. Visual Studio Code does a great job visualizing which stack traces have the most open file handles.

That's a lot of open files!

With this, I saw that there was a cluster of open files that shared common stacktraces. What I needed to identify was which of these grew on consecutive runs, as that would determine the location of the leaks. Again, Visual Studio Code came handy with their "Compare Selected" option when 2 files are selected in the file navigation. However, this comparison was a bit confusing to me at first.

Comparing the base to another run

Typically, I would only see new files added to specific stack traces that pointed out the leaks, yet here we had the previous files being closed and an increasing number of new files being opened. At first, I believed this to be due to a collection that was growing after each run, however I was not able to find any evidence to support this theory. The next approach I took was looking at the stacktraces and understanding a common root for them all. Thankfully, I had Jason Pearson in a Slack thread helping me out identify this common root, which he pointed out to be org.jetbrains.kotlin.cli.jvm.K2JVMCompiler.doExecute(K2JVMCompiler.kt:43).

Now that I knew the general area of where the file leaks were occurring, I was able to dig a bit deeper to understand the root cause, and discovered that both the Kotlin source code and KSP source code were creating URLClassloader objects, but never closing them. By default, URLClassloader caches the underlying .jar files being opened, and keeps them open until the URLClassloader has closed or has been garbage collected (see: Closing a URLClassLoader). With this finding, I filed a bug with the Kotlin team and with the Google/KSP team.

Patching the File Handle Leaks

The fix for these file leaks was rather easy for KSP. I just needed to call URLClassloader.close(). This was quickly achieved in this PR. Testing was a bit more difficult, but after working with the KSP team, I was able to test a local build of it on NowInAndroid and verified the file leaks for KSP were gone!

As for the Kotlin project fix, I made an attempt at a fix, but it turned that it would require quite a bit more effort. Essentially, there were many paths to open a URLClassloader, and my PR didn't cover all of those. Thankfully, Brian Norman on the Kotlin team took the charge on this fix!

"Too Many Open Files" - Debugging Leaking Files on the Gradle Daemon

Pablo Baxter — Mon, 30 Oct 2023 04:30:57 +0000

Banner image created with Dall-E using the prompt "elephant surrounded by files"

Preface

The Too many open files issue can be ignored by increasing the file descriptor limit, but there is no magic bullet to fix the error that may have brought you here. It's going to take some debugging and it is tedious. This article isn't to show you how to increase the file descriptor limit on Linux, or workaround the recent MacOS issue with increasing the file descriptor limit (see https://developer.apple.com/forums//thread/735798). There is no snippet of code that will fix this Gradle issue glaring at you. We'll be looking at how to identify file leaks on the Gradle daemon with the hope that it will provide you with the skills and tools to debug (and possibly fix) your build.

Let's Talk About Gradle

The Daemon

The Gradle build tool uses a long-lived daemon process that performs all Gradle builds to cache information in memory about previous builds, making subsequent builds faster. However, this also allows for open file descriptors to persist between builds, with subsequent builds potentially opening new file descriptors. Eventually this can lead to the dreaded Too many open files exception, which provides basically zero information about what file has leaked or where the leak is occurring.

Due to the nature of the daemon being long-lived, detecting file leaks can be problematic, as you'll be looking for an errant open file in a collection of other validly open files. However, there are some techniques that can be used to find these errant open files, so long as we have a basic understanding of the Gradle build lifecycle.

Build Lifecycle

Gradle has three distinct build phases:

Initialization: Evaluates the settings.gradle file.
Configuration: Evaluates the build.gradle files of projects.
Execution: Performs a run of the Gradle tasks.

Every phase requires the previous to occur and each of these phases can be triggered in different ways via IntelliJ or the command line. For the purposes of this article we'll only be using IntelliJ to trigger each of these phases.

The Tools

File Leak Detector

The tool I found effective for detecting these file leaks is the File Leak Detector by Jenkins (jenkinsci/lib-file-leak-detector). It is a Java Agent that can be applied via JVM arguments or attached after the JVM app has started. This library instruments several Java APIs to track all open file descriptors and either provides a simple HTTP server to show all currently open files or dumps them after the JVM has exited (which isn't helpful in our case, since Gradle is long-lived).
The output from this tool is very noisy. Each file descriptor, the thread it was opened in, time/date, and stacktrace are listed by default. For example:

#885 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:48 PDT 2023
 at java.base/java.nio.file.Files.newDirectoryStream(Files.java:482)
 at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:301)
 at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:374)
 at java.base/java.nio.file.Files.walkFileTree(Files.java:2844)
 at org.gradle.internal.snapshot.impl.DirectorySnapshotter.snapshot(DirectorySnapshotter.java:119)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.lambda$snapshot$8(DefaultFileSystemAccess.java:155)
 at org.gradle.internal.vfs.impl.AbstractVirtualFileSystem.store(AbstractVirtualFileSystem.java:84)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.snapshot(DefaultFileSystemAccess.java:145)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.lambda$read$0(DefaultFileSystemAccess.java:82)
 at java.base/java.util.Optional.orElseGet(Optional.java:364)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.lambda$readSnapshotFromLocation$9(DefaultFileSystemAccess.java:187)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess$StripedProducerGuard.guardByKey(DefaultFileSystemAccess.java:221)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.lambda$readSnapshotFromLocation$10(DefaultFileSystemAccess.java:184)
 at java.base/java.util.Optional.orElseGet(Optional.java:364)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.readSnapshotFromLocation(DefaultFileSystemAccess.java:184)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.readSnapshotFromLocation(DefaultFileSystemAccess.java:169)
 at org.gradle.internal.vfs.impl.DefaultFileSystemAccess.read(DefaultFileSystemAccess.java:82)
 (... Stacktrace manually collapsed for brevity...)
 at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:109)
 at org.gradle.internal.operations.DefaultBuildOperationQueue$WorkerRunnable.runBatch(DefaultBuildOperationQueue.java:224)
 at org.gradle.internal.operations.DefaultBuildOperationQueue$WorkerRunnable.run(DefaultBuildOperationQueue.java:192)
 at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
 at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
 at java.base/java.lang.Thread.run(Thread.java:1589)

Although descriptive, a single IDE sync can have upwards of 1k files opened with each file having a log entry as shown above. Thankfully, there is another tool that parses these raw logs and matches file descriptors with similar stacktraces.

File Leak Postprocessor

We can use the File Leak Postprocessor (centic9/file-leak-postprocess) to collect file descriptors with similar stacktraces and batch them as such:

#885 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:48 PDT 2023
#886 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#884 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2 by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:48 PDT 2023
#888 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#889 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main/java by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#892 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main/java/org/gradle/process by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#893 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main/java/org/gradle/process/internal by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#894 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main/java/org/gradle/process/internal/util by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#890 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main/java/org by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#883 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:48 PDT 2023
#887 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
#891 /home/pablo/.gradle/caches/transforms-3/c762f17d69b567ffc4fbadfe65190e1a/transformed/unzipped-distribution/gradle-8.2/subprojects/core/src/main/java/org/gradle by thread:Unconstrained build operations Thread 20 on Mon Oct 16 22:50:49 PDT 2023
 at java.base/java.nio.file.Files.newDirectoryStream(Files.java:482)
 at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:301)
 at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:374)
 at java.base/java.nio.file.Files.walkFileTree(Files.java:2844)
 ...
 at java.base/java.util.Optional.map(Optional.java:260)
 ...
 at org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess.withFileLock(LockOnDemandCrossProcessCacheAccess.java:90)
 at org.gradle.cache.internal.DefaultCacheCoordinator.withFileLock(DefaultCacheCoordinator.java:219)
 at org.gradle.cache.internal.DefaultPersistentDirectoryStore.withFileLock(DefaultPersistentDirectoryStore.java:188)
 at org.gradle.cache.internal.DefaultCacheFactory$RHunting File LeakseferenceTrackingCache.withFileLock(DefaultCacheFactory.java:214)
 ...
 at org.gradle.cache.Cache.lambda$get$0(Cache.java:31)
 at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
 at org.gradle.cache.ManualEvictionInMemoryCache.get(ManualEvictionInMemoryCache.java:30)
 at org.gradle.cache.internal.DefaultCrossBuildInMemoryCacheFactory$CrossBuildCacheRetainingDataFromPreviousBuild.get(DefaultCrossBuildInMemoryCacheFactory.java:255)
 at org.gradle.cache.Cache.get(Cache.java:31)
 ...

As shown above, the postprocessor tool collapses the stacktraces of many of these file descriptors and batches the ones that have similar traces. This makes it easier to analyze a diff of the processed outputs.

Let's Learn Some Debugging Skills

I have created a simple project (pablobaxter/file-leak-example) which will be used for the rest of this article to showcase how to find file leaks that may occur either in a build script directly or within the plugin of another library. These approaches can be applied to any Gradle project.

Attaching the File Leak Detector

One of the first steps to detecting leaks is by using a... detector. Thankfully, one is available here: https://repo.jenkins-ci.org/releases/org/kohsuke/file-leak-detector/

Regardless of what version you choose, you'll want to download the file-leak-detector-<VERSION>-jar-with-dependencies.jar file, and store it somewhere that is accessible.

Next, you'll need to attach the Java agent from this Jar to the Gradle daemon of the project. This can be done by adding -javaagent: to the org.gradle.jvmargs parameter of the gradle.properties file.

org.gradle.jvmargs=-Xmx1536m -javaagent:/path/to/file-leak-detector-<VERSION>-jar-with-dependencies.jar=http=19999,strong

In this example, we also add in the arguments to start the HTTP server (http=19999) and to not let the JVM garbage collector clean up any open file descriptors (strong). Once this property is added, kill any currently running Gradle daemon and re-sync the project on IntelliJ. This will start a new Gradle daemon and http://localhost:19999 will now start displaying open file descriptors or crashing the web browser if it's too much, but this isn't an issue for us. This output is what we will use for postprocessing. Once the sync has completed, save a copy of this output as a text file, as it will be used to create the baseline of the open file descriptors for the Gradle daemon.

At this point we have accomplished the following:

Instrumented the Gradle daemon with the File Leak Detector.
Started a simple web service that shows all currently open files on the Gradle daemon.
Stored a raw baseline of all open files to compare against.

Now we can start our search for leaking files.

Establishing the Baseline

So far, by syncing Gradle via IntelliJ, we have basically only run the Initialization and Configuration phases of the Gradle build. This means the settings.gradle and build.gradle files have been parsed and evaluated.

Now, let's do some postprocessing of the baseline output. The steps are listed in the centic9/file-leak-postprocess repository, except you will also store the baseline postprocess output somewhere that is easily accessible. With this, we finally have our true baseline file that we will use to diff against. We can now start looking for file leaks.

Hunting File Leaks

With the baseline postprocess file, we can now trigger the Gradle sync on IntelliJ a number of times. I typically start with 5 syncs, but any number is fine, so long as a count of the runs is kept. You'll notice these syncs are slightly faster, which is expected. Wait for the syncs to complete before continuing with the next.

If this is done on the example project, you may notice that the X descriptors are open line on the generated webpage (http://localhost:19999) has increased (probably not by a whole lot), but this may differ if you are testing on a different project. Let's store the new output, and run the postprocess job on it. Now if we look at the diff between the baseline and the new postprocess output, we'll notice a few things here.

On the example project, the first and most obvious difference is that we have multiple mycustom.properties files open:

1139a1140,1261
> #898 /home/pablo/Development/file-leak-example/mycustom.properties by thread:Daemon worker Thread 2 on Tue Oct 17 14:26:01 PDT 2023
> #878 /home/pablo/Development/file-leak-example/mycustom.properties by thread:Daemon worker Thread 2 on Tue Oct 17 14:25:57 PDT 2023
> #888 /home/pablo/Development/file-leak-example/mycustom.properties by thread:Daemon worker Thread 2 on Tue Oct 17 14:25:59 PDT 2023
> #883 /home/pablo/Development/file-leak-example/mycustom.properties by thread:Daemon worker Thread 2 on Tue Oct 17 14:25:58 PDT 2023
> #893 /home/pablo/Development/file-leak-example/mycustom.properties by thread:Daemon worker Thread 2 on Tue Oct 17 14:26:00 PDT 2023
>  at java.base/java.io.FileInputStream.<init>(FileInputStream.java:160)
>  at Build_gradle.<init>(build.gradle.kts:8)
>  at Program.execute(Unknown Source)
>  at org.gradle.kotlin.dsl.execution.Interpreter$ProgramHost.eval(Interpreter.kt:523)
>  at org.gradle.kotlin.dsl.execution.Interpreter$ProgramHost.evaluateSecondStageOf(Interpreter.kt:434)
>  at Program.execute(Unknown Source)
>  at org.gradle.kotlin.dsl.execution.Interpreter$ProgramHost.eval(Interpreter.kt:523)
>  at org.gradle.kotlin.dsl.execution.Interpreter.eval(Interpreter.kt:198)
>  at org.gradle.kotlin.dsl.provider.StandardKotlinScriptEvaluator.evaluate(KotlinScriptEvaluator.kt:121)
>  at org.gradle.kotlin.dsl.provider.KotlinScriptPluginFactory$create$1.invoke(KotlinScriptPluginFactory.kt:51)
>  at org.gradle.kotlin.dsl.provider.KotlinScriptPluginFactory$create$1.invoke(KotlinScriptPluginFactory.kt:48)
>  at org.gradle.kotlin.dsl.provider.KotlinScriptPlugin.apply(KotlinScriptPlugin.kt:35)
>  at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:65)
>  ...
(Rest of stacktrace omitted for brevity)

As can be seen, this file was opened 5 times and has yet to close. The number of times this file has opened coincides with how many syncs you have triggered. This is an obvious file leak, which you can verify by triggering more runs of the IntelliJ sync, downloading the raw logs, running the postprocess job, and diff-ing against the baseline again.

Finding and Fixing the First File Leak!

Now that we have found our file leak, we can begin looking at what may be causing it. If we examine the log closely, we'll see:

>  at java.base/java.io.FileInputStream.<init>(FileInputStream.java:160)
>  at Build_gradle.<init>(build.gradle.kts:8)
>  at Program.execute(Unknown Source)

This tells us that this leak is occurring in a build.gradle.kts file. Unfortunately, it doesn't tell us which project this belongs to, which means we'll have to dig into the build files for all the projects to find any code that opens a FileInputStream. Since I wrote the file leak in the example project, I can tell you that it is in the :someproject project, but it is very likely file leaks you encounter could also be in 3rd party libraries/plugins, buildSrc plugins, and/or included builds. Further investigations will be required on your part to find the location of these leaks.

Coming back to the current leak we found...

someproject/build.gradle.kts

import java.util.Properties

plugins {
    kotlin("jvm")
}

val customProps = Properties().apply {
    load(project.rootProject.file("mycustom.properties").inputStream())
}

afterEvaluate {
    println("SOME_VALUE=${customProps["SOME_VALUE"]}")
}

Looking at this script, we can see that the custom properties file, mycustom.properties, is loaded via the call to inputStream(). This custom properties object is then used to output the value of SOME_VALUE after the project is evaluated. However, on line 8, the inputStream() is never properly closed. This can easily be remedied by utilizing the use {} Kotlin lambda:

someproject/build.gradle.kts

import java.util.Properties

plugins {
    kotlin("jvm")
}

val customProps = Properties().apply {
    project.rootProject.file("mycustom.properties")
        .inputStream().use(::load)
}

afterEvaluate {
    println("SOME_VALUE=${customProps["SOME_VALUE"]}")
}

We can verify this change by killing the Gradle daemon, re-creating our baseline postprocess file, re-running the syncs, capturing the test postprocess results, and doing a diff of these files. As you can see, there are still more files being opened.

5365a5366,5369
> #873 /home/pablo/.gradle/caches/jars-9/851490ff745ba1a7abc0a483e543c397/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:12 PDT 2023
> #885 /home/pablo/.gradle/caches/jars-9/9449ad0b53bb85542ad161a6def1ef0b/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:15 PDT 2023
> #881 /home/pablo/.gradle/caches/jars-9/01fde95ef1bd4e20158029345491b272/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:14 PDT 2023
> #889 /home/pablo/.gradle/caches/jars-9/15e95a482b204e0b70b9ef9625b482b2/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:16 PDT 2023
5366a5371
> #877 /home/pablo/.gradle/caches/jars-9/866081816ac4706c0b764862ea82a630/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:13 PDT 2023
5500a5506
> #875 /home/pablo/.gradle/caches/jars-9/ab8e4f5ceb792f868ec42ef3d07eee30/init.jar by thread:Daemon worker on Tue Oct 17 16:20:12 PDT 2023
5501a5508
> #879 /home/pablo/.gradle/caches/jars-9/bc1404c99cc8606ca09377f830411b42/init.jar by thread:Daemon worker on Tue Oct 17 16:20:13 PDT 2023
5503a5511,5513
> #883 /home/pablo/.gradle/caches/jars-9/b429404e655ec8107d5d074223d8a436/init.jar by thread:Daemon worker on Tue Oct 17 16:20:14 PDT 2023
> #887 /home/pablo/.gradle/caches/jars-9/2c575d0a492aeab5fafd052018371556/init.jar by thread:Daemon worker on Tue Oct 17 16:20:15 PDT 2023
> #891 /home/pablo/.gradle/caches/jars-9/5f04e79805608b2dfa1b7c741b6dd56a/init.jar by thread:Daemon worker on Tue Oct 17 16:20:16 PDT 2023
11426a11437,11439
> #874 /home/pablo/.gradle/caches/jars-9/851490ff745ba1a7abc0a483e543c397/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:12 PDT 2023
> #886 /home/pablo/.gradle/caches/jars-9/9449ad0b53bb85542ad161a6def1ef0b/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:15 PDT 2023
> #882 /home/pablo/.gradle/caches/jars-9/01fde95ef1bd4e20158029345491b272/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:14 PDT 2023
11427a11441,11442
> #878 /home/pablo/.gradle/caches/jars-9/866081816ac4706c0b764862ea82a630/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:13 PDT 2023
> #890 /home/pablo/.gradle/caches/jars-9/15e95a482b204e0b70b9ef9625b482b2/cp_init.jar by thread:Daemon worker on Tue Oct 17 16:20:16 PDT 2023
11556a11572
> #624 /home/pablo/.gradle/caches/jars-9/a008d0781e8859dd03f02bf926b9fbcf/init.jar by thread:Daemon worker on Tue Oct 17 16:19:11 PDT 2023
11557a11574,11577
> #880 /home/pablo/.gradle/caches/jars-9/bc1404c99cc8606ca09377f830411b42/init.jar by thread:Daemon worker on Tue Oct 17 16:20:13 PDT 2023
> #884 /home/pablo/.gradle/caches/jars-9/b429404e655ec8107d5d074223d8a436/init.jar by thread:Daemon worker on Tue Oct 17 16:20:14 PDT 2023
> #888 /home/pablo/.gradle/caches/jars-9/2c575d0a492aeab5fafd052018371556/init.jar by thread:Daemon worker on Tue Oct 17 16:20:15 PDT 2023
> #892 /home/pablo/.gradle/caches/jars-9/5f04e79805608b2dfa1b7c741b6dd56a/init.jar by thread:Daemon worker on Tue Oct 17 16:20:16 PDT 2023
11559c11579
< #624 /home/pablo/.gradle/caches/jars-9/a008d0781e8859dd03f02bf926b9fbcf/init.jar by thread:Daemon worker on Tue Oct 17 16:19:11 PDT 2023
---
> #876 /home/pablo/.gradle/caches/jars-9/ab8e4f5ceb792f868ec42ef3d07eee30/init.jar by thread:Daemon worker on Tue Oct 17 16:20:12 PDT 2023

These files are not part of the example project (nor a part of any other project you may be evaluating). They are actually init script files injected by IntelliJ into the Gradle daemon. As you can see, the number of these files coincide with the number of times the IntelliJ sync was run. Currently, this issue is filed here: https://youtrack.jetbrains.com/issue/IDEA-335172/Potential-file-leak-with-Gradle-init-scripts. For the purpose of this article, we'll ignore these leaking files.

The File Leak Hunt Continues!

So far, we have looked at the Initialization and Configuration phases of the Gradle build. This would mainly look at plugins and build scripts, but we haven't performed any task runs such as a test or build.

Searching for file leaks in the Execution phase can be tricky due to build caching, configuration caching, etc. Each task could also have their own file leak, meaning you'd have to run through all your tasks to detect them. In the example project, we'll just focus on the :generateProtos task, though I would implore you investigate other tasks.

Why :generateProtos?

Spoiler Alert: In the example project, there is another file leak with this specific task. We'll use the skills and tools learned from this article to find it. We can use IntelliJ to run the tasks to keep using the same daemon, but the command line can also be used, though you'll need to kill the current daemon, and re-establish the baseline again. You can do this by just running the task once.

As with the Gradle sync, if we run :generateProtos a number of times (about 5 times is what I go for), capture the logs, run postprocessing on them, and diff them against the baseline, you may see a number of files opened. However, if we investigate this diff, we'll see a new batch of files opened several times with a common stacktrace:

> #935 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #938 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #932 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #933 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #936 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #934 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #937 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
> #931 /home/pablo/Development/file-leak-example/someapplication/../../../.gradle/caches/modules-2/files-2.1/com.squareup.wire/wire-reflector/4.9.1/39f775c2f78dd29b2aecceb61ef42c41a45bb54b/wire-reflector-4.9.1.jar by thread:Execution worker on Tue Oct 24 14:30:51 PDT 2023
>   at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:215)
>   at okio.JvmSystemFileSystem.openReadOnly(JvmSystemFileSystem.kt:83)
>   at okio.ZipFileSystem.metadataOrNull(ZipFileSystem.kt:102)
>   at okio.internal.-FileSystem.commonMetadata(FileSystem.kt:36)
>   at okio.FileSystem.metadata(FileSystem.kt:33)
>   at okio.internal.-FileSystem.symlinkTarget(FileSystem.kt:152)
>   at okio.internal.-FileSystem.collectRecursively(FileSystem.kt:126)
>   at okio.internal.-FileSystem$collectRecursively$1.invokeSuspend(FileSystem.kt)
>   at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
>   at kotlin.sequences.SequenceBuilderIterator.hasNext(SequenceBuilder.kt:129)
>   at kotlin.sequences.FilteringSequence$iterator$1.calcNext(Sequences.kt:169)
>   at kotlin.sequences.FilteringSequence$iterator$1.hasNext(Sequences.kt:194)
>   at kotlin.sequences.TransformingSequence$iterator$1.hasNext(Sequences.kt:214)
>   at kotlin.sequences.SequencesKt___SequencesKt.toCollection(_Sequences.kt:787)
>   at kotlin.sequences.SequencesKt___SequencesKt.toSet(_Sequences.kt:828)
>   at com.squareup.wire.schema.DirectoryRoot.allProtoFiles(Root.kt:128)
>   at com.squareup.wire.schema.internal.CommonSchemaLoader.loadSourcePathFiles$wire_schema(CommonSchemaLoader.kt:117)
>   at com.squareup.wire.schema.internal.CommonSchemaLoader.loadSchema(CommonSchemaLoader.kt:101)
>   at com.squareup.wire.schema.SchemaLoader.loadSchema(SchemaLoader.kt:71)
>   at com.squareup.wire.schema.WireRun.execute$wire_schema(WireRun.kt:234)
>   at com.squareup.wire.schema.WireRun.execute(WireRun.kt:221)
>   at com.squareup.wire.gradle.WireTask.generateWireFiles(WireTask.kt:189)
>   ...
>   at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)
>   at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:337)
>   at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:324)
>   at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:317)
>   at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)
>   at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)
>   at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)
>   ...
>

Fixing a File Leak in the Open

As we saw with mycustom.properties, there are multiple files with a very similar stacktrace being opened. This may not always coincide with how often the task is run (especially if other projects run the same task), but we can see that there is another file leak, as continued runs of the task will open more files. Since this is inside of Okio (square/okio), which is open-sourced, this means we can create a PR fix for this issue! If you happen to find a leak inside closed-source software, these steps should provide enough data to file bugs as appropriate.

To lightly touch on the Okio file leak and the repercussions of it, when a protos Jar is used as the srcJar in Wire (square/wire), Wire will make a call to allProtos, which calls Okio to do a recursive iteration of all the files inside the Jar. However, to determine for symbolic links in Okio, each file's metadata is read, opening a RandomAcessFile, but it is not closed properly. Since this would happen for each project on each run of :generateProtos, this can quickly leak file descriptors to the limit, especially if the Jar file contained a large number of files!

Wrap-up

File leak detection is not easy, although the fixes may be. My goal with this article is to provide enough guidance to developers, so that they may continue finding and fixing these file leak issues, instead of just looking for ways to cover them up. Also, the skills and tools learned in this article can be applied to JVM webservices, long running JVM applications, etc. I chose to write this article with Gradle in mind due to how ubiquitous Gradle is for Android developers, but don't think this is just a Gradle (or even JVM) specific issue.

File Leak Bug Finds

Feel free to post your bug finds below!