Matt's Tidbits #83 - Puzzling unit test failures

Last week I wrote about an unexpected Mockito limitation. This week, I have a slightly different format for Matt's Tidbits!

Usually, my tidbits share knowledge I have gained, or my opinion about something. This time, however, I want to hear from you!

I've been experiencing some strange issues with unit tests on my project lately. Here's a little bit of info on our setup:

• Bitrise (CI server)
• Mockito v2.27.0
• PowerMock v2.0.2
• MockitoKotlin v2.1.0
• JUnit v4.13
• Gradle v6.5
• Multi-module blended Java/Kotlin project

We have about 700 unit tests in one of these modules, and lately we have been seeing test failures in some scenarios.

To be clear, running these unit tests from within Android Studio's GUI (right-clicking the package under the "test" directory and selecting "Run tests in ") works totally fine every time.

However, trying to run these unit tests locally via the command line with the following command
./gradlew :module:testSomeVariantDebugUnitTest

causes an OutOfMemoryError:

java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.ArrayList.<init>(ArrayList.java:153)
    at javassist.bytecode.stackmap.TypeData$TypeVar.<init>(TypeData.java:223)
    at javassist.bytecode.stackmap.TypeData.join(TypeData.java:63)
    at javassist.bytecode.stackmap.MapMaker.recordTypeData(MapMaker.java:303)
    at javassist.bytecode.stackmap.MapMaker.recordStackMap0(MapMaker.java:295)
    at javassist.bytecode.stackmap.MapMaker.recordStackMap(MapMaker.java:269)
    at javassist.bytecode.stackmap.MapMaker.make(MapMaker.java:205)
    at javassist.bytecode.stackmap.MapMaker.make(MapMaker.java:207)
    at javassist.bytecode.stackmap.MapMaker.make(MapMaker.java:207)
    at javassist.bytecode.stackmap.MapMaker.make(MapMaker.java:207)
    at javassist.bytecode.stackmap.MapMaker.make(MapMaker.java:207)
    at javassist.bytecode.stackmap.MapMaker.make(MapMaker.java:207)
    at javassist.bytecode.stackmap.MapMaker.make(MapMaker.java:207)
    at javassist.bytecode.stackmap.MapMaker.make(MapMaker.java:172)
    at javassist.bytecode.stackmap.MapMaker.make(MapMaker.java:116)
    at javassist.bytecode.MethodInfo.rebuildStackMap(MethodInfo.java:458)
    at javassist.bytecode.MethodInfo.rebuildStackMapIf6(MethodInfo.java:440)
    at javassist.expr.ExprEditor.doit(ExprEditor.java:118)
    at javassist.CtClassType.instrument(CtClassType.java:1541)
    at org.powermock.core.transformers.javassist.InstrumentMockTransformer.transform(InstrumentMockTransformer.java:41)
    at org.powermock.core.transformers.javassist.AbstractJavaAssistMockTransformer.transform(AbstractJavaAssistMockTransformer.java:40)
    at org.powermock.core.transformers.support.DefaultMockTransformerChain.transform(DefaultMockTransformerChain.java:43)
    at org.powermock.core.classloader.MockClassLoader.transformClass(MockClassLoader.java:184)
    at org.powermock.core.classloader.javassist.JavassistMockClassLoader.defineAndTransformClass(JavassistMockClassLoader.java:102)
    at org.powermock.core.classloader.MockClassLoader.loadMockClass(MockClassLoader.java:174)
    at org.powermock.core.classloader.MockClassLoader.loadClassByThisClassLoader(MockClassLoader.java:102)
    at org.powermock.core.classloader.DeferSupportingClassLoader.loadClass1(DeferSupportingClassLoader.java:147)
    at org.powermock.core.classloader.DeferSupportingClassLoader.loadClass(DeferSupportingClassLoader.java:98)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
    at org.powermock.core.classloader.MockClassLoader.defineClass(MockClassLoader.java:180)

This is the same command that our Bitrise server runs when it executes unit tests for this module. Interestingly enough, on Friday this was failing in Bitrise, but now they are passing (but still failing when run locally).

I have read countless articles and tried a variety of things, including increasing the amount of memory in -Xms, -Xmx, -XX:PermSize, and also setting -XX:-UseGcOverheadLimit.

However, none of these changes have helped resolve the issue locally. And, we are fairly confident that nothing has changed in our Bitrise server that would cause these tests to start failing (or start passing again).

If you have ideas about any of this (or the specific follow-up questions below), please let me know!

• Have you ever seen something like this before? If so, how did you fix it?
• Do you have suggestions for how to find memory leaks in unit tests?
• Any general comments/suggestions for how to proceed?

I have a hunch that PowerMock is just really really bad (as I've pointed out in numerous past tidbits), and eventually I will rip it all out and perhaps migrate to MockK, but that's a long way into the future.

Thanks for reading, and please leave a comment or send me a message if you have any ideas! When I eventually figure this out, I will share what I learned, but in the meantime, I could use your help! And, please follow me on Medium if you're interested in being notified of future tidbits.

This tidbit was discovered on October 2, 2020.