DEV Community: Vlad

Java 17 migration: bias locks performance regression

Vlad — Thu, 16 Feb 2023 09:53:51 +0000

What issue do I need to solve

We are about to switch from java 11 to java 17 finally, but a few performance tests failed and failed dramatically with around 50% of regression.
It was 6 seconds in java 11 vs. 9 seconds in java 17 in absolute number in some tests.
And this happens just by switching runtime.

So I've got to investigate why the tests failed and fix the issue.

What I did to get some data

First, I've decided to run the test with a profiler. I'm using Intellij IDEA Profiler with the default settings for the smoke test.
Once I ran it, I found strange plateaus on a flame graph that was missing in the java 11 profiling report.

So I've decided to look closer into the method.

Digging deeper

The method looks like this:



public class LazyInputStream extends InputStream{
    private InputStream is;
    ...

    protected synchronized InputStream getInstance() {
       if (is == null) {
           is = factory.create();
       }
       return is;
    }
    ...
    public int read() throws IOException {  
        return getInstance().read();  
    }
}

The method doesn't do any long operation. It only creates an instance of an input stream when required and uses it later.
But we are calling the method for each read in the external input stream. In other words, each time when we are going to read the next portion of data.
And the method is synchronized to guarantee a single instance of input stream creation in a multithreading environment.
And there is no this kind of plateau in java 11 profiling. I've rerun it and checked several times on different machines.

After some googling, I've found this JEP, which says that bias locks are disabled from java 15.

What is a bias lock

Bias lock is optimization for synchronized in java.
It works when the method is not so concurrent, and only a single thread usually acquires a lock.
JVM raises a flag in the monitor object that some thread acquires the lock, so reacquiring and releasing the lock by the same thread is lightweight. But the lock must be revoked when another thread tries to acquire the bias lock. And the revocation is a costly operation.

So community decided to deprecate bias locks and remove them later.

Because if you really have a multithreaded application, you would like to avoid costly bias lock revocation, and if your application is not so multithreaded, you don't need the lock at all.

How did I fix it

First, let's run the tests with -XX:+UseBiasedLockingJVM option to turn on the bias locks.
The regression issue disappears, but the warning appears:



OpenJDK 64-Bit Server VM warning: Option UseBiasedLocking was deprecated in version 15.0 and will likely be removed in a future release.

Ok, so let's implement our lazy initialization more smartly to avoid acquiring the lock every time and use old fashion but still working double-checked locking.
I've found it implemented by Suppliers.memoize in guava library.



public class LazyInputStream extends InputStream {  

    private final Supplier<InputStream> initializer;

    public LazyInputStream(InputStreamFactory factory) {  
        this.initializer = Suppliers.memoize(() -> {  
            try {  
                return factory.create();  
            } catch (IOException e) {  
                throw new IllegalStateException("Failed to create input stream", e);  
            }  
        });  
    }

    ...

    protected synchronized InputStream getInstance() {
       return initialized.get();
    }
    ...
    public int read() throws IOException {  
        return getInstance().read();  
    }

The memoizing supplier uses a 2-fields variant of double-checked locking, but it is because the supplier may return null as a valid value.



public T get() {  
  if (!initialized) {          // boolean flag  
    synchronized (this) {  
      if (!initialized) {      // double check to avoid races
        T t = delegate.get();  // calling delegate to get a value
        value = t;             // and remember it
        initialized = true;    // rise the boolean flag
        return t;  
      }  
    }  
  }  
  return value;  
}

Once I rerun the tests, I found that regression did not disappear completely, so it looks like we have similar locks, but they are not too hot to be visible on flame graphs.
Finally, we decided to go with turned-on bias locks and continue working on our code to improve it continuously.

Conclusion

You may get some performance boost only by turning on bias locks if you have something similar to us and using java 15+. But it is always better to rewrite problem pieces.
Test your code performance in every java version. It will help you find regression earlier and only spend a little time looking into strange plateaus.

If you find the post helpful, please support me and:

3 ways to know used memory

Vlad — Mon, 13 Feb 2023 11:44:03 +0000

What task I would like to solve

Recently I tried to know how much memory an application wastes during some operation.
The operation I want to measure is data snapshot creation.
The application stores a sequence of operations with objects and periodically creates object snapshots.
So if we want to know the object's state at any point in time, we won't replay all operations from the beginning to this point but get the object's state from the nearest snapshot and play only a limited amount of operation from this snapshot to our point.

Obviously, we would like to store as many recent snapshots as possible in heap memory, but in the real world, we have limited memory. We should limit the amount of memory for snapshots, and we are doing it traditionally by setting the threshold via application properties.

Next step, we want to provide some meaningful defaults for this property.
The issue here is that we cannot predict the value because different customers have different: environments, data, sizes of snapshots, and so on.
This means we need to have more data to make a decision.

Finally, we decided to log how much heap was occupied before and after snapshot creation and end with something like this:

public Snapshot buildSnapshot() {
    long usedMemory = calculateUsedMemory();
    try {
        return doBuildSnapshot();
    } finally {
        log.debug("Used memory: " + (calculateUsedMemory() - usedMemory));
    }
}

`java.lang.Runtime` way

If you google "java used memory in runtime" you will see this method: Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory() and different variations over it.

Let's look closer at what this method returns to us.

For total memory:

Returns the total amount of memory in the Java virtual machine. The value returned by this method may vary over time, depending on the host environment.

For free memory:

Returns the amount of free memory in the Java Virtual Machine. Calling the gc method may result in increasing the value returned by freeMemory.

In other words, total memory may contain references no longer needed objects. The object will be freed after garbage collection, but we don't have any control over GC and don't know when it is executed.

So we need to find some better solution.

JMX way

JMX is stated from Java Management Extensions. We may use these built-in extensions to get different information about Runtime internals.

In java.lang.management.ManagementFactory, we may find a lot of factory methods for any task we like, but we are most interested in MemoryMXBean getMemoryMXBean().

The method returns a manageble bean we can use to request heap memory usage.

private static final MemoryMXBean MEMORY_MX_BEAN = ManagementFactory.getMemoryMXBean();

private long calculateUsedMemory() {
    return  MEMORY_MX_BEAN.getHeapMemoryUsage().getUsed();
}

So far, good, but the methods above show only used memory at some specific time for the whole application, and we still need to find out how much memory the application waste on storing the snapshot.

Let's look at the task from another point, how much memory we used to complete the operation.

In other words, how effective is the code from a memory usage perspective?
Knowing this and GC information, we could predict the default value.

Secret JMX way

JMX has two ThreadMXBean interfaces, java.lang.management.ThreadMXBean and com.sun.management.ThreadMXBean.
Both could be used to get information about threads.

The first one is public and returned by ManagementFactory, but it does not have any memory-related methods.

The second one declares 2 excellent methods:

long getThreadAllocatedBytes(long id) - Returns an approximation of the total amount of memory, in bytes, allocated in heap memory for each thread whose ID is in the input array ids.
long getCurrentThreadAllocatedBytes() - Returns an approximation of the total amount of memory, in bytes, allocated in heap memory for the current thread. More than com.sun.management.ThreadMXBean extends java.lang.management.ThreadMXBean, so we can use the interface instead of the first one.

But you will not find any methods returning these interfaces.

If will look at ThreadMXBean hierarchy, I may see that, in my JDK, both interfaces are implemented by a single class.

I've used Amazon Correto 11, but the same picture in other JDKs (checked on Eclipse Temurin and Azul).

Now my code may look like this:

private final LongSupplier allocatedByThread = initAllocatedMemoryProvider(); 

private static LongSupplier initAllocatedMemoryProvider() {  

    ThreadMXBean threadMXBean = ManagementFactory.getThreadMXBean();  
    if (threadMXBean instanceof com.sun.management.ThreadMXBean) {  
        com.sun.management.ThreadMXBean casted = (com.sun.management.ThreadMXBean) threadMXBean;  
        return casted::getCurrentThreadAllocatedBytes;  
    }  
    return () -> 0;  
}

private long calculateUsedMemory() {
    return initAllocatedMemoryProvider().getAsLong();
}

The issue is only that com.sun.management.ThreadMXBean in jdk.management module, so if a customer runs our application with JRE we will not have the interface in the classpath. The application will crush with NoClassDefFoundError, but we may fix it with Reflection API.

private static LongSupplier initAllocatedMemoryProvider() {  
    try {  
        Class<?> internalIntf = Class.forName("com.sun.management.ThreadMXBean");  
        ThreadMXBean bean = ManagementFactory.getThreadMXBean();  
        if (!internalIntf.isAssignableFrom(bean.getClass())) {  
            // Attempts to get the interface from PlatformMXBean
            Class<?> pmo = Class.forName("java.lang.management.PlatformManagedObject");  
            Method m = ManagementFactory.class.getMethod("getPlatformMXBean", Class.class, pmo);  
            bean = (ThreadMXBean) m.invoke(null, internalIntf);  
            if (bean == null) {  
                throw new UnsupportedOperationException("No way to access private ThreadMXBean");  
            }  
        }  

        ThreadMXBean allocMxBean = bean;  
        Method allocMxBeanGetter = internalIntf.getMethod("getCurrentThreadAllocatedBytes");  

        return () -> (long)allocMxBeanGetter.invoke(allocMxBean);
    } catch (Exception e) {  
        return () -> 0;  
    } 
}

Conclusion

You may get access to different runtime information with JMX, but you may find a treasure if you dig slightly deeper.

You may find the code on GitHub.

If you find the post helpful, please support me and:

Running JMH benchmark from Eclipse

Vlad — Fri, 18 Nov 2022 22:07:18 +0000

What issue I'm trying to solve

A few months ago, we started to use JMH in our project to test and find performance issues.
The tool provides multiple modes and profilers, and we found this useful for our purposes.

Intellij IDEA, which I'm using, has a useful Intellij IDEA plugin for Java Microbenchmark Harness. The plugin's functionality is similar to JUnit Plugin. It simplifies benchmarks development and debugging by allowing running some benchmarks together or separately.

But... half of our team uses Eclipse as a major IDE and the IDE does not have any plugins or support for the tool.
Even if we may run it with the main method, it is inconvenient to change the include pattern and do not forget to revert the changes before committing them into git.

So, after small brainstorming, we decided to write a custom JUnit Runner with functionality to run benchmark.

JUnit 4 runners

JUnit 4 has API to run any class as a test suite.
All you need is only to:

write class extending org.junit.runner.Runner class

public class BenchmarkRunner extends Runner {
    //...
}

implement constructor and few methods

public class BenchmarkRunner extends Runner {
   public BenchmarkRunner(Class<?> benchmarkClass) {
   }

   public Description getDescription() {
       //...
   }  

   public void run(RunNotifier notifier) {
       //...
   }
}

add the runner to your test class

@RunWith(BenchmarkRunner.class)  
public class CustomCollectionBenchmark {
    //...
}

Implementing Benchmark JUnit runner

First, we need to provide information about tests to the JUnit engine.

JUnit runners have the getDescription() method for it. But how to get information about test classes and test methods?

From JavaDoc:

When creating a custom runner, in addition to implementing the abstract methods here you must also provide a constructor that takes as an argument the Class containing the tests.

So, we may get the class as a constructor argument and get all information with reflection's help.

public class BenchmarkRunner extends Runner {
    private final Class<?> benchmarkClass; // (1)  
    private final List<Method> benchmarkMethods; // (2) 

    public BenchmarkRunner(Class<?> benchmarkClass) {  
        this.benchmarkClass = benchmarkClass;  
        this.benchmarkMethods = Arrays.stream(benchmarkClass.getDeclaredMethods())  
                .filter(m -> m.isAnnotationPresent(Benchmark.class))  
                .collect(Collectors.toList());  
    }
    //...
}

Now I have all the required information to provide a test suite description to the JUnit engine:

Actual benchmark class
All methods marked with @Benchmark in this class

public class BenchmarkRunner extends Runner {  
    //... 
    @Override  
    public Description getDescription() {  
        Description result = Description.createSuiteDescription(benchmarkClass);  
        benchmarkMethods.stream()  
                .map(m -> Description.createTestDescription(benchmarkClass, m.getName()))  
                .forEach(result::addChild);  
        return result;  
    }  
    //...
}

Once we run the test we may see something like this:

Let's run our benchmarks.
We need to implement the method org.junit.runner.Runner.run(RunNotifier) where RunNotifier is responsible to notify the engine about test runs.

The idea is we run sequentially one-by-one benchmark methods, each in separate org.openjdk.jmh.runner.Runner.

public class BenchmarkRunner extends Runner {

    ...

    @Override
    public void run(RunNotifier notifier) {
        for (Method benchmarkMethod : benchmarkMethods) {
            Description testDescription = getBenchmarkMethodDescription(benchmarkMethod);
            try {
                notifier.fireTestStarted(testDescription);
                Options opt = new OptionsBuilder()
                        .include(".*" + benchmarkClass.getName() + "." + benchmarkMethod.getName() + ".*")
                        .jvmArgsAppend("-Djmh.separateClasspathJAR=true")
                        .build();

                new org.openjdk.jmh.runner.Runner(opt).run();

                notifier.fireTestFinished(testDescription);
            } catch (Exception e) {
                notifier.fireTestFailure(new Failure(testDescription, e));
                return;
            }
        }
    }

    private Description getBenchmarkMethodDescription(Method benchmarkMethod) {
        return Description.createTestDescription(benchmarkClass, benchmarkMethod.getName());
    }
}

Options mean follow:

include - benchmark we would like to include in the run.
jvmArgsAppend("-Djmh.separateClasspathJAR=true") - specific option, telling JMH to build classpath.jar to avoid the The filename or extension is too long error

As you may see, we notify RunNotifier when we start and finish our test, successful or not.
Looks good, but we are running all tests even when choosing to run only a single one.

Filtering

Our runner should implement the org.junit.runner.manipulation.Filterable interface to allow to JUnit engine to tell our code what tests should be run.
The interface has only a single method void filter(Filter) and org.junit.runner.manipulation.Filter argument has the shouldRun(Description) method we may use to know if the test requested to run.
The method doesn't return anything, so looks like we need to store filtered results and use them later.


public class BenchmarkRunner extends Runner implements Filterable {

    //...

    private List<Method> readyToRunMethods; // <= add new field to store filter result  

    @Override  
    public Description getDescription() {  
        Description result = Description.createSuiteDescription(benchmarkClass);  
        readyToRunMethods.stream()  // <= use the field here
                .map(this::getBenchmarkMethodDescription)  
                .forEach(result::addChild);  
        return result;  
    }  

    //...

    @Override  
    public void run(RunNotifier notifier) {  
        for (Method benchmarkMethod : readyToRunMethods) {  // <= and here
            //...  
        }  
    }  

    @Override  
    public void filter(Filter filter) throws NoTestsRemainException {  
        List<Method> filteredMethods = new ArrayList<>();  

        for (Method benchmarkMethod : benchmarkMethods) {  
            if (filter.shouldRun(getBenchmarkMethodDescription(benchmarkMethod))) {  
                filteredMethods.add(benchmarkMethod);  
            }  
        }  

        if (filteredMethods.isEmpty()) {  
            throw new NoTestsRemainException();  
        }  

        this.readyToRunMethods = filteredMethods;  
    }  
}

Now it runs only methods we ask to run.

Conclusion

Finally, we've got unified way to run benchmarks from any IDE during development and debugging. This is really simplify our daily life.
We are still running the benchmarks with main method to reduce environment noise and get reliable results for analysis.

You may find the code in GitHub.

If you find the post helpful, please support me and:

How to change already written code: Aspects

Vlad — Thu, 20 Oct 2022 11:17:44 +0000

What an issue I'm trying to solve

Now I'm trying to solve some generic issue. We have a lot of idempotent methods, methods without side effects and returning the same result for the same arguments.
Additionally, we have some methods of reading data outside, but the data are rarely changed.

We may use, and we are using, caching to improve response time in this case.
The issue is that all these methods are different, as well as the ways we are using caches (from local value holders and hash maps to EhCache).
So we would like to unify a caching approach to be able to configure different caches, their eviction, and capacity policies and ways how we store them.

Also, I wouldn't like to pollute the business code with extra supporting code.

Please compare without cache:

public Value get() {
    return readValue();
}

and with cache:

public Value get() {
    var value = cache.get("value");
    if (value != null) {
        return value;
    }

    value = readValue();
    cache.put("value", value);
    return value
}

Changes I've made before

First, I've unified caching interface and replaced it in already-known places:

public interface Cache<K, V> {
    V get(K key);
    void put(K key, V value);
}

public final class CacheManager {
    public static <K, V> Cache<K, V> getInstance(String region) {
        ...
    }
}

Now code with caching looks even uglier:

public Value get() {
    var cache = CacheManager.getInstance("region"); 
    var value = cache.get("value");
    if (value != null) {
        return value;
    }

    value = readValue();
    cache.put("value", value);
    return value
}

Luckily, mankind invents AOP to solve this kind of task.

Aspect oriented programming overview

AOP allows you to add actions (called Advice) to some points in an application (called Join points) specified by expressions (called Pointcut).
All this stuff together is called Aspect.

So in a few steps:

You have a method you would like to improve. This is your Join point

    package com.example.apects;  

    public class Example {  

        public int calculateTwoPlusTwo() {  
            return 4;  
        }  
    }

Write what you would like to do. It is your Advice

    public class ExampleAspect {  

        public int returnFive() {  
            return 5;  
        }  
    }

Specify where you would like to apply it. It is you Pointcut

    @Aspect  
    public class ExampleAspect {  

        @Pointcut("execution(public int com.example.apects.Example.calculateTwoPlusTwo())")  
        public void returnFivePointcut() {  
        }  
        ...  
    }

Add bind all together. Now you have Aspect

    @Aspect  
    public class ExampleAspect {  

        @Pointcut("execution(public int com.example.apects.Example.calculateTwoPlusTwo())")  
        public void returnFivePointcut() {  
        }  

        @Around("returnFivePointcut()")  
        public Object returnFive() {  
            return 5;  
        }  
    }

Applying aspects for the caches problem

As you may see above, I need to know and specify all methods I would like to wrap with caching.
Bad news, there is no pattern in method names I could use to reduce the number of pointcuts.
The good news, Java has annotations and Aspects that may work with them.

So I can do something like:

Create custom annotation

@Target({ElementType.METHOD})  
@Retention(RetentionPolicy.RUNTIME)  
public @interface Cacheable {  
    String value() default "common";  
}

Write Aspect

@Aspect  
public class CachingAspect {  

    @Pointcut("@annotation(cacheable)")  
    public void cachingAnnotation(Cacheable cacheable) {}  

    @Around("cachingAnnotation(cacheable)")  
    public Object checkCache(ProceedingJoinPoint pjp, Cacheable cacheable) throws Throwable {  
        var cache = CacheManager.getInstance(cachable.value());  
        var value = cache.get("value");  
        if (value != null) {  
            return value;  
        }  
        value = pjp.proceed();  
        cache.put("value", value);  
        return value;  
    }  
}

Put annotation to target method:

@Cacheble("region")
public Value get() {
    return readValue();
}

Looks good. I have caching separated from business code but:

I need to solve an issue with the cache key
- Somehow the logic should depend on the method name and argument
I don't have any clue if the aspect was really applied to a method or not (just IDE highlights)

For first issue, I would use a full class and target method names and args as array.
I would need to override equals\hashCode for arguments, and some arguments can be excluded. But I will care about it later.

@Aspect  
public class CachingAspect {  

    ...

    @Around("cachingAnnotation(cacheable)")  
    public Object checkCache(ProceedingJoinPoint pjp, Cacheable cacheable) throws Throwable {  
        var cache = CacheManager.getInstance(cacheable.value());  
        Signature signature = pjp.getSignature();  
        if (!(signature instanceof MethodSignature)) {  
            return pjp.proceed();
        }  

        MethodSignature methodSignature = (MethodSignature)signature;  
        Method method = methodSignature.getMethod();  
        CacheKey key = new CacheKey(  
                method.getDeclaringClass().getCanonicalName(),  
                method.getName(),  
                pjp.getArgs()  
        );  

        var value = cache.get(key);  
        if (value != null) {  
            return value;  
        }  
        value = pjp.proceed();  

        cache.put(key, value);  
        return value;  
    }  

    public static class CacheKey {  
        private final String className;  
        private final String methodName;  
        private final Object[] args;  

        public CacheKey(String className, String methodName, Object[] args) {  
            this.className = className;  
            this.methodName = methodName;  
            this.args = args;  
        }  

        @Override  
        public boolean equals(Object o) {  
            ...    
        }  

        @Override  
        public int hashCode() {  
            ...
        }  
    }
}

About the second issue, I will use the AspectJ maven plugin for compile time weaving.
The plugin has a showWeaveInfo option to log all information about what and how is weaved.


<build>  
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>  
            <artifactId>aspectj-maven-plugin</artifactId>  
            <version>1.14.0</version>  
            <configuration>
                <complianceLevel>11</complianceLevel>  
                <source>11</source>  
                <target>11</target>  
                <showWeaveInfo>true</showWeaveInfo>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>  
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Once build is done, I may check logs

[INFO] Join point 'method-call(com.example.apects.Example$Value com.example.apects.Example.getValue())' in Type 'com.example.apects.Example' (Example.java:7) advised by around advice from 'com.example.apects.CachingAspect' (CachingAspect.java:22)
[INFO] Join point 'method-execution(com.example.apects.Example$Value com.example.apects.Example.getValue())' in Type 'com.example.apects.Example' (Example.java:12) advised by around advice from 'com.example.apects.CachingAspect' (CachingAspect.java:22)

OK, here I may see that my join point was advised twice, so my code will also be called twice.

the first time in method invocation point (method-call)
the second time in method (method-execution)

So I would like to modify the pointcut to be used only in execution because I would like to cache only methods in my project.
If I cache some methods in a third-party library, I need to use a method-call advice to put cache-related code around method invocation.

So I will add execution(* *.*(..)) into advice. You may read it like this: execution of any method in any class with any arguments and any modifier.

@Around(value = "execution(* *.*(..)) && cachingAnnotation(cacheable)", argNames = "pjp,cacheable")  
public Object checkCache(ProceedingJoinPoint pjp, Cacheable cacheable) throws Throwable {  
    ...
}

Rebuild and check logs:

[INFO] Join point 'method-execution(com.example.apects.Example$Value com.example.apects.Example.getValue())' in Type 'com.example.apects.Example' (Example.java:12) advised by around advice from 'com.example.apects.CachingAspect' (CachingAspect.java:22)

If you find the post helpful, please support me and:

How to change author in pushed commit

Vlad — Mon, 10 Oct 2022 21:36:42 +0000

What issue I've tried to solve

Some time ago, I set up a new VM to work on a new project, and after a few days of intensive work and multiple pushed commits, I figured out a typo in my email. Luckily, I've made commits to the dedicated feature branch, so I may fix it without disturbing the team

How to fix it

Open the terminal (console, command prompt, etc.) and go to the source code folder.
Switch to rebase mode

   ➜  git rebase -i HEAD~<N>

where <N> is amount of commits from HEAD.
Or you may specify commit hash or branch

   ➜  git rebase -i 0893420285a5813ddd4a2c86f40054fa4968c9cf
   ➜  git rebase -i master

Mark required commits to edit (with e letter)
Repeat required times

   ➜  git commit --amend --author="Vladislav Bochenin <vladislav.bochenin@email.com>" --no-edit
   ➜  git rebase --continue

Check if everything is ok

   ➜  git log

Push changes

   ➜  git push --force

If you find the post helpful, please support me and:

How to create limited-size directory

Vlad — Fri, 07 Oct 2022 14:37:56 +0000

What an issue I've got

Today I've got a task to test (and fix) how the application behaves during a backup when it does not have a space on the device.
As result, there shouldn't be any new files once the backup is failed with "no space left" error.

How did I investigate the issue

So far, I see the following options for how to reproduce the issue:

to have enormous application data to backup, either on some shared machine or generate the content locally
to create a limited-size directory and try to write a backup there

First option is time-consuming to reproduce it locally, so let's start with creating limited-size directories.

I have two target environments for testing: Windows 10 and Debian-based Linux.
I've chosen Ubuntu 18.04 because I have the VM already installed.

So, let's make a backup for the newly installed application and check its size.
And empty backup is 8.5Mb. It is suspicious the app has an 8Mb backup from scratch, and I will write it down to fix later, but I have another issue so far.

Windows

I haven't found a way to create a limited-size directory on Windows, but I can create a limited-size virtual disk and use it for my purposes.

Go to Disk management
Select your local disk and choose "Create VHD" in "Action"
Choose Virtual Disk file location and set disk size. For my pupose I need 10Mb (8Mb for backup plus some extra for partitions and system information) and I need "Fixed size".
Initialize the new disk (choose Master Boot Record)
Create new simple volume.
Next... Next... Next... and disk is ready

Wow!!! It took almost 7Mb from the original 10Mb for system data. WOW!!!

But... it is enough for my purposes.

Linux

Create file with size you need

➜  touch 10mbarea
➜  truncate -s 10M 10mbarea

Create file system with the file

➜  mke2fs -t ext4 -F 10mbarea
mke2fs 1.46.5 (30-Dec-2021)
Discarding device blocks: done                            
Creating filesystem with 10240 1k blocks and 2560 inodes
Filesystem UUID: 03249ba7-6abc-4afb-a496-b5857f529859
Superblock backups stored on blocks: 
        8193

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done

Create a directory you will use later

➜  mkdir 10mbup

Mount the file system to the folder

➜  sudo mount 10mbarea 10mbup 
[sudo] password for vbochenin:

And limited size directory is ready

➜  df -h 10mbup              
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      8.3M   14K  7.5M   1% /home/vbochenin/10mbup

Once limited size directories created, I may specify them as backup target directory and run the same empty backup.

If you find the post helpful, please support me and:

When I'm using JMH

Vlad — Thu, 06 Oct 2022 21:22:40 +0000

What an issue I've tried to solve

My task was to investigate the reason why typed enumeration occupies 13 Mb per instance in a customer environment. The typed enumeration is a domain object allowing to users configure enumeration type for custom fields.
The use case is reproducible once enumeration has around 1k elements.

So I may assume the following possible causes:

a memory leak
- keeping in memory some stale caches values
storing big files in memory
- the enumeration is uploaded by user as XML file, so some intermediate big object can be cached
duplicating of data (if so, then why?).
- multiple copies of same data sorted in cache for different keys

What I did to find the cause

First, I've rerun the scenario locally, made a heap dump, and analyzed it with MemoryAnalizer.
It is visible from the report below (Histogram -> Choose TypedEnumeration in filter -> List of object with outgoing reference -> Sort by Retained Heap), that we are storing duplicated data for each key in control2Options field.



Class Name                                                              | Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------------
TypedEnumeration @ 0xdedf8050                                           |           64 |     4,637,960
|- control2Options java.util.HashMap @ 0xdef01f68                       |           48 |     3,925,856
|  |- table java.util.HashMap$Node[32] @ 0xdf7d5b10                     |          144 |     3,925,808
|  |  |- [26] java.util.HashMap$Node @ 0xdf7d5ba0                       |           32 |       712,256
|  |  |- [4] java.util.HashMap$Node @ 0xdfc3b790                        |           32 |       712,256
|  |  |- [14] java.util.HashMap$Node @ 0xdf12a800                       |           32 |       356,128
|  |  |- [13] java.util.HashMap$Node @ 0xdf24eab8                       |           32 |       356,128
|  |  |- [22] java.util.HashMap$Node @ 0xdf3a9608                       |           32 |       356,128
|  |  |- [23] java.util.HashMap$Node @ 0xdfa2e9a0                       |           32 |       356,128
|  |  |- [11] java.util.HashMap$Node @ 0xdfa50ff0                       |           32 |       356,128
|  |  |- [6] java.util.HashMap$Node @ 0xdfb91928                        |           32 |       356,128
|  |  |- [5] java.util.HashMap$Node @ 0xdfd3ca10                        |           32 |       356,128
|  |  |- [0] java.util.HashMap$Node @ 0xde742630                        |           32 |         4,128
|  |  |- [19] java.util.HashMap$Node @ 0xdf38cad0                       |           32 |         4,128
|  |  |-  class java.util.HashMap$Node[] @ 0xd444d5a8                   |            0 |             0
|  |  '- Total: 12 entries                                              |              |              
|  |- <class> class java.util.HashMap @ 0xd4450dd0 System Class         |           40 |           144
|  '- Total: 2 entries                                                  |              |              
|- allOptions java.util.ArrayList @ 0xdedf8090                          |           24 |         8,040
.....
'- Total: 10 entries                                                    |              |              
TypedEnumeration @ 0xe0064a60                                           |           64 |     4,637,960
|- control2Options java.util.HashMap @ 0xe055ecb0                       |           48 |     3,925,856
|  |- table java.util.HashMap$Node[32] @ 0xe08dd030                     |          144 |     3,925,808
|  |  |- [4] java.util.HashMap$Node @ 0xe086f630                        |           32 |       712,256
|  |  |- [26] java.util.HashMap$Node @ 0xe08dd0c0                       |           32 |       712,256
|  |  |- [23] java.util.HashMap$Node @ 0xe053daf8                       |           32 |       356,128
|  |  |- [5] java.util.HashMap$Node @ 0xe084cd10                        |           32 |       356,128
|  |  |- [13] java.util.HashMap$Node @ 0xe08521f0                       |           32 |       356,128
|  |  |- [6] java.util.HashMap$Node @ 0xe08dc138                        |           32 |       356,128
|  |  |- [11] java.util.HashMap$Node @ 0xe09de780                       |           32 |       356,128
|  |  |- [14] java.util.HashMap$Node @ 0xe0a665c0                       |           32 |       356,128
|  |  |- [22] java.util.HashMap$Node @ 0xe0acc8c0                       |           32 |       356,128
|  |  |- [0] java.util.HashMap$Node @ 0xdfa1e7d0                        |           32 |         4,128
|  |  |- [19] java.util.HashMap$Node @ 0xe0aefcb0                       |           32 |         4,128
|  |  |-  class java.util.HashMap$Node[] @ 0xd444d5a8                   |            0 |             0
|  |  '- Total: 12 entries                                              |              |              
|  |- <class> class java.util.HashMap @ 0xd4450dd0 System Class         |           40 |           144
|  '- Total: 2 entries                                                  |              |              
|- allOptions java.util.ArrayList @ 0xe0064b08                          |           24 |         8,040
....
'- Total: 10 entries                                                    |              |

Lets rerun the scenario with the profiler turned on.
I've used JMC and Flight Records, and it showed me that the application occupies the extra memory during bootstrap.

Not exactly the place, but it reduced the scope significantly and after some code investigation, I found where the application is wasting memory.

Success? Nope, I still need to fix the issue and prove that our fix is fixing something.

This is where JMH is coming to the stage.

What we can do with the tool

JMH is a microbenchmark framework for Java.
It cares about memory alignments, method inlines, warmups, and all other stuff that makes benchmarks hard to write.
It also has multiple profilers you may use in your benchmarks.



- Profiler (org.openjdk.jmh.profile)
    |- ExternalProfiler (org.openjdk.jmh.profile)
    |   |- AbstractPerfAsmProfiler (org.openjdk.jmh.profile)
    |   |   |- DTraceAsmProfiler (org.openjdk.jmh.profile)
    |   |   |- LinuxPerfAsmProfiler (org.openjdk.jmh.profile)
    |   |   '- WinPerfAsmProfiler (org.openjdk.jmh.profile)
    |   |- AsyncProfiler (org.openjdk.jmh.profile)
    |   |- JavaFlightRecorderProfiler (org.openjdk.jmh.profile)
    |   |- LinuxPerfC2CProfiler (org.openjdk.jmh.profile)
    |   |- LinuxPerfNormProfiler (org.openjdk.jmh.profile)
    |   |- LinuxPerfProfiler (org.openjdk.jmh.profile)
    |   '- SafepointsProfiler (org.openjdk.jmh.profile)
    '- InternalProfiler (org.openjdk.jmh.profile)
        |- AsyncProfiler (org.openjdk.jmh.profile)
        |- ClassloaderProfiler (org.openjdk.jmh.profile)
        |- CompilerProfiler (org.openjdk.jmh.profile)
        |- GCProfiler (org.openjdk.jmh.profile)
        |- JavaFlightRecorderProfiler (org.openjdk.jmh.profile)
        |- PausesProfiler (org.openjdk.jmh.profile)
        '- StackProfiler (org.openjdk.jmh.profile)

One I've used is GCProfiler.
The profiler calculates memory allocations and garbage collections during benchmark execution.
Statistics it shows, I'm interested in, is gc.alloc.rate (MB/sec) and gc.alloc.rate.norm (B/op)
gc.alloc.rate shows how much memory the benchmark occupies during the trial, but gc.alloc.rate.norm is about how much memory was allocated during single benchmark execution.

Having investigation results in mind, I may assume that less memory the application allocated, less GC pressure and less data application is storing in cache.

Warning: the assumption is fair only for the issue above because the investigation showed application is duplicating data it may never use.

So I've ended up with the following plan:

write benchmarks for suspicious methods
run benchmarks with GCProfiler and get some numbers
improve code or fix the issue
run the benchmark again
analyze numbers
repeat until successful


 java
public class TypeEnumerationAllocationBenchmark {  

    @Benchmark  
    public Object newObject(TypedEnumerationBenchmarkExecution execution) {  
        return new BenchmarkTypeEnumeration(...);  
    }  

    @Benchmark  
    public Object getAvailableOptions(TypedEnumerationBenchmarkExecution execution) {  
        return new BenchmarkTypeEnumeration(...).getAvailableOptions(null);  
    }  

    @Benchmark  
    public Object getAllOptions(TypedEnumerationBenchmarkExecution execution) {  
        return new BenchmarkTypeEnumeration(...).getAllSortedOptions();  
    }  

    @Benchmark  
    public void preLoadEnumeration(TypedEnumerationBenchmarkExecution execution, Blackhole blackhole) {  
        BenchmarkTypeEnumeration enumeration = new BenchmarkTypeEnumeration(...);  
        blackhole.consume(enumeration.getAllSortedOptions());  
        for (String controlKey : execution.controlKeys) {  
            blackhole.consume(enumeration.getAvailableOptions(controlKey));  
        }  
    }  
}

In parallel, I would run the simplest performance benchmark for the nominal use cases to prevent dramatic performance degradation.


 java
public class TypeEnumerationBenchmark {  

    @Benchmark  
    public Object getAvailableOptions(Execution execution) {  
        return execution.enumeration.getAvailableOptions(null);  
    }  

    @Benchmark  
    public Object getAllOptions(Execution execution) {  
        return execution.enumeration.getAllSortedOptions();  
    }  

    public static void main(String[] args) throws Exception {  
        Options opt = new OptionsBuilder()  
                .parent(new CommandLineOptions(args))  
                .include(".*TypeEnumerationBenchmark.*")  
                .build();  

        new Runner(opt).run();  
    }  
}

How does it work

JMH works with an Annotation Processing API and generates multiple classes for each benchmark method (method marked with @Benchmark annotation).


 java
benchmark
    jmh_generated
        benchmark.jmh_generated.TypedEnumerationBenchmarkExecution_jmhType
        benchmark.jmh_generated.TypedEnumerationBenchmarkExecution_jmhType_B1
        benchmark.jmh_generated.TypedEnumerationBenchmarkExecution_jmhType_B2
        benchmark.jmh_generated.TypedEnumerationBenchmarkExecution_jmhType_B3
        benchmark.jmh_generated.TypeEnumerationAllocationBenchmark_getAllOptions_jmhTest
        benchmark.jmh_generated.TypeEnumerationAllocationBenchmark_getAvailableOptions_jmhTest
        benchmark.jmh_generated.TypeEnumerationAllocationBenchmark_jmhType
        benchmark.jmh_generated.TypeEnumerationAllocationBenchmark_jmhType_B1
        benchmark.jmh_generated.TypeEnumerationAllocationBenchmark_jmhType_B2
        benchmark.jmh_generated.TypeEnumerationAllocationBenchmark_jmhType_B3
        benchmark.jmh_generated.TypeEnumerationAllocationBenchmark_newObject_jmhTest
        benchmark.jmh_generated.TypeEnumerationAllocationBenchmark_preLoadEnumeration_jmhTest
        benchmark.jmh_generated.TypeEnumerationBenchmark_Execution_jmhType
        benchmark.jmh_generated.TypeEnumerationBenchmark_Execution_jmhType_B1
        benchmark.jmh_generated.TypeEnumerationBenchmark_Execution_jmhType_B2
        benchmark.jmh_generated.TypeEnumerationBenchmark_Execution_jmhType_B3
        benchmark.jmh_generated.TypeEnumerationBenchmark_getAllOptions_jmhTest
        benchmark.jmh_generated.TypeEnumerationBenchmark_getAvailableOptions_jmhTest
        benchmark.jmh_generated.TypeEnumerationBenchmark_jmhType
        benchmark.jmh_generated.TypeEnumerationBenchmark_jmhType_B1
        benchmark.jmh_generated.TypeEnumerationBenchmark_jmhType_B2
        benchmark.jmh_generated.TypeEnumerationBenchmark_jmhType_B3
    benchmark.TypeEnumerationAllocationBenchmark
    benchmark.TypeEnumerationBenchmark

So, once you run your benchmark, it will try to find the generated classes in a classpath and use them to run the actual method and collect results.

How to run it

There are the few ways how to run it.

The best one with more precise results is to write the main method, build a .jar file, and run it.


 java
public class Benchmark {

    ....

    public static void main(String[] args) throws Exception {  
        Options opt = new OptionsBuilder()  
                .jvmArgsAppend("-Djmh.separateClasspathJAR=true")  
                .parent(new CommandLineOptions(args))  
                .include(".*Benchmark.*")  
                .build();  

        new Runner(opt).run();
    }
}

You may also run the main method from IDE, but likely your IDE will add some agent into JVM, so it makes the result less precise.

If you don't need to have precise results but rather to see performance trends, you may run the test as a JUnit test.


 java
public class Benchmark {

    ....
    @Benchmark  
    public void benchmarkMethod(...) {
        ...
    }

    @Test  
    public void shouldRunBenchmark() throws Exception {  
        Options opt = new OptionsBuilder()  
                .include(".*Benchmark.benchmarkMethod*")  
                .build();  

        new Runner(opt).run();  
    }
}

I've chosen to run from IDE because I'm interesting in memory allocations (a small side effect from IDE agents) and I'm going to use it during bugfix (I don't want to switch between IDE and console).

Once you run it you will see something like


 bash
# JMH version: 1.35
# VM version: JDK 11.0.15, OpenJDK 64-Bit Server VM, 11.0.15+9-LTS
# VM invoker: C:\workdir\bin\amazon-correto\jdk11.0.15_9\bin\java.exe
# VM options: -ea -Didea.test.cyclic.buffer.size=1048576 -javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2022.2.1\lib\idea_rt.jar=50905:C:\Program Files\JetBrains\IntelliJ IDEA 2022.2.1\bin -Dfile.encoding=UTF-8 -Djmh.separateClasspathJAR=true
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: benchmark.TypeEnumerationAllocationBenchmark.preLoadEnumeration
# Parameters: (controlKeysCount = 10, enumsCount = 10, prefixesCount = 1)

# Run progress: 0.00% complete, ETA 00:08:20
# Fork: 1 of 5
# Warmup Iteration   1: 100917.316 ops/s
# Warmup Iteration   2: 105528.728 ops/s
# Warmup Iteration   3: 105569.479 ops/s
# Warmup Iteration   4: 104947.364 ops/s
# Warmup Iteration   5: 105071.139 ops/s
Iteration   1: 104137.948 ops/s
                 ·gc.alloc.rate:               1900.765 MB/sec
                 ·gc.alloc.rate.norm:          20120.000 B/op
                 ·gc.churn.G1_Eden_Space:      1906.794 MB/sec
                 ·gc.churn.G1_Eden_Space.norm: 20183.824 B/op
                 ·gc.churn.G1_Old_Gen:         0.002 MB/sec
                 ·gc.churn.G1_Old_Gen.norm:    0.018 B/op
                 ·gc.count:                    33.000 counts
                 ·gc.time:                     22.000 ms

Iteration   2: 104582.692 ops/s
....

# Run complete. Total time: 00:08:22

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                                                                            (controlKeysCount)  (enumsCount)  (prefixesCount)   Mode  Cnt        Score       Error   Units
TypeEnumerationAllocationBenchmark.preLoadEnumeration                                                10          1000                1  thrpt   10     104137.948 ±    31.242   ops/s
TypeEnumerationAllocationBenchmark.preLoadEnumeration:·gc.alloc.rate                                 10          1000                1  thrpt   10     1900.765 ±    48.356  MB/sec
TypeEnumerationAllocationBenchmark.preLoadEnumeration:·gc.alloc.rate.norm                            10          1000                1  thrpt   10  20120.000   ±    39.771    B/op
TypeEnumerationAllocationBenchmark.preLoadEnumeration:·gc.churn.G1_Eden_Space                        10          1000                1  thrpt   10     1906.794 ±    58.341  MB/sec
TypeEnumerationAllocationBenchmark.preLoadEnumeration:·gc.churn.G1_Eden_Space.norm                   10          1000                1  thrpt   10  20183.824   ± 40835.901    B/op
TypeEnumerationAllocationBenchmark.preLoadEnumeration:·gc.churn.G1_Old_Gen                           10          1000                1  thrpt   10        0.002 ±     0.027  MB/sec
TypeEnumerationAllocationBenchmark.preLoadEnumeration:·gc.churn.G1_Old_Gen.norm                      10          1000                1  thrpt   10       0.018 ±    0.012    B/op
TypeEnumerationAllocationBenchmark.preLoadEnumeration:·gc.count                                      10          1000                1  thrpt   10      33.000              counts
TypeEnumerationAllocationBenchmark.preLoadEnumeration:·gc.time                                       10          1000                1  thrpt   10      22.000                  ms
....

where:

Fork - single benchmark execution. Every method executed several times. The number could be changed with -f option. Default value is 5. 0 means no fork and could be useful during benchmark debugging.
Warmup Iteration - some benchmark execution for warmup. Warmup results will not be used as benchmark result and requires mainly to eliminate some internal Java compilations and optimizations
Iteration # - actual benchmark execution and intermediate results.

More execution options you may find here

If you find the post helpful, please support me and:

DEV Community: Vlad

Java 17 migration: bias locks performance regression

What issue do I need to solve

What I did to get some data

Digging deeper

What is a bias lock

How did I fix it

Conclusion

3 ways to know used memory

What task I would like to solve

java.lang.Runtime way

JMX way

Secret JMX way

Conclusion

Running JMH benchmark from Eclipse

What issue I'm trying to solve

JUnit 4 runners

Implementing Benchmark JUnit runner

Filtering

Conclusion

How to change already written code: Aspects

What an issue I'm trying to solve

Changes I've made before

Aspect oriented programming overview

Applying aspects for the caches problem

How to change author in pushed commit

What issue I've tried to solve

How to fix it

How to create limited-size directory

What an issue I've got

How did I investigate the issue

Windows

Linux

When I'm using JMH

What an issue I've tried to solve

What I did to find the cause

What we can do with the tool

How does it work

How to run it

`java.lang.Runtime` way