DEV Community: Miguel Ortega

Exploring maven incremental builds with maven-build-cache-extension

Miguel Ortega — Wed, 15 Feb 2023 14:27:25 +0000

With the release of maven 3.9.0, it is now possible to leverage the maven-build-cache-extension to benefit from incremental builds in your maven project.

This feature can improve build time (in your local workflow and your CI). It caches module builds, avoiding unnecessary and/or expensive tasks:

The idea of the build cache is to calculate key from module inputs, store outputs in cache and restore them later
transparently to the standard Maven core. In order to calculate the key cache engine analyzes source code, build flow,
plugins and their parameters. This allows to deterministically associate each project state with unique key and
restore
up-to-date (not changed) projects from cache and rebuild out-of-date(changed) ones. Restoring artifacts associated
with
a particular project state improves build times by avoiding re-building unnecessary modules.[...]

From plugin documentation

Let's dive into the easiest way to getting started.

Installing the extension

As any other maven extension, you can load it via .mvn/extensions.xml or by modifying your pom.xml.

If your using the .mvn/extensions.xml approach, you can use following configuration file:


<extensions xmlns="http://maven.apache.org/EXTENSIONS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://maven.apache.org/EXTENSIONS/1.0.0 http://maven.apache.org/xsd/core-extensions-1.0.0.xsd">

    <extension>
        <groupId>org.apache.maven.extensions</groupId>
        <artifactId>maven-build-cache-extension</artifactId>
        <version>1.0.0</version>
    </extension>

</extensions>

This solution offers more flexibility as you can configure the extension via a maven-build-cache-config.xml file stored in the .mvn folder. If no configuration file is found, sensible defaults will be used.

If you are using the pom.xml approach, you can add the plugin to your project->build->extensions.


<project>
    ...
    <build>
        <extensions>
            <extension>
                <groupId>org.apache.maven.extensions</groupId>
                <artifactId>maven-build-cache-extension</artifactId>
                <version>1.0.0</version>
            </extension>
        </extensions>
    </build>
    ...
</project>

Running maven with the extension

Cache miss

When running any¹ maven goal with the extension, you should see new information printed on your maven console

# Running a maven goal
mvn clean test
[...]
# Enabling cache and hash algorithm
[INFO] Cache configuration is not available at configured path C:\<REDACTED>\.mvn\maven-build-cache-config.xml, cache is enabled with defaults
[INFO] Using XX hash algorithm for cache
[...]
# For each module ...
[INFO] Going to calculate checksum for project [groupId=com.example, artifactId=demo]
[INFO] Scanning plugins configurations to find input files. Probing is enabled, values will be checked for presence in file system
[INFO] Found 3 input files. Project dir processing: 16, plugins: 8 millis
[INFO] Project inputs calculated in 58 ms. XX checksum [596f60b3f5056d7d] calculated in 25 ms.
[INFO] Attempting to restore project com.example:demo from build cache
[INFO] Remote cache is incomplete or missing, trying local build for com.example:demo
[INFO] Local build was not found by checksum 596f60b3f5056d7d for com.example:demo

We can see that there is a lot going on here. The extension:

detects default configuration with XX hashing algorithm
calculates module build execution checksum based on input files and plugin configuration
searches a corresponding build cache
If no cache is found, build continues (cache miss)

Build cache information is stored in your ~/.m2 repository, under a build-cache folder. This can be useful to debug any cache errors.

For every module a file buildinfo.xml (ref) is stored containing cache data (project input, maven execution) and produced artifacts.

Cache hit

Running the same command again will re-use the previously stored build.

# Running a maven goal
mvn clean test
[...]
# Enabling cache and hash algorithm
[INFO] Cache configuration is not available at configured path C:\<REDACTED>\.mvn\maven-build-cache-config.xml, cache is enabled with defaults
[INFO] Using XX hash algorithm for cache
[...]
# For each module ...
[INFO] Going to calculate checksum for project [groupId=com.example, artifactId=demo]
[INFO] Scanning plugins configurations to find input files. Probing is enabled, values will be checked for presence in file system
[INFO] Found 3 input files. Project dir processing: 13, plugins: 6 millis
[INFO] Project inputs calculated in 39 ms. XX checksum [596f60b3f5056d7d] calculated in 16 ms.
[INFO] Attempting to restore project com.example:demo from build cache
[INFO] Remote cache is incomplete or missing, trying local build for com.example:demo
[INFO] Local build found by checksum 596f60b3f5056d7d
[INFO] Found cached build, restoring com.example:demo from cache by checksum 596f60b3f5056d7d
[INFO] Skipping plugin execution (cached): resources:resources
[INFO] Skipping plugin execution (cached): compiler:compile
[INFO] Skipping plugin execution (cached): resources:testResources
[INFO] Skipping plugin execution (cached): compiler:testCompile
[INFO] Skipping plugin execution (cached): surefire:test

We can see that maven totally skips compile and test execution, re-using cached data.

This also work by a per-module cache, meaning that in a multi-module maven project, only affected modules will be built.

Performance improvements

I did some minor testing, by running mvn clean verify -DskipTests on two projects:

A minimal spring-boot app created via https://start.spring.io/
Building my current project: a multi-module projet with +15 modules, ranging from spring boot apps to common libs, generated code source, etc ...

💬 Note: this is not a serious benchmark, and is provided just to give an example and showcase usage. I'm purposely ignoring tests since i don't have the patience to run them on my current "low resource" laptop 😅

project	1st	2nd (no modification)	3rd (modifying one module)
minimal mono-module	~7.4 s	~0.9 s	`n/a`
real life multi-module	~120.2 s	~16 s	~25s

I saw x4-10 performance improvements on my build, that will be more significant if we take tests into consideration which we should always be doing anyway. I'm instantly sold 🚀.

Alternatives

Before testing this plugin I used mainly two techniques to speed up my maven build (Nicolas Fränkel has a nice write-up on this).

The first is the maven daemon: this project aims at faster maven builds by simplifying parallel builds. It does not provide incremental builds provided by this extension.

The good news is that it can by used in addition to this extension, for faster build extravaganza (check related github issue - targeted for release 0.10.x).

The second is the use of maven reactor options to build only specific modules. _Eg._running mvn -pl :module1 --also-make will build the specific module1 while also building required project² (ignoring the rest).

While this a neat trick that i use everyday in by local and CI build, it still does not benefit from the cache performance improvements provided by the extension.

I love that with this extension I can still benefit from these tools, and still gain on massive build improvements.

Troubleshooting

I found the two best ways to handle errors for now are:

Manually deleting cache folder, located at ~/.m2/build-cache
Disabling the extension via command line -Dmaven.build.cache.enabled=false

Conclusion

Leveraging maven-build-cache-extension in your Maven project can significantly improve build times by avoiding unnecessary or expensive tasks. Hopefully by following the steps outlined in this article, you can easily get started and benefit from faster build times in your local workflow and CI.

The extension is fairly new, there might be some edge cases to resolve, but it's a bright step in the right direction, while we wait for maven 4.

If you find any issues on the plugin, don't hesitate get involved, raise on issue on the dedicated Apache JIRA. This really feels like the future of maven ❤️.

📝 References

Github Repo: https://github.com/apache/maven-build-cache-extension
Official Docs: https://maven.apache.org/extensions/maven-build-cache-extension/

when running custom goals.
check this blog post or the
brand-new official docs

any "standard" maven goal, there seems to be some issues ↩
For more information, ↩

til: Generate a file with specific size in java

Miguel Ortega — Tue, 10 May 2022 20:53:09 +0000

Recently I wanted to implement a new feature in our codebase, that dealt with uploading a file via a simple upload or a multi-part upload to an AWS S3 storage. It was actually a fun feature to develop, I learned a lot about AWS SDK on uploading an object using multipart upload and using presign urls which can be extremely useful to delegate the upload responsibility (and the dedicated resources) to the client apps.

Testing this feature required the creation of dummy tests files with specific size in order to trigger (or not) the multi-part upload:

Committing dummy files to our git repository was a definite no-no.
Files should be thrown away after running the tests.
Files creation should as fast as possible.
The actual content of the file is not important.

First two requirements can be answered using Junit 5's TempDirectory extension¹

The other requirements could be met with sparse files:

Sparse files are files stored in a file system where consecutive data blocks consisting of all zero-bytes (null-bytes) are compressed to nothing. There is often no reason to store lots of empty data, so the file system just records how long the sequence of empty data is instead of writing it out on the storage media. This optimization can save significant amounts of storage space for other purposes.

source

The following code generate a sparse file, open the file for writing, seeks a given position and adds somes bytes at the end. It leverages the FileChannel class.

@Test
void example(@TempDir Path tempFolder) throws IOException {
    // The sparse option is only taken into account if the underlying filesystem
    // supports it
    final OpenOption[] options = {
            StandardOpenOption.WRITE, 
            StandardOpenOption.CREATE_NEW , 
            StandardOpenOption.SPARSE };

    final Path hugeFile = tempFolder.resolve("hugefile.txt");

    try (final SeekableByteChannel channel = Files.newByteChannel(hugeFile, options)) {
        // or any other size
        long giB = 1024L * 1014L * 1024L;
        channel.position(giB);

        // Write some random bytes
        final ByteBuffer buf = ByteBuffer.allocate(4).putInt(2);
        buf.rewind();
        channel.write(buf);
    }

    //Do something with my dummy file
}

Since we don't need to actually allocate disk space, large sparse files can be created in a relative short time, making a good fit
for quick "unit" testing.

Additional Resources:

This feature is still experimental ↩

Intro to Playwright Web Automation Framework in Java

Miguel Ortega — Mon, 31 Jan 2022 19:34:45 +0000

In a recent project I found myself looking for a web automation framework.

I had the following requirements:

A dev oriented framework: so no Cucumber, FitNess or other BDD Frameworks¹
Preferably java based: I've had success using Robot Framework with maven, and while integration isn't hard, it was not seamless to say the least.
Something easy to setup and "fast" (one can dream)

What is Playwright ?

Playwright is a new web application framework, created by Microsoft, allowing engineers to test their web applications with cross-browser support. Their stable Java API landed less than a year ago.

The feature list is huge just by reading the docs, here are some of them that I found particularly relevant:

Auto-waiting on elements
Network Interception
Support for tricky scenarios like shadow-dom, iframes
Test tracing via screenshots, videos and a specific tracing via a dedicated explorer
API Testing (similar to your good old REST client for preparing server side state in the system under test)
Code generation

and the list goes on and on ...

WebDriver vs DevTools Protocol

One particularity of Playwright is the use of the Chrome DevTools Protocol (CDP).

Usually Webdriver is the de-facto standard, acting as a middleman between the browser and the testing framework.

Using the DevTools protocol allows a more direct control of browser, cutting the middleman and enabling Playwright to do some neat tricks like interacting with the network (intercepting calls or emulating network conditions).
To my knowledge, it's the only framework in the JVM ecosystem using CDP.

Faster tests

Playwright comes with BrowserContexts feature. This means each test can be isolated from each other, as if every tab was in a new incognito mode.

By sharing the same browsers instance, we can also speed up our test suite by not paying the costly browser instanciation.

// Create a new incognito browser context
BrowserContext context = browser.newContext();
// Create a new page inside context.
Page page = context.newPage();

Where do I start ?

Let's start by adding the latest dependency to your project.

<dependency>
  <groupId>com.microsoft.playwright</groupId>
  <artifactId>playwright</artifactId>
  <version>${playwright.version}</version>
</dependency>

In the following sections we are going to talk about my favorite features.

Code generation

I found the easiest way to get started was to use de code generation tool.

# At the root of your project
mvn exec:java -e "-Dexec.mainClass=com.microsoft.playwright.CLI" "-Dexec.args=codegen https://google.com"

This command will download all the need browsers (see configuration if you are behind a corporate proxy).
Then it will open your testing tab and the Playwright inspector, generating the code corresponding to the given user interactions
(even OAuth authentication).

Maybe it's because I've been out of touch of web testing frameworks lately, but this blew me away 🔥².
It's a great developer experience when you can benefit from code generation tools and refactor the base code to your needs, adding needed assertions or refactoring to create readable abstractions via a Page Object Model.

Network interception

You can mock API endpoints, by handling requests in your test.

// Using custom testData to as a response to this API call
page.route("**/api/fetch_data", route -> route.fulfill(new Route.FulfillOptions()
  .setStatus(200)
  .setBody(testData)));

page.navigate("https://example.com");

Mocking (in this case a stub) is great for out-of-process dependencies that you have no control over.

Note: while writing this article I came across an excellent post describing when to mock,
clarifying also many related terms.

Tracing

Finally, I really loved the possibility to generate a trace of your testing script.

Browser browser = chromium.launch();
BrowserContext context = browser.newContext();

// Setup tracing options
context.tracing().start(new Tracing.StartOptions()
        .setScreenshots(true)
        .setSnapshots(true));
Page page = context.newPage();

page.navigate("https://playwright.dev");
context.tracing().stop(new Tracing.StopOptions().setPath(Paths.get("trace.zip")));

The generated zip can be viewed via a command line

mvn exec:java -e "-Dexec.mainClass=com.microsoft.playwright.CLI" "-Dexec.args=show-trace target/trace.zip"

or just by uploading it to https://trace.playwright.dev/.

In the upper section, you can see the timeline of your tests
On the left and center side you can see each action in your test
- with an attached screenshot before and after the action
- you can even see a little red dot indicating where the action took place (e.g. a clicked button)
On the right section you can see network call, console logs and your tests script source code

Mind. Blown. 🤯

Conclusion

Hopefully by now you have a nice overview of Playwright and what it can do for you.
We only scratched the surface but here are some take aways

Great cross-browser, cross-os, cross-language library
Simple setup and great developer experience for java
Very complete API and awesome tooling.

It may reconcile me with web automation frameworks, who would have thought !

You don't event have to use it for your UI tests, via the Request API you can just automate your REST calls to prepare your system for a manual test (e.g. if you have a staging environment that you reset regularly) and iterate from there. Or just test your API, if that fits your needs.

The only one missing key I found is a nice reporting output, existing only in the node implementation.

I hacked a project integrating Playwright with Junit5 and Allure Reporting framework - code is available [on Github] https://github.com/mikomatic/playwright-demo) - but an out-of-box reporting tool would make this incredible tool even greater.

til: SSH Connexion troubleshooting

Miguel Ortega — Sun, 30 Jan 2022 10:21:51 +0000

A few weeks ago several developers on our team were unable to connect to our Gitlab instance using the SSH protocol.
Their git commands kept asking for password with no apparent error.

It was a bit strange because nothing had changed in their setup (or at least, that's what they thought). Most of them recreated their ssh keys, which seemed to correct the problem, and we all moved on - it seemed a PEBKAC¹ issue.

I didn't gave it much thought until it happened to me too 😅.

The gitlab documentation offers a path forward:

# Options -T to Disable pseudo-tty allocation.
# Options -v to set verbose mode. Multiple -v options increase the verbosity.  
# The maximum is 3.
ssh -Tv git@example.com

# ouput
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user/.ssh/id_rsa RSA ... agent
debug1: send_pubkey_test: no mutual signature algorithm
debug1: No more authentication methods to try.

The no mutual signature algorithm when offering my public ìd_rsa indicates that ssh-rsa is not enabled.

For me this happened after updating my Git version, and after digging a bit I found out that Git updated its OpenSSH version to 8.8 since version 2.33.1. Furthermore, reading OpenSSH 8.8 release notes I found the root
cause (emphasis mine):

This release disables RSA signatures using the SHA-1 hash algorithm by default. [...]

For most users, this change should be invisible and there is no need to replace ssh-rsa keys. [...]

Incompatibility is more likely when connecting to older SSH implementations that have not been upgraded
or have not closely tracked improvements in the SSH protocol.
For these cases, it may be necessary to selectively re-enable RSA/SHA1 to allow connection and/or user
authentication via the HostkeyAlgorithms and PubkeyAcceptedAlgorithms options.

This seems to be exactly my case, as I was interacting with a server using OpenSSH 6.x.

Two solutions are offered. Either generate a new key using a more robust algorithm² (which I did)

$ ssh-keygen -t ed25519 -C "your_email@example.com"

or re-enable RSA SHA-1 support on the affected ssh client (not recommended)

#In ~/.ssh/config
PubkeyAcceptedAlgorithms +ssh-rsa

So while, the solution was actually the same as other teams members, i'm glad i actually understood the why.

Some References:

Troubleshooting Git on BitBucket support page
Github article on improving git protocol security

Problem Exists Between Keyboard And Chair ↩
Gitlab/Github recommend ED25519 keys ↩

Generate PDF Documents in your Spring Boot App with AsciidoctorJ

Miguel Ortega — Mon, 17 Jan 2022 08:41:45 +0000

In a recent project I worked there was a need to generate PDF documents.

The document itself required to display information about a complex domain hierarchical object, containing ~100+ attributes and other child objects.
Before the existing application came to place, most users were manually creating Word documents to handle this requirement (with all the copy-pasting and human errors that one can imagine).

Existing solutions

The JVM ecosystem offers many possibilities to generate printable documents, to name a few:

While these libraries certainly offer battle-tested solutions and many features (some have been around for ~20 years !) for small or even medium project I find them a bit excessive.
(Disclaimer: I only have experience with the first two, that you won't find in my CV 😀)

I also have found that either the documentation is outdated, the learning curve is steep, the code is very low-level, only commercial licence is available or a combination of all the above.

Why Asciidoc can be good choice

Asciidoc is a lightweight markup language, where you only focus on content rather than layout.
It's offer a mature ecosystem to writing articles, documentation, books, and so on, with output formats ranging from HTML to ePub (and of course PDF).
It can easily integrate images, diagrams, code.

I have been using it for years for the technical documentation of personal and non-personal projects.

Show me the code

If you are in a hurry you can check the code directly on Github

Integrating Asciidoctor into your app is as simple as adding a maven (or gradle) dependency

<dependency>
    <groupId>org.asciidoctor</groupId>
    <artifactId>asciidoctorj</artifactId>
    <version>${asciidoctorj.version}</version>
</dependency>

<dependency>
    <groupId>org.asciidoctor</groupId>
    <artifactId>asciidoctorj-pdf</artifactId>
    <version>${asciidoctorj.pdf.version}</version>
</dependency>

Now let's see how you can generate a document :

try (Asciidoctor asciidoctor = Asciidoctor.Factory.create()) { //<1>

    Attributes attributes = Attributes.builder()//<2>
          .attribute("pdf-theme", theme) // optional theme
          .attribute("doctype", "book")
          .attribute("icons", "font")
          .build();

    Options options = Options.builder()
          .backend("pdf")//<3>
          .attributes(attributes)
          .toFile(outputLocation.toFile()).build();

    asciidoctor.convert(asciidocContent, options);//<4>
}

Create Asciidoctor factory
Define common attributes (you can even define a theme, more on that later)
Set PDF backend
Do the actual conversion, where asciidocContent is a string containing a ASCIIDOC template

That's pretty much it !

The asciidocContent can come from a "static" file on your classpath (or anywhere really).
For more dynamic document, it is possible to use any templating engine (in our project we used mustache).

It is possible to provide customization to the default theme, while it is not as powerful as other solutions, it can be good enough for most needs.
Several examples can be found on github.

Themes can also be packaged as jar for easier distribution.
The documentation is quite detailed.

Caveats

While this solution has served me well, it does come with some limitations:

Customization can be quite limited depending on your needs.
While diagrams as text is very practical, it does currently need an external dependency (graphviz, mermaid) present on your PATH.
Last but not least, the dependency that wraps a JRuby runtime does not work well in UBER jars (nested jars)
- For Spring Boot apps, this can be solved via requiresUnpack option of spring-boot-maven-plugin. ➡️Documentation
- For Quarkus, this is not possible yet #issue

a quick demo project demonstrating the possibilities of this solution, with a custom theme,
is available on github.

DEV Community: Miguel Ortega

Exploring maven incremental builds with maven-build-cache-extension

Installing the extension

Running maven with the extension

Cache miss

Cache hit

Performance improvements

Alternatives

Troubleshooting

Conclusion

📝 References

til: Generate a file with specific size in java

Additional Resources:

Intro to Playwright Web Automation Framework in Java

What is Playwright ?

WebDriver vs DevTools Protocol

Faster tests

Where do I start ?

Code generation

Network interception

Tracing

Conclusion

Further reading

til: SSH Connexion troubleshooting

Some References:

Generate PDF Documents in your Spring Boot App with AsciidoctorJ

Existing solutions

Why Asciidoc can be good choice

Show me the code

Caveats

Further reading