DEV Community: Katrin Grunert

Signing and verifying SOAP messages with wss4j and Scala

Katrin Grunert — Tue, 29 Jun 2021 09:01:51 +0000

Signing and verfiying SOAP messages with wss4j and Scala

SOAP is not dead. It is an established, XML-based and mature messaging protocol that comes with built-in security mechanisms, integrity checks, content validation and much more. A lot of enterprises and corporations are using it ~~sadly~~ still.
Just recently, Vandebron had to implement a SOAP client to communicate with an external party.
This blog post will explain with code examples how we at Vandebron are signing and verifying SOAP messages for our latest SOAP client implementation.

For this process, we are using Apache's Web Service Security Library wss4j as it is a proven tool in the WSS context and provides, as a Java library, great interoperability with the programming language Scala.

Signing SOAP messages

Here we will take a look at the necessary steps to sign a SOAP message like this one:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Header/>
  <soapenv:Body>
    <heading>Hello World</heading>
    <body>I am just a test</body>
  </soapenv:Body>
</soapenv:Envelope>

To look after signing like this:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Header>
    <wsse:Security 
    soapenv:mustUnderstand="1" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
      <ds:Signature 
      Id="SIG-ec946953-2470-4689-ad2f-0c579e1e06e3" xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
        <ds:SignedInfo>
          <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
            <ec:InclusiveNamespaces PrefixList="soapenv" xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#"/>
          </ds:CanonicalizationMethod>
          <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
          <ds:Reference URI="#id-47817454-f6e2-470c-9109-870e7895e3e0">
            <ds:Transforms>
              <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
            </ds:Transforms>
            <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
            <ds:DigestValue>7KfPcTwDYWtLj4ZVWmWmVqX4IGwbBAAmUPigCdXdk4U=</ds:DigestValue>
          </ds:Reference>
        </ds:SignedInfo>
        <ds:SignatureValue>
          OBnbBWv8S70xDDn5uG++7cTRFa2Uz3D47oxTHuO163Y3/V7H35M1GHXbKaUDOHsgsfx3SdVmVi++ra06cpwJknzqoIQgDV9Qc0ydzfxljCqupPKBnfONDYJtihEE1jtQ0RP7OLzPVNUpgOgHqbLwJu2pRUA05ool+lxIs924OwPVPKyUryoYwWhwY1ttY4P+WY2L3ZqsH3fgoLCyjlvhDEAhsP9PCxsEzPSq3ECC55Nh7nqMoHPj2uNxonuMlPeYbrlMnwyiqEW8s3Sc+WmfiIOgekRE1AdNhpn3ARlO490nObQtXCU/TxeTfbh98TMbQRZWWyT4HuLS3fF6aeyD/Q==
        </ds:SignatureValue>
        <ds:KeyInfo Id="KI-e18395de-9a26-4cad-9501-7c6cf6c7c74a">
          <wsse:SecurityTokenReference wsu:Id="STR-daa47836-f1f9-4d71-95cc-b7bcc6051c84">
            <wsse:KeyIdentifier 
            ValueType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509SubjectKeyIdentifier" EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary">
              ox4ajWTdigy9oApTYs97CuCV/4k=
            </wsse:KeyIdentifier>
          </wsse:SecurityTokenReference>
        </ds:KeyInfo>
      </ds:Signature>
    </wsse:Security>
  </soapenv:Header>
  <soapenv:Body 
    wsu:Id="id-47817454-f6e2-470c-9109-870e7895e3e0" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
      <heading>Hello World</heading>
      <body>I am just a test</body>
  </soapenv:Body>
</soapenv:Envelope>

For implementing the steps of the blog post you will need:

a SOAP service you want to send messages to
documentation of that SOAP service that describes:
- signature algorithm
- canonicalization method
- digest algorithm
- key identifier type
a private key with which you will sign your messages
a certificate that is the counterpart of the private key
(optional) a pool of trusted certificates

Our private and public key pair are available in the PKCS#12-format (.p12 file extension). Check out this to learn more about this format and how to achieve it.
The pool of trusted certificates are in the PKCS#7 format (.p7b file extension).

First we have to setup the necessary dependencies:

   // in your build.sbt or project/Dependencies.scala
  // enabling signing and signature verification for SOAP messages
  lazy val webServiceSecurity = Seq(
    "org.apache.wss4j" % "wss4j"                    % "2.3.1" pomOnly (),
    "org.apache.wss4j" % "wss4j-ws-security-dom"    % "2.3.1",
    "org.apache.wss4j" % "wss4j-ws-security-common" % "2.3.1"
  )

  libraryDependencies ++= webServiceSecurity

Next, we continue with a scala representation of our certificate we are using for signing:

  import org.apache.wss4j.dom.WSConstants

  // algorithm configuration
  object SigningCertificate {
    val CanonicalizationMethodURI: String = "http://www.w3.org/2001/10/xml-exc-c14n#"
    val DigestAlgorithmURI: String        = DigestMethod.SHA256
    val SignatureAlgorithmURI: String     = "http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"
    val KeyIdentifierType: Int             = WSConstants.SKI_KEY_IDENTIFIER
  }

  case class SigningCertificate(keyStore: KeyStore, password: String) {
    require(
      keyStore.aliases().asScala.size == 1,
      s"Certificate of Keystore needs to have one alias but had ${keyStore.aliases().asScala.size}"
    )
    val alias: String = keyStore.aliases().nextElement()

    override def toString: String = s"SigningCertificate(alias=$alias)"
  }

In the documentation of the SOAP service that you want to call should stand some information regarding the canonicalization method, signature algorithm, digest algorithm, and the key identifier type. Those are algorithms and information that define the signing process and we explain roughly now.

Before signing a message it has to be canonicalized. "Canonicalization is a method for generating a physical representation, the canonical form, of an XML document that accounts for syntactic changes permitted by the XML specification" (from here). In our case, the Exclusive XML Canonicalization is used.

The digest algorithm is used to ensure the integrity of the message during the verification of a signature. The algorithm is used to calculate a hash of the signed message. It should be documented in the SOAP service documentation. Here we will use SHA256 as a hashing algorithm.

The signature algorithm describes how the message will be signed. It can be defined in the SOAP service documentation but in the worst case you can read this algorithm from the certificate itself by using keytool:

$ keytool -list -v -keystore signature.p12
Enter keystore password: ...

[...] # more information about the certificates

Signature algorithm name: SHA256withRSA # thats what we are after!

[...] # more information about the certificates

According to the keytool inspection we will use SHA256withRSA (http://www.w3.org/2001/04/xmldsig-more#rsa-sha256) for signing.

Last but not least, in our signature, a <KeyInfo> element is included. This element contains information about the public key of the sender (us) and is needed for the signature verification once the message is received (read more here). Since we have our public key provided we don't need to do much here. The KeyIdentifierType describes which form of key identifier is used to present the public key information.

Having all this information about our certificate in place, we build the mechanism to load in our signing certificate. For this, we create the object KeyStoreBuilder.

import java.io.{File, FileInputStream}

object KeyStoreBuilder {

  def loadSigningCertificate(signingCertificate: File, password: String): SigningCertificate = {
    val fis = new FileInputStream(signingCertificate)
    val ks: KeyStore               = KeyStore.getInstance("PKCS12")
    ks.load(fis, password.toCharArray)
    SigningCertificate(ks, password)
  } 
}

Bear in mind, that you probably don't want to version any sensitive information like private keys and passwords hard-coded or in any environment variables, so a safe mechanism for storing/fetching passwords and certificates (like Vault) should be in place.

With the signing certificate in place, we can actually start signing a message. The next code example contains quite some Java boilerplate from wss4j that is required to make the signing mechanism work.

To restrict the usage of Java classes to a small portion of our code we will firstly implement a conversion method .toElem inside of the companion object SigningService:

  import java.io.StringWriter
  import javax.xml.transform.{OutputKeys, TransformerFactory}
  import javax.xml.transform.dom.DOMSource
  import javax.xml.transform.stream.StreamResult

  import org.w3c.dom.Document

  import scala.xml.Elem

  object SigningService {
    implicit class RichDocument(document: Document) {
      private val tf = TransformerFactory.newInstance()

      def toElem: Elem =
        val transformer = tf.newTransformer()
        transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
        val stringWriter = new StringWriter()
        transformer.transform(new DOMSource(document), new StreamResult(stringWriter))
        scala.xml.XML.loadString(stringWriter.getBuffer.toString)
    }
  }

With that, we can convert any Document SOAP message representation back to the scala.xml supported Elem format.

class SigningService(signingCertificate: SigningCertificate) {

  // importing our conversion method
  import SigningService.RichDocument

  /**
    * REQUIRED, otherwise it will throw:
    *
    * org.apache.wss4j.common.ext.WSSecurityException:
    * You must initialize the xml-security library correctly before you use it.
    * Call the static method "org.apache.xml.security.Init.init();"
    * to do that before you use any functionality from that library
    */
  org.apache.xml.security.Init.init()

  private val documentBuilderFactory = DocumentBuilderFactory.newInstance()
  private val crypto: Merlin = getCrypto

  crypto.setKeyStore(signingCertificate.keyStore)

  def signElement(elem: Elem): Elem = {
    documentBuilderFactory.setNamespaceAware(true)
    // converting Elem to Document (Scala to Java conversion)
    val doc = documentBuilderFactory.newDocumentBuilder().parse(new InputSource(new StringReader(elem.toString())))

    // WSSecHeader wraps around the document we want to sign
    val header = new WSSecHeader(doc)
    header.setMustUnderstand(true)
    header.insertSecurityHeader()

    // start building Signature, use the (wrapper) header-instance
    val builder = new WSSecSignature(header)
    builder.setUserInfo(signingCertificate.alias, signingCertificate.password)

    // setting algorithms
    builder.setSignatureAlgorithm(SigningCertificate.SignatureAlgorithmURI)
    builder.setSigCanonicalization(SigningCertificate.CanonicalizationMethodURI)
    builder.setDigestAlgo(SigningCertificate.DigestAlgorithmURI)
    builder.setKeyIdentifierType(SigningCertificate.KeyIdentifierType)
    builder.setAddInclusivePrefixes(true)

    // signing the document!
    val signedDocument = builder.build(crypto)
    // conversion back to Elem
    signedDocument.toElem
  }

  private def getCrypto: Merlin = {
    val properties = new Properties()
    properties.setProperty("org.apache.wss4j.crypto.provider", "class org.apache.ws.security.components.crypto.Merlin")
    CryptoFactory.getInstance().asInstanceOf[Merlin]
  }
}

Wss4j is a library that maintains an internal state during a signing process, but to avoid confusion it can be summarized as:

WSSecHeader wraps around the document to be signed
the WSSecHeader instance header will be used as part of the WSSecSignature-Builder
the WSSecSignature instance builder gets configured with all necessary information, which algorithms are used for signing, digesting, canonicalization, which key identifier should be included. Those settings an vary from webservice to webservice.

The actual signing of the document, which is now nested like a matryoshka doll, is happening with the help of an instance of Crypto. Crypto will contain either a keystore or a truststore or even both. It needs to be specified in the crypto.properties file or a runtime which class of Crypto will be used.
The most common one is Merlin.
We have decided to specify its configuration during runtime, since it is more visible than a properties file. Nevertheless, the crypto.properties-file needs to exist in your resources folder neverthless otherwise you will get a following WSSecurityException:

  org.apache.wss4j.common.ext.WSSecurityException: No message with ID "resourceNotFound" found in resource bundle "org/apache/xml/security/resource/xmlsecurity"
  [... rest of stacktrace ...]
  Cause: java.nio.file.NoSuchFileException: crypto.properties

And that's it! The KeyStoreBuilder helps us to load a SigningCertificate and the SigningService uses this loaded certificate to sign SOAP messages.
A receiver of our SOAP message has all the necessary information in our signature to verify that this message has not been tampered with and we are the original sender.

This verification is something we should also do on our side for incoming messages. So let's take a look at how we can verify the signature of received messages.

Verification of SOAP messages

Verifying the signature of incoming messages is equally important to ensure that the connection is secure. A verification process will tell you if the message is coming from a trusted source and has not been tampered with.

As previously mentioned we need our source of truth, a pool of trusted public keys from all parties which will receive our SOAP messages. These build the basis of the trust store.

We will create a TrustedCertificates wrapper class in which we will load in the trust store and add this method to the KeyStoreBuilder.

case class TrustedCertificates(keyStore: KeyStore)

object KeyStoreBuilder {

    def loadTrustedCertificate(certificates: Seq[File]): TrustedCertificates = {
    val ks = KeyStore.getInstance(KeyStore.getDefaultType)
    // we just want the keystore to act as a truststore (only containing trusted certificates), so we initialize it empty
    ks.load(null, null)
    val cf = CertificateFactory.getInstance("X.509")
    certificates.foreach { file =>
      CloseableUtil.using(getClass.getResourceAsStream(file.getPath)) { fis =>
        val certPath = cf.generateCertPath(fis, "PKCS7")
        certPath.getCertificates.asScala.toList.foreach { certificate =>
          ks.setCertificateEntry(file.getName, certificate)
        }
      }
    }
    TrustedCertificates(ks)
  }
}

This trust store is under the hood also just a KeyStore, without containing a private key that requires a password, that's why we can initialize the KeyStore with null-parameters.

Now, the SigningService needs to be extended with this trusted certificates and a verifySignatureOf-method:

import java.io.StringReader
import java.util.Properties
import javax.xml.parsers.DocumentBuilderFactory

import org.apache.wss4j.common.crypto.{ CryptoFactory, Merlin }
import org.apache.wss4j.dom.engine.WSSecurityEngine
import org.xml.sax.InputSource

import scala.util.{Failure, Success, Try}
import scala.xml.Elem

class SigningService(signingCertificate: SigningCertificate, trustedCertificates: TrustedCertificates) {

    private val engine = new WSSecurityEngine()
    private val documentBuilderFactory = DocumentBuilderFactory.newInstance()
    private val crypto: Merlin = getCrypto

    crypto.setKeyStore(signingCertificate.keyStore)
    crypto.setTrustStore(trustedCertificates.keyStore)

    def verifySignatureOf(elem: Elem): Boolean = {
      documentBuilderFactory.setNamespaceAware(true)
      val doc = documentBuilderFactory.newDocumentBuilder().parse(new InputSource(new StringReader(elem.toString())))

      Try(engine.processSecurityHeader(doc, null, null, crypto)) match {
        case Success(_) => true
        case Failure(exception) =>
          // replace with proper logging
          println(
            s"Unsuccessful signature verification, it is most likely that the certificate used for signing is not in our Truststore: ${exception.getMessage}")
          false
      }
  }

  private def getCrypto: Merlin = {
    val properties = new Properties()
    properties.setProperty("org.apache.wss4j.crypto.provider", "class org.apache.ws.security.components.crypto.Merlin")
    CryptoFactory.getInstance().asInstanceOf[Merlin]
  }
}

And with that, we have completed our roundtrip of signing and verifying SOAP messages!

Here are gists, articles, and documentation that inspired and helped us to figure out the signing and verification process for our SOAP client. Feel free to check them out!

Sources

WSSecurityVerifier by Luis Wolff

WSSecuritySigner by Luis Wolff

Unit Tests from ws-wss4j

An Introduction to XML Digital Signatures

SOAP vs. REST

Building native images and compiling with GraalVM and sbt

Katrin Grunert — Wed, 07 Oct 2020 11:05:34 +0000

At Vandebron we organize a two-day long Hackathon every quarter, and a colleague and I took this chance to dig into the wonderful world of GraalVM.

I've first heard of GraalVM around two years ago when Oleg Šelajev toured through Java User Groups in Germany and held talks about GraalVM. Here is one from 2019 (not Germany, but Spain this time).

GraalVM promises a significant speedup in compile times and as I am working with Scala, which is notoriously known for its long compile times, this seems interesting. Furthermore, GraalVM provides functionality to build native executables. Meaning, an application can be run without a Java Virtual Machine (JVM).

Thanks to the Hackathon I finally took the time to get to know GraalVM a bit better. With this blog post, I want to share our findings, experiences, and results, as they might be helpful for you too!

What is GraalVM?

GraalVM is a high-performance JVM that supports efficient ahead-of-time (AOT) and just-in-time (JIT) compilation, but also allows non-JVM languages (e.g. Ruby, Python, C++) to run on the JVM. The ahead-of-time compilation feature is the base for creating native executable programs, meaning an application can be run independently from the JVM. Seeing the versatile features of GraalVM, it is worth looking a bit under its hood.

Actually, GraalVM is defined by three main technologies:

Graal compiler, a high-performance JIT-compiler that can make JVM applications run faster from within the JVM
SubstrateVM, includes the necessary components to run a JVM-app as a native executable ( Garbage Collector, Thread Scheduler, etc.)
Truffle Language Implementation Framework, the basis for the polyglot support from GraalVM

Our motivation for trying out GraalVM was tackling the pain points of Scala, Java projects, and microservices. Shipping microservices written in Scala as Docker containers to your production system comes with the cost that startup can be a bit slow, having JVM and Docker overhead, and that those containers can be fairly large, as the application can only be run with a JVM. See Building Docker images for more information.

During the hackathon, we were most interested in building native images for Scala applications. Hoping to reduce the size of our docker containers and reducing up the startup time.

Project setup

The project we worked on during the Hackathon is an API that should be used for applicants to submit their applications at Vandebron in the future. By exposing one endpoint through which a resume and contact information can be submitted.

It is also a good project to test out GraalVM, nothing too complex but also not as simple as "Hello World".

The full setup can be found on Github. But I'll summarise the used stack below. The project is built around the following libraries, no particular reason, simply because I like them.

cats for working with effects, such as IO
http4s for running the server
tapir for defining the endpoints
circe for JSON de/serialisation
pureconfig for reading config-files
logback for logging

The project can be run via sbt run and with Postman or similar a POST-request can be sent like so:

POST localhost:8080/api/v1/apply

{
    "email": "my@email.de",
    "name": "My Name",
    "phoneNumber": "+310123456789",
    "applicationBase64": "VGhpcyBjb3VsZCBiZSB5b3VyIGFwcGxpY2F0aW9uIQ=="
}

Response:
"*confetti* Thanks for handing in your application, we will get back to you within the next days! *confetti*"

Setup GraalVM with sbt

With this initial project setup in mind, GraalVM needs to be installed locally.

For the installation of GraalVM the setup guide can be followed.

After the installation sbt needs to know that not the regular JDK/JVM is used. This can be done with the java-home option on sbt bootup.
To make the path to GraalVM a bit more accessible and easy to use it can be exported as an environment variable.

export GRAAL_HOME=/Library/Java/JavaVirtualMachines/graalvm-ce-java8-20.1.0/Contents/Home
sbt -java-home $GRAALHOME

The path to GraalVM can vary depending on OS and installation. We followed the basic installation for macOS.

Now sbt using GraalVM can be verified with:

sbt -java-home $GRAALHOME
scala> eval System.getProperty("java.home")
[info] ans: String = /Library/Java/JavaVirtualMachines/graalvm-ce-java8-20.1.0/Contents/Home/jre

That means everything running in this sbt instance is getting compiled by GraalVM. Awesome!

The next step is to become strong and independent and learn how to run without an underlying JVM with the help of building native images.

Building native images

GraalVM ships with the GraalVM Updater (gu) to install the native-image on your machine.

$GRAALHOME/bin/gu install native-image

sbt-native-packager provides functionality to build packages efficiently (e.g. building Docker images) and added to that, it also provides support for building native images.
In order to build native images with sbt commands this plugin has to be added to the project:

// inside project/plugins.sbt
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.7.3")

And the GraalVMNativeImagePlugin needs to be enabled:

// inside build.sbt
enablePlugins(GraalVMNativeImagePlugin)

From within sbt it should be able to autocomplete and suggest graal-commands, e.g.:

sbt:apply-at-vdb> graalvm
graalvm-native-image:       graalvmNativeImageOptions

With that setup, native images are just a stone's throw away!

Disclaimer

The next three sections are not a write-up but rather the main steps we had to take to make the project work. This includes failing images and troubleshooting.
I want to keep this in because it might be interesting for others when they have to troubleshoot.
For the summary and happy path, you can jump directly to Roundup.

First try building a native image

Next up graalvm-native-image:packageBin can be run from within sbt. This might take a while (on our systems it took about a minute)

Some warnings start to pop up:

[error] warning: unknown locality of class Lnl/vandebron/applyatvdb/Main$anon$exportedReader$macro$24$1;, assuming class is not local. To remove the warning report an issue to the library or language author. The issue is caused by Lnl/vandebron/applyatvdb/Main$anon$exportedReader$macro$24$1; which is not following the naming convention.

[error] warning: unknown locality of class Lfs2/internal/Algebra$Done$2$;, assuming class is not local. To remove the warning report an issue to the library or language author. The issue is caused by Lfs2/internal/Algebra$Done$2$; which is not following the naming convention.

The library-specific warnings can be ignored for now. Ultimately it fails with:

Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException:
Discovered unresolved type during parsing: org.slf4j.impl.StaticLoggerBinder.
To diagnose the issue you can use the --allow-incomplete-classpath option.
The missing type is then reported at run time when it is accessed the first time.

Actually a good hint on where to start fine-tuning the GraalVM config:

// inside build.sbt
graalVMNativeImageOptions ++= Seq(
    "--allow-incomplete-classpath",
)

Some things like a StaticLoggerBinder only get resolved at runtime, meaning at build time the classpath needs to be allowed to be incomplete. This option allows resolution errors to be ignored at build time and only pop up during runtime.

During the build of a native image, GraalVM tries to resolve those runtime dependencies already at compile-time, as it is part of the Ahead-Of-Time-compilation process. With this flag, GraalVM knows "hey, don't worry about it now, we cross the bridge when we get there" (or something like that).

Adding resource files

A reload (or restart) of sbt is needed to activate these new options. And we can try to build the native image up new.
This time the build finished successfully and the executable file target/graalvm-native-image/apply-at-vdb has been created!
This is an executable that can be run without a JVM:

target/graalvm-native-image/apply-at-vdb

But what's that? It actually cannot be started...

target/graalvm-native-image/apply-at-vdb

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
*** An error occured! ***
Cannot convert configuration to a de.erewl.pricetracker.server.Config. Failures are:
at the root:
- Key not found: 'host'.
- Key not found: 'port'.

The first three lines relate to the error that occurred during the first build. It simply says that logging hasn't been set up correctly (maybe due to the absence of a src/main/resources/logback.xml or some other misconfiguration), triggering the default setting of not logging anything at all.
The second error states that a configuration file does not have the right keys or cannot be found at all.
Looking into src/main/resources:

ls src/main/resources/
application.conf logback.xml

and peeking into application.conf:

cat src/main/resources/application.conf
    host = "localhost"
    port = 8080

Hm, so everything is actually in place. But somehow GraalVM can't find those files.
It still requires some more GraalVM fine-tuning here.

By default, GraalVM doesn't include any resource or configuration-files.
The option -H:ResourceConfigurationFiles=path/to/resource-config.json defines a path to a JSON configuration file. So inside the resource-config.json we can include our application.conf and our logback.xml.

But writing those config files can be tedious and it is difficult in larger projects to find all necessary classes that need to be included. GraalVM provides some support with writing those files and actually does all the work. In the project's root directory a configs-folder can be created which will contain all necessary config-files.

For writing the configuration files we will build a normal JAR-file with the help of the sbt-assembly plugin. Adding it to the project like so:

  // inside project/plugins.sbt
  addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6")

The JAR-file will be built with sbt assembly.

With that we can now start the application, providing the path to the JAR-file that just has been created:

mkdir configs
$GRAALHOME/bin/java -agentlib:native-image-agent=config-output-dir=./configs -jar target/scala-2.12/apply-at-vdb-assembly-0.1.0-SNAPSHOT.jar

With the command above the JAR gets to run with GraalVM but adds dynamic lookups that are being intercepted during runtime and written to the files: jni-config.json, proxy-config.json, reflect-config.json and resource-config.json.

Those generated files can be included in the GraalVMNativeImageOptions:

// build.sbt
graalVMNativeImageOptions ++= Seq(
    "--allow-incomplete-classpath",
    "-H:ResourceConfigurationFiles=../../configs/resource-config.json",
    "-H:ReflectionConfigurationFiles=../../configs/reflect-config.json",
    "-H:JNIConfigurationFiles=../../configs/jni-config.json",
    "-H:DynamicProxyConfigurationFiles=../../configs/proxy-config.json"
)

The build with those updated options should succeed and the app can be run once again:

target/graalvm-native-image/apply-at-vdb

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Still no logging, sadly. But the server is actually running and responds to POST requests via its exposed endpoint:

POST localhost:8080/api/v1/apply

{
    "email": "my@email.de",
    "name": "My Name",
    "phoneNumber": "+310123456789",
    "applicationBase64": "VGhpcyBjb3VsZCBiZSB5b3VyIGFwcGxpY2F0aW9uIQ=="
}

Response:
"*confetti* Thanks for handing in your application, we will get back to you within the next days! *confetti*"

The next and last step will investigate why logging is not picked up by GraalVM.

Investigating the missing logging

So first I wanted to have a look if it was an overall issue with logging. I stepped back from using logging-framework and tried the most basic logging with the java-integrated java.util.Logging. GraalVM's docs stated that GraalVM supports any logging that depends on that.

Building and running the native-image with java.util.Logging instead of logback succeeded and everything is logged properly.

So it must be something with the dependencies?

For further investigation, I added the sbt-dependency-graph plugin and checked out the dependency-tree with sbt dependencyBrowserTree. The library logback wasn't included in the dependency tree.
Which is odd, since logback is clearly present in the project's library-dependencies.

// inside build.sbt
libraryDependencies ++= Seq(
    ...
    "ch.qos.logback" % "logback-classic" % "1.2.3" % Runtime,
    "ch.qos.logback" % "logback-core" % "1.2.3" % Runtime,
    ...
)

Having a closer look, the appendix % Runtime on logback's dependency is present.

Not sure where this was coming from but it is most probably blindly copy-pasted from somewhere when gathering the dependencies for this project.

sbt reference manual states that the appendix Runtime defines that this dependency will be only included in the runtime classpath.

So this explains probably why logging was only working when the server was run from inside sbt.

With removing this and building the native-image, logback appears in the dependency-tree, and logging works when the native image is executed!

This "bug" was interesting as it emphasized what GraalVM can NOT do for you. Dynamic class loading/linking can not be supported by GraalVM as classes and dependencies have to be present during compile time to make a fully functional application.

Roundup

A successful setup of sbt and GraalVM to build native-images requires to:

install GraalVM's native-image functionality via it's graal-updater:

  gu install native-image

add sbt-native-packager and sbt-assembly to sbt:

  // inside project/plugins.sbt
  addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.7.3")
  addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6")

enable the GraalVM-Plugin:

  // inside build.sbt
  enablePlugins(GraalVMNativeImagePlugin)

create a fat JAR and define which resource and configuration files should be intergated by intercepting look up calls during its execution:

  sbt assembly
  mkdir configs
  $GRAALHOME/bin/java -agentlib:native-image-agent=config-output-dir=./configs -jar target/scala-2.12/apply-at-vdb-assembly-0.1.0-SNAPSHOT.jar

fine-tune GraalVM with the following options and include the files that have been created in the previous step:

  // build.sbt
  graalVMNativeImageOptions ++= Seq(
    "--allow-incomplete-classpath",
    "-H:ResourceConfigurationFiles=../../configs/resource-config.json",
    "-H:ReflectionConfigurationFiles=../../configs/reflect-config.json",
    "-H:JNIConfigurationFiles=../../configs/jni-config.json",
    "-H:DynamicProxyConfigurationFiles=../../configs/proxy-config.json"
  )

build the native image with:

  sbt graalvm-native-image:packageBin

run the executable file without the need of java

  ./target/graalvm-native-image/apply-at-vdb

Even without benchmarking, you notice that the startup time is way faster than with a traditional JAR-file and the application is up and running almost instantly.

It is worth noting that the creation of a native image is a quite time-consuming process. For this project, it took between 1 and 2 minutes. This is, of course, something a CI/CD-Server like Jenkins would take care of but it has to be kept in mind.

With a working native-image, it is time to dockerize.

Building Docker images

In this section two Docker containers will be built. One, following the "normal"-java way and the other will be using the native-image to build a Docker-container without Java.

Before getting started with native images, a regular JAR-file and Docker image for comparison can be built.

With the sbt-assembly plugin you can create JAR-files with all of its dependencies (fat JARs).
sbt assembly creates this target/scala-2.12/apply-at-vdb-assembly-0.1.0-SNAPSHOT.jar which has a size of around 42MB:

 sbt assembly 
 ls -lh target/scala-2.12/apply-at-vdb-assembly-0.1.0-SNAPSHOT.jar

  ...  ...   42M   target/scala-2.12/apply-at-vdb-assembly-0.1.0-SNAPSHOT.jar

This application can be run locally via java -jar target/scala-2.12/apply-at-vdb-assembly-0.1.0-SNAPSHOT.jar with the prerequisite that Java is installed on that machine.

Creating the Docker image for this JAR-file can be done manually, but luckily sbt-native-package supports building regular Docker images out of the box, only the DockerPlugin needs to be enabled:

// build.sbt
enablePlugins(DockerPlugin)

sbt docker:publishLocal creates the Docker image apply-at-vdb.

docker images | grep apply-at-vdb
  apply-at-vdb  0.1.0-SNAPSHOT      f488d4c06f28    555MB

A whopping 555MB for a tiny app exposing one endpoint which JAR-file was only 42MB. But to run this JAR-file in a container, this container needs to ship with a JVM, and that's where the overhead lies.

With that Docker image and JAR-file as a reference, we can now look into how the native-image operates together with Docker.

GraalVM does not support cross-building, meaning an application cannot be expected to be built in a MacOS environment and run in a Linux environment. It has to be built and run on the same platform. With the help of Docker, the desired built environment can be provided.
The Dockerfile looks as follows:

FROM oracle/graalvm-ce AS builder
WORKDIR /app/vdb
RUN gu install native-image
RUN curl https://bintray.com/sbt/rpm/rpm > bintray-sbt-rpm.repo \
    && mv bintray-sbt-rpm.repo /etc/yum.repos.d/ \
    && yum install -y sbt
COPY . /app/vdb
WORKDIR /app/vdb
RUN sbt "graalvm-native-image:packageBin"

FROM oraclelinux:7-slim
COPY --from=builder /app/vdb/target/graalvm-native-image/apply-at-vdb ./app/
CMD ./app/apply-at-vdb

And can be run with:

docker build -t native-apply-at-vdb .

The Dockerfile describes to do the following:
The first docker container, as the name implies, is the builder. As a base image the official GraalVM image is used.

This image needs two more things, GraalVM's native-image command, and sbt, and this is what the two follow-up rows are providing. Once that's done, the project is copied into this container and the native image is built from within sbt.

The next steps bring the native executable into its own docker container.
As a base image, we use an Oracle Linux image and from our builder-container, we copy the native executable to this new container. The last step is that the app gets run on container startup.

docker run -p 8080:8080 -it native-apply-at-vdb starts the container and shows that everything is working just as before.

But what about the image size? Let's have a look.

docker images | grep apply-at-vdb
  native-apply-at-vdb       latest              17b559e78645        199MB
  apply-at-vdb          0.1.0-SNAPSHOT      f488d4c06f28        555MB

That is impressive! We created an app that is approx. 2.8 times smaller than our original app.

Summary

We learned how to set up a Scala project with GraalVM, what steps have to be taken to build a native image with GraalVM, and let it run inside a Docker container. We also received a good overview of what's possible with GraalVM and what's not.

The initial start and setup of GraalVM with sbt is pretty easy and straightforward. Getting GraalVM to compile an sbt project is nice and simple.

This Hackathon showed us that it is difficult and requires a lot of fine-tuning to integrate GraalVM into an existing project or product. At Vandebron we work with a complex stack of technologies including Spark, Kafka, and Akka which made it difficult to port the findings from this small toy service to one of our existing microservices. This made extensive troubleshooting in the Hackathon not possible.

All in all, GraalVM allows you to give up some Java overhead and create significant smaller Docker images. Sadly, this comes at the cost of giving up dynamic linking and class loading.
A silver lining is, that inside Scala's ecosystem this rarely a problem. Scala relies heavily on compile-time mechanisms for detecting bugs early and creating type-safe applications (read here but also see e.g. Scala's compiler phases).

Sources and Reading

Building Serverless Scala Services with GraalVM by Noel Welsh
Small & fast Docker images using GraalVM’s native-image by Adam Warski
Run Scala applications with GraalVM and Docker by @itseranga
Getting Started with GraalVM and Scala by Oleg Šelajev
Updates on Class Initialization in GraalVM Native Image Generation by Christian Wimmer
GraalVM's Reference Manuals