DEV Community: dansiviter

Distroless Alpine

dansiviter — Tue, 10 May 2022 19:53:39 +0000

In my day job, we've been using Google's Distroless images for some time. The benefits of this are well known: smaller image and attack surface. However, what we didn't expect is to still to have to deal with a significant amount of toil dealing with vulnerability triage (see Snyk output below for distroless/base). It could be argued our SDLC is a little, clunky and we're not quite ready for scratch (Graal, Golang, etc.,) images yet so what can we do in the interim? 🤔

That lead me to think about a distroless Alpine Linux image. A quick google shows I'm not the only one as a recent Medium article attests. But as we're mostly dealing with Java to get a working image by pulling library-by-library is somewhat of a faff. Fortunately, APKO to the rescue.

APKO is a tool to create lean images using Alpine with only the stuff we need.

Demo

At time of writing an Alpine 3.15 image was 5.57MB. Pretty small but still includes some things we don't need, such as sh shell. So with the help of the examples from APKO I came up with:

contents:
  repositories:
  - https://dl-cdn.alpinelinux.org/alpine/v3.15/main
  packages:
  - musl
  - zlib

accounts:
  groups:
  - groupname: nonroot
    gid: 10000
  users:
  - username: nonroot
    uid: 10000
  run-as: nonroot

os-release:
  version-id: '3.15'  # defaults to 3.16/edge

A few things to note here:

Using 3.15 repos, rather than edge used in the examples,
Adding a few packages:
- musl: Needed for pretty much everything,
- zlib: Required for JRE 17, crashes without it.
Using non-root user.

PS ...> docker run --rm `
  -v $pwd/:/app:rw `
  -w /app ghcr.io/chainguard-dev/apko:v0.3.3 `
  build rootless.yaml alpine:3.15-apko apko.tar
2022/05/10 09:21:44 loading config file: rootless.yaml
2022/05/10 09:21:45 apko (x86_64): building image 'alpine:3.15-apko'
...
PS ...>  docker load -i .\apko.tar 
Loaded image: alpine:3.15-apko

The result is a very small image:

REPOSITORY   TAG               IMAGE ID       CREATED         SIZE
alpine       3.15              0ac33e5f5afa   4 weeks ago     5.57MB
alpine       3.15-apko         5a3ea808f8ed   52 years ago    709kB

That's nearly 1/8th of the size (12.7%) and a grand total of just 2 packages (down from 14):

PS ...> docker scan alpine:3.15-apko

Testing alpine:3.15-apko...

Package manager:   apk
Project name:      docker-image|alpine
Docker image:      alpine:3.15-apko
Platform:          linux/amd64

✔ Tested 2 dependencies for known vulnerabilities, no vulnerable paths found.

That's much smaller than roughly equivalent distroless/base:

REPOSITORY               TAG       IMAGE ID       CREATED        SIZE
gcr.io/distroless/base   nonroot   555ca12a9222   52 years ago   20.3MB

That's over 28x larger and that does have potential issues:

PS ...> docker scan gcr.io/distroless/base:nonroot

Testing gcr.io/distroless/base:nonroot...

...
Package manager:   deb
Project name:      docker-image|gcr.io/distroless/base
Docker image:      gcr.io/distroless/base:nonroot
Platform:          linux/amd64

Tested 6 dependencies for known vulnerabilities, found 11 vulnerabilities.

I've trialled my new image with an existing project via JLink that's heavy on Netty and gRPC the image works great (with a small tweak to exclude grpc-netty-shaded due to grpc-java#9083).

What about Java?

Ok, so I don't want to use a JLink minimal JRE, what the difference between a Java 17 JRE:

REPOSITORY                          TAG         IMAGE ID       CREATED        SIZE
gcr.io/distroless/java17-debian11   nonroot     678ed8ce3ba5   52 years ago   231MB
alpine                              jre17-apko  d2302101850c   52 years ago   202MB

29MB less, that's what!

Conclusion

So, we can have a distroless Alpine image, that's even smaller than Google's with a smaller attack surface. 🔥

The maintainers of APKO highlight it's not released yet and liable to change, but it's a great start.

Opinionated take on Java Microservices Frameworks

dansiviter — Sun, 27 Jun 2021 19:21:03 +0000

I've just kicked off a spike within my team on what framework we should use going forward for our microservices by applying a bit of science. But, as I'm known to have somewhat strong opinions on these things, I'll be posting this post after we, as a team, have picked a way forward to not skew the results (too much!). Even so, I'll try and be objective. More of that later, first the problem statement...

We no longer want to manage our own infrastructure so we're packing our bags and heading to the cloud. We currently use Spring Boot but is there something more cloud native? We're predominantly a Java team, and it's enough upskilling to cloud without throwing in a new language into the mix.

So, in a nutshell, our requirements are:

Java,
Container friendly (lean, starts/stops quickly),
Easy to pickup,
...and unit test,
Promotes integration with cloud native services (observability, databases, networking, etc).

The Contenders

There are probably many, many more, but I think I've captured the front runners here (including their marketing tag line):

Dropwizard - "Java framework for developing ops-friendly, high-performance, RESTful web services."
Helidon - "Lightweight. Fast. Crafted for Microservices.". This comes in two flavours:
- SE - Lean reactive first framework,
- Microprofile (MP) - Implements the Microprofile specification for standards based API,
Micronaut - "A modern, JVM-based, full-stack framework for building modular, easily testable Microservice and Serverless applications."
Quarkus - "Supersonic Subatomic Java"
Spring Boot - "Spring Boot makes it easy to create stand-alone, production-grade Spring based Applications that you can "just run".". Two flavors will be trialled here:
- WebMVC: A more imperative style leveraging Servlet API under the hood,
- WebFlux: Reactive ReST API leveraging Netty,
Vert.x - "Reactive applications on the JVM."

Frankly, all of these tick the majority of requirements listed above and with some tweaks could get all of them without too much pain. So, we must pick another way of separating the wheat from the chaff:

Relative tests:
- Build speed - You might be doing this a lot so can be a real time drain,
- Start-up speed - Important for scalability and recovery,
- Image size - Small images generally have lower attack surface and moved around the K8s estate faster,
- Request Round Trip Time (RTT) - It's a ReST service so we need prompt responses,
Subjective test: How easy is it to develop with?

To aid for each one I'll develop a simple ReST service that has two endpoints and matching unit test:

/hello/{name} - This just simply returns Hello {name}!
/hello/error - This throws a custom RuntimeException which must be handled as a 400 Bad Request and return Oh no!.

This will build into an image using Jib and the same base adoptopenjdk/openjdk16:alpine. Where possible this will use the recommended testing unit approach to verify the endpoints.

Relatively Scientific

My methodology:

Build speed: just run maven, including jib:dockerBuild, 10 times and get the mean average. This will include a test for each of the endpoints using approach recommended in their documentation,
Image size: the reported size of the image from the above build from docker images,
Start-up speed: start the images 10 times and take the mean average,
RTT: call the /hello/bob endpoint using JMH using average time.

⚠️ As ever, take microbenchmarks with a pinch of salt. There are so many factors affecting performance that although these results may be indicative, they may also no be true for the types of workloads you have. Therefore, your mileage may vary.

Results:

Test	Dropwizard	Helidon MP	Helidon SE	Micronaut	Quarkus	Spring WebFlux	Spring WebMVC	Vert.x
Build Speed (s)	11.954	9.574	11.790	11.322	22.605	14.364	11.840	9.345
Image Size (MBy)	382	376	368	376	375	381	383	370
Start Speed (ms)	1,665	1,573.4	424.9	753.5	665.8	1,557.8	1,774.5	374.6
RTT (us/op)	1856.903 ± 47.301	1986.689 ± 34.877	1690.168 ± 80.618	1730.723 ± 64.688	1821.905 ± 124.261	1857.136 ± 82.583	2045.205 ± 78.650	1513.232 ± 74.767

Lets start with build times: Quarkus is just slow. I suspected it was the unit test (their recommended approach) or offline mode but even without either of them it, it was consistently slower than all the rest. It might also be their Jib wrapper, but as you're forced to use it not much can be done. The rest are all within a few of seconds of each other so not much to be garnered from that.

ℹ️ Quarkus was the only framework that mandated using it's own wrapper of Jib. I tried using Jib directly but kept complaining about missing pom.xml. This also meant that I had issues configuring offline mode causing build failures.

Next to build images: these all has the same base image and tool to try and get some comparable results and the winner was Helidon SE, which did surprise me as I was expecting Vert.x which did come in as runner up. In the middle with very similar results were Helidon MP, Quarkus and Micronaut with both Spring builds and Dropwizard at the tail. Frankly, they're all within a few MB so no clear winner. Maybe looking at using other tooling such as JLink or GraalVM would be beneficial to really slim them down, but not all of the frameworks support this.

Now to the start up times: Again, lean reactive frameworks were expected to be fastest here as they're avoiding bean lifecycle management... and they were here too: Vert.x and Helidon SE getting the two stop spots. For a framework that manages bean lifecycle Quarkus does a great job of starting quickly and picks up 3rd place, that might be due to their own CDI implementation: ArC. This is closely followed up by Micronaut. At the tail Spring WebFlux, Helidon MP and Dropwizard with Spring WebMVC at the rear but all within ~200ms between the also-rans. However, with none of them being over 2 seconds, we've have come a long way [Toto] from full-fat application servers!

Finally, lets review the RTT times; they are somewhat inconclusive. As anticipated the smaller, more functional reactive based APIs (Vert.x & Helidon SE) tend to be quicker than the more imperative approaches. The only outlier seems to be Spring Boot WebFlux, but I suspect that is due to the ApplicationContext managing the bean lifecycle. Either way from difference between the best and worst was just 531 microseconds (0.5 milliseconds). For ReST traffic, you're unlikely to notice any of difference unless you're serving HUUUUUGE amounts of traffic.

Subjectively Anecdotal

First and foremost: Quarkus is just a pain to work with compared to the others. The code is pretty clean and simple, but at every turn you need to take special steps to get it to build or bundle into an image and it's just painful. A few of the issues:

I hit the dreaded Docker Rate Limit during this, and for Jib is was easy to work around (i.e. --offline or using mirrors). However, with Quarkus, with their own Jib wrapper, that's not possible,
It behaves very differently between IDE and container, which makes debugging inconsistent. Their debug mode is their way of addressing it but just seems like an overengineered 'fix' to a problem that shouldn't exist,
The test harness is just plain slow compared to the rest,
Why do I need to use the maven plugin to add features/extensions? And it reformatted the pom.xml to use spaces (yes, I'm that Maverick Renegade that uses tabs!),
Why do I need to use custom tooling to create projects? Why isn't there Maven archetypes for this like everyone else.

The next most difficult was Vert.x, largely due to the test harness. It does concern me that you have to reach out to a 3rd party to get another library that works around all the problems created by your own test fixtures. Once up and running is was fairly straight forward to use but certainly not developer friendly at the start.

Functional/Reactive patterns have somewhat of a steep learning curve so a graceful, easy to follow API massively helps adoption. I couldn't call Spring Boot WebFlux graceful... in fact, quite the contrary, it's a total dog's dinner! Really verbose, confusing naming, difficult error handling and just looks horrible at the end. It took a fair while to be up and running, but I don't envy the person that has to maintain this.

The rest of them were all pretty straightforward and I couldn't materially differentiate between them.

Conclusion

ℹ️ Code for this is available here. If I've made some glaring omissions, please do point them out.

Lets start at the weaker end: The two that get the most column inches within blogs and news sites are the two I'd avoid:

Quarkus - It's just maddening and therefore frustrating to use. I don't understand why they need to engineer solutions to problems of their own making!? If I can't trust the behaviour locally will be the same as in test or in a container I can't trust it in production. About its only benefit is it starts quicker than other frameworks that implement Microprofile,
Spring Boot - They need to just put it in the bin and start again; WebFlux is difficult to use, WebMVC is a poor version of JAX-RS, XML config should have been dropped years ago and Spring Boot AutoConfiguration Magic is just non-sensical for anything other than a very basic application. I have had issues where totally unrelated things are initialising because it found a library on the classpath... this is too easy to do and a huge security concern.

I know many will disagree with me on this, especially if they've already bought into either of these, but I see no benefit of using them over the other options that are far more straightforward to use. This would probably be a different story if they had some significant benefit (e.g. simpler to use or significantly faster), but they don't.

I'm struggling to pick an outright winner here. At the upper end we have:

Helidon - Near the top in every test and it's flexible SE and MP modes (which can mix and match too) are a great feature,
Micronaut - Just really well thought out and simple to use,
Dropwizard - Biggest surprise here. Few bells and whistles, but gets the job done with little fuss,
Vert.x - Although somewhat difficult to test Vert.x gets credit for being really lean and just plain damn fast!

With all of these, you'll be up and running in very little time and require no special fixtures to build them into container images, but if I had to pick one it would be Helidon. I have been a fan for a while (and, full disclosure, also contributed code) and doing these tests has shown that it's certainly up there with the best. Project Loom and io_uring support are exciting which should generate some serious performance improvements. However, in the meantime I'm going to look much closer at Micronaut and Dropwizard, unless I raw performance which Vert.x would be high on my list.

As for my team, what did they pick? Micronaut.

Not another logger!

dansiviter — Fri, 05 Mar 2021 13:31:26 +0000

Java is not short of logging libraries: Log4J, SLF4J, JBoss Logger Manager, Flogger, Apache Commons Logger, Logback plus probably many more I've not heard of. There is the also much unloved java.util.logger (or JUL) which is part of Java since v1.4. Even with all these options I've been thinking that we need a new logger.

Step away from the pitch fork and hear me out.

Many of these libraries were created to address a particular concern, primarily performance:

Avoiding array allocation or auto-boxing causing memory churn,
IO performance to console or file,
Asynchronous execution,
Low/zero garbage.

With all these other libraries it's a really crowded space and it doesn't help with minimal microservices. Low/zero garbage would require a total re-write and if that's a concern stick with Log4J v2, but we can address the others... so I did: dansiviter.uk:juli.

Oh no, not another logging framework I hear you groan... well, kinda, but first lets see it in action.

Spring Clean

It's March, so what better time to clean up your code. The current way of using JUL would typically look something like this:

var log = Logger.getLogger(getClass().getName());
if (log.isLoggable(Level.INFO)) {  // <- needed to avoid array initialisation and auto-boxing
  log.log(Level.INFO, "Hello {0}", "world");  // Grrr, can't use #info(...) with params!?
}

This could be simplified to look like this:

var log = LogProducer.log(MyLog.class);
log.hello("world");

What is this heresy?! Well, I've created an interface which, via annotation processing, creates an implementation that delegates to JUL but layers on the #isLoggable(...) call automatically to avoids the dreaded array initialisation and auto-boxing unless it's strictly needed.

@Log
public interface MyLog {
  @Message("Hello {0}")
  void hello(String str);
}

To verify if we get a performance hit I wrote a small JMH benchmark comparing the two ways except forcing both auto-boxing and array initialisation in the scenarios:

Benchmark                                          Mode  Cnt      Score       Error  Units
LogBenchmark.legLog                               thrpt   25  27921.784 ±  1280.815  ops/s
LogBenchmark.legLog:Used memory heap              thrpt   25  26950.054 ±  8478.381     KB
LogBenchmark.newLog                               thrpt   25  40111.066 ±  1170.407  ops/s
LogBenchmark.newLog:Used memory heap              thrpt   25  25288.212 ± 10124.257     KB

Figures are based on my i7-1065G7 using Docker for Desktop so are purely indicative, however the new approach gives both a ~44% increase in throughput and a ~6% decrease in memory usage. I should mention that on repeat runs I did see some cases where memory usage was higher but throughput was consistently around 40% improvement.

This is great start, but we can go further.

Asynchronicity

Every single implementation of java.util.logger.Handler provided with Java is synchronous; it puts IO directly within the critical path of execution. Log4J and Logback have asynchronous implementations to decouple this and speed up the code, so why can't JUL? So, I created uk.dansiviter.juli.AsyncHandler to address this.

There is a concrete implementation of this which delegates to java.util.logging.ConsoleHandler. So I created some more JMH tests:

Benchmark                                               (handlerName)   Mode  Cnt      Score       Error  Units
ConsoleHandlerBenchmark.handler                        ConsoleHandler  thrpt   25  31041.836 ±  7986.031  ops/s
ConsoleHandlerBenchmark.handler:Used memory heap       ConsoleHandler  thrpt   25  37237.451 ± 15369.245     KB
ConsoleHandlerBenchmark.handler                   AsyncConsoleHandler  thrpt   25  85540.769 ±  6482.011  ops/s
ConsoleHandlerBenchmark.handler:Used memory heap  AsyncConsoleHandler  thrpt   25  41799.724 ± 16472.828     KB

For a ~12% increase in memory usage amount of memory usage we get a ~175% increase in throughput is great, however it's important to have some context with these results. This uses java.util.concurrent.Flow under the hood and it uses a default buffer size of 256. To prevent log messages being dropped backpressure is handled by blocking when the buffer is full. Therefore, buffer saturation will slow it down, hence the increased variance. However, in reality saturation will rarely occur and this will decouple IO from the log message creation improving performance. In smaller, less 'chatty' tests, massive (often incalculable) improvements are common:

x50 - Sync=PT0.018177S, Async=PT0S (NaN%)  // <- async so fast it didn't even register
x100 - Sync=PT0.0317765S, Async=PT0S (NaN%)  // <- ...and again
x200 - Sync=PT0.0644191S, Async=PT0.0009579S (6725.03400%)
x400 - Sync=PT0.1168272S, Async=PT0.0450558S (259.294500%)  // <- dramatic slow down
x800 - Sync=PT0.2164862S, Async=PT0.1705798S (126.912000%)
x1600 - Sync=PT0.4423355S, Async=PT0.4237862S (104.377000%)  // <- almost parity

See uk.dansiviter.juli.RoughBenchmark for code.

ℹ️ With modern containerised workloads a FileHandler is pretty pointless as they rarely have persistent disks to write to and generally write directly to STDERR/STDOUT or to the log aggregator directly. So I've not created an asynchronous version.

Summary

So, with this logging wrapper and asynchronous handlers I can have my cake an eat it:

Cleaner code,
Performance improvement (often huge!),
Minimal increase in binary size (~14KB).

However, I feel I should address the title of this post. No, this isn't a new logger, but it does make the existing logger a much nicer experience. Check out the README which explains more about how it's used including exception handling.

I'm already using this with a CDI project which injects instances into the code making unit testing super-simple. I plan on seeing if this plays nicely with JPMS and GraalVM soon for even more leanness.

I'm ready for my lynching now... do your worst!

Quest for the Holy Graal

dansiviter — Fri, 19 Feb 2021 19:05:01 +0000

Statically linked Java images are a superb idea; what's not to love about super small, super fast images that have a low attack surface. However, the reality is far from it. This post is describing some of the ~~fun~~ frustration I've had trying to get something rather basic working: namely Cloud Logging.

`UnknownHostException`

The first problem is statically linked Graal Native Images cannot do some things without libc set, in this case they can't perform a DNS lookup which results in a java.net.UnknownHostException. I'm not going to dwell on how stupid this is and just move on. There are rough instructions available on the Graal site to use musl for this but a quick Google came up with something from, erm, Google Cloud. I bodged this into Helidon and... it didn't build:

#11 274.5 [WARNING] Error: Detected a direct/mapped ByteBuffer in the image heap. A direct ByteBuffer has a pointer to unmanaged C memory, and C memory from the image generator is not available at image runtime.A mapped ByteBuffer references a file descriptor, which is no longer open and mapped at run time.   To see how this object got instantiated use --trace-object-instantiation=java.nio.DirectByteBuffer. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
#11 274.5 [WARNING] Trace: Object was reached by
#11 274.5 [WARNING]     reading field io.grpc.netty.shaded.io.netty.buffer.PoolChunk.memory of
#11 274.5 [WARNING]             constant io.grpc.netty.shaded.io.netty.buffer.PoolChunk@42d125b0 reached by
...

Googling

The build problem was due to the embedded Netty within the gRPC libraries. Fortunately the fix for this was pretty easy: Google have their own library for it:

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-graalvm-support</artifactId>
  <version>0.3.0</version>
</dependency>

With this added it built and ran, but now it couldn't find any GCP config:

java.lang.IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment.  Please set a project ID using the builder.
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
        at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:304)

Confuddling

Normally, I would have just mounted the volume into the users home, but it appears Graal doesn't have value for user.home. I've not digged into this to confirm it, so thought I'd just work around it:

docker run --rm -it -p 8080:8080 `
  -e CLOUDSDK_CONFIG=/gcloud `
  -v $Env:APPDATA/gcloud/:/gcloud:ro `
  helidon-quickstart-mp:native

Booom! It ran up without any errors.

Alas, there was one final hiccup which was answered by microsoft/WSL#5324. In a nutshell, if you let Windows go to sleep then the clock in WSL2 drifts. That meant I couldn't find any of my log messages as they were timestamped incorrectly. With WSL restarted we got some logs in Cloud Logging. Hurrah!

The resulting image is a somewhat portly 116MB which is 24MB larger than last time and only 11MB off Alpine JLink. I don't think could be explained by the extra libraries but something to worry about another day. However, it starts in around 160ms with -m 32m set on the docker container which can't be sniffed at. Regardless of how compelling Graal is, there are lots of bear traps to be aware of. I hope this improves soon.

Now for metrics and tracing...

Java 16 EA Alpine & JLink vs Graal

dansiviter — Sun, 17 Jan 2021 19:24:29 +0000

My first post... hurrah!

As the saying goes: good things come in small packages. For container images it means a smaller attack surface and faster to deploy and start-up. Unfortunately, Java hasn't been the best candidate for containerisation but that has changed with a few interesting developments:

JEP 386: Alpine Linux Port: There have been other Alpine builds of various quality, but now it's official,
JEP 282: jlink: The Java Linker: Leveraging the Module System to create builds with just the bits you use,
Graal Native Image: A statically linked Java image.

With Java 16 we can now use JLink with Alpine images to get all the next gen GCs (a-la ZGC and Shenandoah) goodness.

I thought it would be useful to compare and contrast them with a simple ReST service. For this I'm using io.helidon.archetypes:helidon-quickstart-mp:2.2.0 archetype as a base with some customisations to the Dockerfiles.

ℹ️ This is a Microprofile implementation so that means full-fat CDI, JAX-RS goodness which IMHO gives a good compromise between standards, developer productivity and scalability.

There were some changes required to the files.

For Dockerfile:

-FROM openjdk:11-jre-slim
+FROM openjdk:16-jdk-alpine

For Dockerfile.jlink:

-FROM maven:3.6.3-jdk-11-slim as build
+FROM openjdk:16-jdk-alpine as build
+RUN apk add --no-cache bash maven

-FROM debian:stretch-slim
+FROM alpine:3.13.0
+RUN apk add --no-cache bash

ℹ️ We must add bash to the Alpine JLink image which adds 2MB as Helidon is using a little script to start up. Ideally if productionising this I would aim to remove this dependency.

So, what do the images sizes look like using the Slim JDK image as a control:

Image	Size	Change
JDK 11 Slim (Debian)	220MB	100%
Alpine	338MB	154%
Alpine JLink	127MB	58%
Graal native image	94MB	43%

As expected Graal based on scratch image is very small followed by Alpine JLink, Slim and finally Alpine at the rear. Now for start up time:

Image	Time	Change
JDK 11 Slim (Debian)	4,493ms	100%
Alpine	3,310ms	74%
Alpine JLink	1,844ms	41%
Graal native image	80ms	2%

Again, not many surprises here except Graal is faster than I thought!

Thoughts

It's a very interesting time for Java and containers. Java can be lean, light and fast which still makes it relevant in a container native deployment.

Both JLink (gRPC#3522) and Graal have some issues; I'm especially concerned about the Serial GC in Graal so will be putting that under some stress soon to see if that confirms my suspicions. I'll also be good when some Java 16 JRE Alpine images appear as the JDK is too bloaty.

Jury is out if Graal or JLink is my preferred approach until I can stick load on them and see how they behave in Kubernetes but things are looking good.