DEV Community: Matthew Gilliard

How to build a CLI app in Java using jbang and picocli

Matthew Gilliard — Mon, 17 Aug 2020 00:00:00 +0000

Traditionally Java applications have been used for long-running processes - web application servers can run for days or weeks at a time. The JVM handles this well: Garbage Collection is efficient over huge amounts of memory, and Profile-Guided Optimization can make your code faster the longer it runs.

However, it’s perfectly possible to write short-lived apps too, and in this post I’ll show how to build a CLI app whose total runtime is just a couple of seconds. You can build sophisticated CLI tools in Java for data processing, connection to databases, fetching data from the web, or taking advantage of any of the Java libraries that you're used to.

I’ll use jbang for packaging and running the app, and picocli to handle argument parsing and output. The app will send an SMS using Twilio’s Messaging API, in a single Java source file less than 100 lines long.

To follow along you will need:

SDKMAN! - I wrote about why I love SDKMAN! Before.
a Twilio account. If you don’t have one already, get one for free at https://twilio.com/try-twilio.
your Twilio account credentials set as environment variables: TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN.

❗ Jbang

Jbang is a one-stop shop for making Java applications that can be run as scripts. It can download the required version of Java, add dependencies and create self-executing source files. It can also run Java code directly from the web, from github repos or even from Twitter (if you can fit your code into 280 characters).

Using SDKMAN! install jbang with: sdk install jbang.

For the impatient: Once you have jbang installed you can skip the rest of this post and run the code directly from GitHub with jbang https://github.com/mjg123/SendSmsJbang/blob/master/SendSms.java - the first time you run it, you will be reminded that running code from URLs is inherently unsafe - you can trust me here though 😉

To create a new script for a CLI app, in an empty directory run:

jbang init --template=cli SendSms.java

This creates a single file, SendSms.java which you can run directly with ./SendSms.java. This file is valid both as Java source and as a shell script, thanks to the first line:

//usr/bin/env jbang "$0" "$@" ; exit $?

This is a neat trick that works as Java since it’s a comment. As a shell script it runs jbang, passing the name of the source file and any arguments.

The second line shows how to add dependencies to a jbang script:

//DEPS info.picocli:picocli:4.2.0

This is the standard GROUP_ID:ARTIFACT_ID:VERSION format for dependencies, used by gradle and other build tools.

The rest of the file is normal Java code which is the “Hello World” for picocli, a library for creating CLI apps in Java.

The last thing to do with jbang is to create a project which works in an IDE. You can’t just open the SendSms.java file because jbang needs to download dependencies and add them to the classpath.

Find out how to launch your IDE from the command line. For me on Linux it’s idea.sh - for you it might be idea or eclipse or code or something else, depending on which IDE you like and how you installed it. The following command will create a project that your IDE can understand, open the IDE and then monitor the project for changes:

jbang edit --live=<YOUR IDE COMMAND> SendSms.java

This command will wait indefinitely, watching for changes to the project. For testing the app as you are working, I find it easiest to have another open terminal in the same directory. Make sure to open this after installing jbang to make sure that it is on your $PATH. Now let's get into the code.

⌨️ Picocli

Picocli is a library for creating CLI apps in Java. The goal for today is to modify the existing SendSms.java so that it actually sends SMS. The arguments to the script will be:

the to phone number (probably your cell number)
the from number (your Twilio number)
the message body can be specified at the end of the command

The phone numbers should always be in E.164 format. Running the script would look like this:

./SendSms.java \
    --to +4477xxxxxx46 \
    --from +1528xxxxx734 \
    Ahoy there 👋

If the message isn't at the end of the command, it will be read from standard in, which means you can use the script in a pipeline like this:

fortune | ./SendSms.java --to +4477xxxxxx46 --from +1528xxxxx734

(fortune is a UNIX command that prints "funny" quotes)

⁉️ Command line Options and Parameters

Picocli distinguishes between options and parameters :

Options have names, specified by flags, and can appear in any order: --to and --from are options.
Parameters are positional, the words of the message will be parameters if they're given at the end of the command.

Delete the private String greeting and its annotation, and add parameters and options for --to --from and the message:

@CommandLine.Option(
    names = {"-t", "--to"},
    description = "The number you're sending the message @|bold to|@",
    required = true)
private String toPhoneNumber;

@CommandLine.Option(
    names = {"-f", "--from"},
    description = "The number you're sending the message @|bold from|@",
    required = true)
private String fromPhoneNumber;

@Parameters(
    index = "0",
    description = "The message to send",
    arity = "0..\*")
private String[] message;

[this code on GitHub]

The first two highlighted lines include text surrounded by @|bold … |@ which tells picocli to format the words to and from.

For the message we specify the arity as 0..* which means "zero or more parameters", which will be gathered into an array of Strings. Note that if there are zero then message will be null rather than an empty array.

🚪 About exit codes

Skip over the main method - the action happens in the call method, which returns an Integer used as the script's exit code. An exit code of 0 means "success" and any other number means "failure". This script will exit with 1 for any kind of error. There are a few exit codes with special meanings, but none of those will apply to this script.

📩 Building the message text

If the message is given at the end of the command then it will be in the String[] message. If not, the array will be null and we need to read from Standard Input (stdin). Picocli doesn't have anything for reading from stdin but you can do it using a Scanner.

Remove the code from the call() method and replace it with this:

String wholeMessage;

if (message != null) {
  wholeMessage = String.join(" ", message);
} else {
  var scanner = new Scanner(System.in).useDelimiter("\\A");
  wholeMessage = "";
  if (scanner.hasNext()) {
    wholeMessage = scanner.next();
  }
}

if (wholeMessage.isBlank()){
  printlnAnsi("@|red You need to provide a message somehow|@");
  return 1;
}

[this code on GitHub]

If the message is non-null, join the parts together with spaces, otherwise use a Scanner to read from stdin. Using a delimiter of \\A with a Scanner will read the whole input as a single token - known as the Stupid Scanner Trick since at least 2004.

Finally, if there isn't a message from either source, print an error and exit with a 1. I added a printlnAnsi method above call() for including formatted output - it looks like this:

private void printlnAnsi(String msg) {
  System.out.println(CommandLine.Help.Ansi.AUTO.string(msg));
}

[this code on GitHub]

📲 Sending the SMS

Below the call() method, add this sendSMS method:

private void sendSMS(String to, String from, String wholeMessage) {

  Twilio.init(
    System.getenv("TWILIO_ACCOUNT_SID"),
    System.getenv("TWILIO\_AUTH\_TOKEN"));

  Message.creator(
    new PhoneNumber(to),
    new PhoneNumber(from),
    wholeMessage)
  .create();
}

[this code on GitHub]

This is all that's needed to send an SMS with Twilio. For it to work we need to add a dependency on the Twilio Java Helper Library, so add this line to the top of the file, underneath the picocli dependency:

//DEPS com.twilio.sdk:twilio:7.54.2

🛠️ Joining it up

The last thing to do is call sendSMS from call. Adding this code a the end of the call method will do just that:

try {
  System.out.print("Sending to ..." + toPhoneNumber + ": ");
  sendSMS(toPhoneNumber, fromPhoneNumber, wholeMessage);
  printlnAnsi("@|green OK|@");

} catch (Exception e) {
  printlnAnsi("@|red FAILED|@");
  printlnAnsi("@|red " + e.getMessage() + "|@");
  return 1;
}

return 0;

[this code on GitHub]

All being well, this will send the SMS. Any exception, such as bad credentials or using a non-existent phone number will put us in the catch block where we print the error in red and exit with a 1.

That's it! You can run the final script with:

./SendSms.java --to <YOUR CELL NUMBER> --from <YOUR TWILIO NUMBER> AHOY-HOY

...and the message will arrive.

🎁 Wrapping up

We've seen that Java isn't just useful for big web applications - it works just fine for short-running CLI applications too, made easier with jbang and picocli. This was just an example though - Twilio already has a very useful CLI already which can send SMS and do a lot more besides.

Just for fun I did also create a version of the code that would fit in a tweet.

I'd love to hear about the fun (or serious) CLI tools you're building with Java, don't worry if they're more than 280 characters. Find me on Twitter or by email:

🐦 @MaximumGilliard

📧 mgilliard@twilio.com

I can't wait to see what you build!

5 ways to make HTTP requests in Java

Matthew Gilliard — Thu, 23 Jul 2020 11:28:15 +0000

[header image credit: Iron in the Butterfly Nebula, NASA Astronomy Picture of the Day July 21 2020 (modified)]

Making HTTP requests is a core feature of modern programming, and is often one of the first things you want to do when learning a new programming language. For Java programmers there are many ways to do it - core libraries in the JDK and third-party libraries. This post will introduce you to the Java HTTP clients that I reach for. If you use other ones, that’s great! Let me know about it. In this post I’ll cover:

Core Java:

HttpURLConnection
HttpClient

Popular Libraries:

ApacheHttpClient
OkHttp
Retrofit

I’ll use the Astronomy Picture of the Day API from the NASA APIs for the code samples, and the code is all on GitHub in a project based on Java 11.

Core Java APIs for making Java http requests

Since Java 1.1 there has been an HTTP client in the core libraries provided with the JDK. With Java 11 a new client was added. One of these might be a good choice if you are sensitive about adding extra dependencies to your project.

Java 1.1 HttpURLConnection

First of all, do we capitalize acronyms in class names or not? Make your mind up. Anyway, close your eyes and center yourself in 1997. Titanic was rocking the box office and inspiring a thousand memes, Spice Girls had a best-selling album, but the biggest news of the year was surely HttpURLConnection being added to Java 1.1. Here’s how you would use it to make a GET request to get the APOD data:

// Create a neat value object to hold the URL
URL url = new URL("https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY");

// Open a connection(?) on the URL(??) and cast the response(???)
HttpURLConnection connection = (HttpURLConnection) url.openConnection();

// Now it's "open", we can set the request method, headers etc.
connection.setRequestProperty("accept", "application/json");

// This line makes the request
InputStream responseStream = connection.getInputStream();

// Manually converting the response body InputStream to APOD using Jackson
ObjectMapper mapper = new ObjectMapper();
APOD apod = mapper.readValue(responseStream, APOD.class);

// Finally we have the response
System.out.println(apod.title);

[full code on GitHub]

This seems quite verbose, and I find the order that we have to do things is confusing (why do we set headers after opening the URL?). If you need to make more complex requests with POST bodies, or custom timeouts etc then it’s all possible but I never found this API intuitive at all.

When would you use HTTPUrlConnection, then? If you are supporting clients who are using older versions of Java, and you can’t add a dependency then this might be for you. I suspect that’s only a small minority of developers, but you might see it in older codebases - for more modern approaches, read on.

Java 11 HttpClient

More than twenty years after HttpURLConnection we had Black Panther in the cinemas and a new HTTP client added to Java 11: java.net.http.HttpClient. This has a much more logical API and can handle HTTP/2, and Websockets. It also has the option to make requests synchronously or asynchronously by using the CompletableFuture API.

99 times out of 100 when I make an HTTP request I want to read the response body into my code. Libraries that make this difficult will not spark joy in me. HttpClient accepts a BodyHandler which can convert an HTTP response into a class of your choosing. There are some built-in handlers: String, byte[] for binary data, Stream<String> which splits by lines, and a few others. You can also define your own, which might be helpful as there isn’t a built-in BodyHandler for parsing JSON. I’ve written one (here) based on Jackson following an example from Java Docs. It returns a Supplier for the APOD class, so we call .get() when we need the result.

This is a synchronous request:

// create a client
var client = HttpClient.newHttpClient();

// create a request
var request = HttpRequest
    .newBuilder(URI.create("https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY"))
    .header("accept", "application/json")
    .build();

// use the client to send the request
var response = client.send(request, new JsonBodyHandler<>(APOD.class));

// the response:
System.out.println(response.body().get().title);

For an asynchronous request the client and request are made in the same way, then call .sendAsync instead of .send:

// use the client to send the request
var responseFuture = client.sendAsync(request, new JsonBodyHandler<>(APOD.class));

// We can do other things here while the request is in-flight

// This blocks until the request is complete
var response = responseFuture.get();

// the response:
System.out.println(response.body().get().title);

[full code on GitHub]

Third-party Java HTTP client libraries

If the built-in clients don’t work for you, don’t worry! There are plenty of libraries you can bring into your project which will do the job.

Apache HttpClient

The Apache Software Foundation’s HTTP clients have been around for a long time. They’re widely-used and are the foundation for a lot of higher-level libraries. The history is a little confusing. The old Commons HttpClient is no longer being developed, and the new version (also called HttpClient), is under the HttpComponents project. Version 5.0 was released in early 2020, adding HTTP/2 support. The library also supports synchronous and asynchronous requests.

Overall the API is rather low-level - you are left to implement a lot for yourself. The following code calls the NASA API. It doesn’t look too hard to use but I have skipped a lot of the error handling which you would want in production code and again I had to add Jackson code to parse the JSON response. You might also want to configure a logging framework to avoid warnings on stdout (no big deal, but it does irk me a bit). Anyway here’s the code:

ObjectMapper mapper = new ObjectMapper();

try (CloseableHttpClient client = HttpClients.createDefault()) {
    HttpGet request = new HttpGet("https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY");

    APOD response = client.execute(request, httpResponse ->
        mapper.readValue(httpResponse.getEntity().getContent(), APOD.class));

    System.out.println(response.title);
}

[full code on GitHub]

Apache provides several more examples for sync and async requests.

OkHttp

OkHttp is an HTTP client from Square with a lot of helpful built-in features, like automatic handling of GZIP, response caching and retries or fallback to other hosts in case of network errors as well as HTTP/2 and WebSocket support. The API is clean although there is no built-in parsing of JSON responses so again I added code to parse the JSON with Jackson:

ObjectMapper mapper = new ObjectMapper();

OkHttpClient client = new OkHttpClient();

Request request = new Request.Builder()
    .url("https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY")
    .build(); // defaults to GET

Response response = client.newCall(request).execute();

APOD apod = mapper.readValue(response.body().byteStream(), APOD.class);
System.out.println(apod.title);

[full code on GitHub]

This is fine, but the real power of OkHttp is clear when you add Retrofit over the top.

Retrofit

Retrofit is another library from Square, built on top of OkHttp. Along with all the low-level features of OkHttp it adds a way to build Java classes which abstract the HTTP details and present a nice Java-friendly API.

First, we need to create an interface which declares the methods we want to call against the APOD API, with annotations defining how those correspond to HTTP requests:

public interface APODClient {
    @GET("/planetary/apod")
    @Headers("accept: application/json")
    CompletableFuture<APOD> getApod(@Query("api_key") String apiKey);
}

The return type of CompletableFuture<APOD> makes this an asynchronous client. Square provide other adapters or you could write your own. Having an interface like this helps with mocking the client for tests, which I appreciate.

After declaring the interface we ask Retrofit to create an implementation which we can use to make requests against a given base URL. It’s also helpful for integration testing to be able to switch the base URL. To generate the client, the code looks like this:

Retrofit retrofit = new Retrofit.Builder()
    .baseUrl("https://api.nasa.gov")
    .addConverterFactory(JacksonConverterFactory.create())
    .build();

APODClient apodClient = retrofit.create(APODClient.class);

CompletableFuture<APOD> response = apodClient.getApod("DEMO_KEY");

// do other stuff here while the request is in-flight

APOD apod = response.get();System.out.println(apod.title);

[full code on GitHub]

API Authentication

If there are multiple methods in our interface which all need an API key it is possible to configure that by adding an HttpInterceptor to the base OkHttpClient. The custom client can be added to the Retrofit.Builder. The code needed to create the custom client is:

private OkHttpClient clientWithApiKey(String apiKey) {
  return new OkHttpClient.Builder()
    .addInterceptor(chain -> {
      Request originalRequest = chain.request();
      HttpUrl newUrl = originalRequest.url().newBuilder()
        .addQueryParameter("api_key", apiKey)
        .build();
      Request request = originalRequest.newBuilder()
        .url(newUrl)
        .build();
      return chain.proceed(request);
    })
  .build();
}

[full code on GitHub]

I like this kind of Java API for all but the simplest cases. Building classes to represent remote APIs is a nice abstraction that plays well with dependency injection, and having Retrofit create them for you based on a customizable OkHttp client is great.

Other HTTP clients for Java

Since posting this article on Twitter I have been delighted to see people discussing which HTTP clients they use. If none of the above is quite what you want, have a look at these suggestions:

REST Assured - an HTTP client designed for testing your REST services. Offers a fluent interface for making requests and helpful methods for making assertions about responses.
cvurl - a wrapper for the Java 11 HttpClient which rounds off some of the sharp edges you might encounter making complex requests.
Feign - Similar to Retrofit, Feign can build classes from annotated interfaces. Feign is highly flexible with multiple options for making and reading requests, metrics, retries and more.
Spring RestTemplate (synchronous) and WebClient (asynchronous) clients - if you’ve used Spring for everything else in your project it could be a good idea to stick with that ecosystem. Baeldung has an article comparing them.

Summary

There are a lot of choices for HTTP clients in Java - for simple cases I would recommend the built-in java.net.http.HttpClient. For more complex use-cases or if you want to have your HTTP APIs abstracted as Java classes as part of a larger application look at Retrofit or Feign. Happy hacking, I can’t wait to see what you build!

📧 mgilliard@twilio.com

🐦 @MaximumGilliard

Using SDKMAN! to work with multiple versions of Java

Matthew Gilliard — Fri, 06 Mar 2020 11:32:00 +0000

If you code in Java, you might be coding against Java 11, or 8, or maybe an even older version. You may also be prototyping code against newer versions of Java, the current version is 13. Java 14 will be released this month, and 15 later this year. At the same time, you might be investigating different builds of OpenJDK - there are several free alternatives. I use AdoptOpenJDK’s builds of OpenJDK which is an increasingly popular choice according to the JVM Ecosystem Report 2020.

You could also be trying out different build tools like Maven and Gradle. I use both, depending on the project.

Put bluntly, managing all this can be difficult. It’s not impossible to manage by hand, but it’s fiddly and if you get it wrong the error messages can be hard to understand.

Enter SDKMAN! It’s a tool for managing the installation and selection of Software Development Kits - not just different versions and builds of Java itself, but tools for building, debugging, monitoring, documenting and deploying too. It is available for Windows, Linux and MacOS.

Installing SDKMAN!

Follow the installation instructions for your platform. If you use the terminal a lot, I highly recommend checking out Oh My Zsh which can add a ton of useful information and behaviour. There is an SDKMAN! plugin for Oh My Zsh which adds tab completion to the sdk command which is really helpful. Enable this by finding the plugins= line in the .zshrc file in your home directory and adding sdk. Mine reads:

plugins=(git sdk)

If you do edit that file you’ll need to reload the Zsh config with source ~/.zshrc.

Installing a specific version of Java

Let’s say we want to install the latest build of Java 11 from AdoptOpenJDK. First of all let's check that it’s available, using sdk list java. If you’ve set up Oh My Zsh this is tab-completable:

So the latest Java 11 OpenJDK build from AdoptOpenJDK has the identifier 11.0.6.hs-adpt. Install it with:

sdk install java 11.0.6.hs-adpt

SDKMAN! will download that version of Java, and unzip it into a directory on your computer. You don’t need to worry about exactly where, because SDKMAN! will also update environment variables so that you can use java in the terminal immediately:

➜ java -version
openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.6+10, mixed mode)

Switching between Java versions

I’ve also installed the latest early-access version of Java 14 (due to be released March 17th 2020) so that I can try out some of the new features. I did that with:

sdk install java 14.ea.36-open

I said “no” when it asked if I wanted to use that as the default version. Later on, switch between them using sdk use java <version>:

I think you would agree that this is a lot easier than managing downloads, installations and environment variables by hand.

You can see what version you are using right now with sdk current:

➜ sdk current Using:java: 14.ea.36-open

Installing other tools

For different projects I use either Maven or Gradle as my build tool, and for debugging and profiling I sometimes use VisualVM. Install the latest versions of these with:

sdk install maven 3.6.3
sdk install gradle 6.2.2
sdk install visualvm 2.0

And they’re instantly available.

Using SDKMAN! with an IDE

To select a particular version of Java for a project in IntelliJ IDEA or Eclipse, you will need to know exactly where SDKMAN! has unpacked your Java installations. This is $HOME/.sdkman/candidates/java, with subdirectories named after the versions. The currently selected version of Java will also be available as current in that directory:

➜ ls -l ~/.sdkman/candidates/java
total 8
drwxr-xr-x 10 mjg mjg 4096 Jan 15 12:14 11.0.6.hs-adpt
drwxrwxr-x 8 sdk mjg 4096 Mar 4 12:43 14.ea.36-open
lrwxrwxrwx 1 mjg mjg 52 Mar 4 12:35 current -> /home/mjg/.sdkman/candidates/java/14.ea.36-open

I recommend keeping things clearer by configuring projects in your IDE to use a specific installation, rather than “current”.

For IntelliJ IDEA this can be done from the “Project Structure” dialog. Either choose an existing version from the drop-down, or add a new Java version by selecting “New…” from the “Project SDK” section:

For Eclipse, this is done by heading to “Project Properties”, then “Java Build Path”, “Libraries” and selecting “Modulepath”:

You should select the version that’s there already (11.0.6.hs-adpt in this example) and remove it, then select “Modulepath” again, “Add Library” and select “JRE System Library” from the dialog:

In the “Add Library” dialog, select “Alternate JRE” and click “Installed JREs” to open another dialog which lets you choose an existing version, or add a new Java installation by browsing to where it is installed:

Happy Hacking!

Using SDKMAN! can save you a lot of time and trouble if you work with multiple versions of Java and related tools. Maybe use that time to check out how to build cool things with Java and Twilio like a WhatsApp bot which does image recognition or sending daily SMS reminders?

I’d love to hear what you’re building, tell me about it on Twitter @MaximumGilliard or by email on mgilliard@twilio.com, or post it on Reddit to /r/twilio for the whole world to see.

Getting Started with the Java Streams API

Matthew Gilliard — Tue, 02 Jul 2019 12:10:09 +0000

The Streams API was added in 2014 with the release of Java 8 so you’ve almost certainly got it available today. It is used to pass a series of objects through a chain of operations, so we can program in a more functional style than plain iteration allows. Still, when working with collections many developers still reach for the classic for loop.

In this post, I’ll introduce the terms used when talking about Streams, show some examples of each term and how they can be used together to create compact and descriptive code. Then I’ll show a real-world example of Streams code I wrote recently to pick winners in a raffle.

What’s a Stream?

A Stream is a (possibly never-ending) series of Objects. A Stream starts from a Source. The objects in a Stream flow through Intermediate Operations , each of which results in another stream, and are gathered up at the end in a Terminal Operation. Streams are lazy, which means the source and intermediate operations do nothing until objects are needed by the terminal operation.

If we were making a fruit salad using the Streams API it might look like this:

Streams vs Collections

At first look, Streams may appear similar to Collections - here's a few differences:

Streams

may be infinite
do not allow you to directly access individual items
may include a way to create more elements on demand
can only be read once

Collections

designed for holding a finite number of existing objects
provide efficient ways to access individual items
may be accessed as many times as you need to

The ways of using Streams and Collections are quite different. As you will see below, it’s straightforward to create a Stream from a Collection and vice-versa.

Stream Sources

A Stream Source is something which creates a Stream, so what kinds of sources are there?

Collections

A common source for a Stream is a Collection, like a List or a Set (you cannot stream a Map directly, but you can stream its entrySet). Here’s a Stream from a List:

Stream<String> typesOfFruit = 
    List.of("apple", "banana", "clementine", "dragonfruit").stream();

I/O

I/O (or Input/Output) concerns moving data in or out of your application, for example reading data over a network, or reading a file from disk. A Stream of lines can be made from a File like this:

Stream<String> lines = Files.lines(Path.of("my-fruit-list.txt"));

Generators

A Stream can be generated from scratch by providing the contents up-front using Stream.of(...), or providing a function which returns each object for the Stream with generate or iterate, for example:

Random random = new Random();
Stream<Integer> randomNumbers = Stream.generate(random::nextInt);

The Stream class also has concat for combining multiple Streams into one.

Intermediate Stream Operations

An Intermediate Operation is something which takes a Stream and transforms it into another Stream, possibly containing a different type of Object. Each element in the original Stream will be passed one at a time to the operation, and depending on how the operation works the resulting Stream may be the same length as the original, but could also be shorter or longer:

// Same length: returns a Stream of "APPLE", "BANANA", "CLEMENTINE", "DRAGONFRUIT"
typesOfFruit.map(String::toUpperCase);

// Shorter: this returns a Stream of "clementine", "dragonfruit"
typesOfFruit.filter(fruit -> fruit.length() > 7);

// Longer: returns a Stream of "a", "p", "p", "l", "e", "b", "a" ...
typesOfFruit.flatMap(fruit -> Stream.of(fruit.split("")));

Notice the way that .flatMap(...) works is interesting. The lambda returns a Stream<String>, so if this were a regular .map(...) we would have a Stream<Stream<String>>. What flatMap does is join the inner Streams together to make a Stream<String> again.

Remember that you can only traverse a Stream once, so you won’t be able to do all three of those operations on the same typesOfFruit Stream. If you try it you will see java.lang.IllegalStateException: stream has already been operated upon or closed.

Chaining Intermediate Operations

The superpower of the Streams API, is that Intermediate Operations work on a Stream and return another Stream, so they can be chained together :

// These operations return a Stream of all the letters in the fruit names which 
// are after "G" in the alphabet, uppercased: "P", "P", "L", "N" ...
typesOfFruit
    .map(String::toUpperCase)
    .flatMap(fruit -> Stream.of(fruit.split("")))
    .filter(s -> s.compareTo("G") > 0);

If you try writing that without using Streams it will take far more code. You’ll have to create a lot of variables to hold intermediate values and the result will be far less compact and easy to read.

Terminal Stream Operations

Remember that Streams are lazy , which means that Sources and Intermediate Operations don’t do any work until they need to. It’s the Terminal Operations that create that need.

Another way to say that is that Terminal Operations turn a Stream back into something else. Until you need that something else the Stream just sits there, chilling.

There are lots of possible Terminal Operations including Collectors for gathering the Stream back into a Collection like a List or a Map, .foreach(...) to run some operation on each item and methods like .count() and .reduce(...) to compute some value from the Stream. Here’s a few examples:

// Collecting the Stream elements into a List
List<String> fruitCount = typesOfFruit.collect(Collectors.toList());

// Doing something with each element of the Stream
typesOfFruit.forEach(System.out::println);

// How many elements are in the Stream?
long fruitCount = typesOfFruit.count();

A Real-World Example of Streams

By now you’ve probably thought of lots of ways you could use Streams in your own code, but I’d like to end with an example where I used Streams to pick winners for a raffle we decided to run on the Twilio booth at JBcnConf. Folks could enter by sending an SMS to a phone number I manage with Twilio and we needed to pick three people to win space-themed Lego. The Streams API made this code pretty nice to write.

The code had to:

Fetch all the messages we received, using the Twilio Java helper library
Remove entries which arrived too early or too late
Grab the number each message was sent from
Remove duplicates so that each person has an equal chance of winning
Randomly shuffle the numbers
Take only as many numbers as we had prizes to give
Send an SMS to each winner

This looks like a case for the Streams API:

Here’s the code (it’s also on GitHub):

First, fetch all the SMSs sent to the CONTEST NUMBER:

Twilio.init(System.getenv("ACCOUNT_SID"), System.getenv("AUTH_TOKEN"));
ResourceSet<Message> allMessages = Message.reader().setTo(CONTEST_NUMBER).read();

ResourceSet is an Iterable so we can use it as a Source via the StreamSupport class in java.util.stream. I thought Iterable might have its own .stream() method but it doesn’t:

StreamSupport.stream(allMessages.spliterator(), false)

Now for the Intermediate Operations :

.filter( m -> m.getDateSent().isAfter(DRAW_START))
.filter( m -> m.getDateSent().isBefore(DRAW_END))
.map(Message::getFrom) // returns a PhoneNumber
.distinct()
.collect(Collectors.collectingAndThen(toList(), list -> {
    Collections.shuffle(list);
    return list.stream();}))
.limit(NUMBER_OF_WINNERS)

The trickiest operation is the shuffle (lines 5-7). Although Streams have .sorted(), they don’t have a .shuffled() so I used a .collect(...) as an Intermediate Operation by collecting into a List, shuffling it and returning a Stream of that List.

At this point we have a Stream<PhoneNumber> with the right number of items in it. Finally, the Terminal Operation takes each phone number and uses the Twilio API to send a message someone will be happy to receive:

.forEach( winner -> {
    Message.creator(winner, CONTEST_NUMBER, CONGRATULATIONS_YOU_WON_MESSAGE)
           .create();
});

The code is on GitHub including the Maven config.

Wrapping up

The Streams API can help you write shorter and more direct code for dealing with bulk operations on Collections and more. In this post I’ve shown some of the basics - there’s more to Streams, such as Parallel Streams, Primitive Streams (Streams of int, double or long) which can help with performance.

If you find some code which could be improved with Streams your IDE might even suggest refactorings for you, and I’d love to hear how you get on.

@MaximumGilliard

mgilliard@twilio.com

How to read and understand a Java Stacktrace

Matthew Gilliard — Mon, 03 Jun 2019 12:42:46 +0000

When things go wrong in a running Java application, often the first sign you will have is lines printed to the screen that look like this:

Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
  at com.myproject.module.MyProject.badMethod(MyProject.java:22)
  at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
  at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
  at com.myproject.module.MyProject.someMethod(MyProject.java:10)
  at com.myproject.module.MyProject.main(MyProject.java:6)

This is a Stacktrace , and in this post I'll explain what they are, how they are made and how to read and understand them. If that looks painful to you then read on...

Anatomy of a Stacktrace

Usually a Stacktrace is shown when an Exception is not handled correctly in code. This may be one of the built-in Exception types, or a custom Exception created by a program or a library.

The Stacktrace contains the Exception’s type and a message , and a list of all the method calls which were in progress when it was thrown.

Let’s dissect that Stacktrace. The first line tells us the details of the Exception:

This is a good start. Line 2 shows what code was running when that happened:

That helps us narrow down the problem, but what part of the code called badMethod? The answer is on the next line down, which can be read in the exact same way. And how did we get there? Look on the next line. And so on, until you get to the last line, which is the main method of the application. Reading the Stacktrace from bottom to top you can trace the exact path from the beginning of your code, right to the Exception.

What went wrong?

The thing that causes an Exception is usually an explicit throw statement. Using the file name and line number you can check exactly what code threw the Exception. It will probably look something like this:

  throw new RuntimeException("Something has gone wrong, aborting!");

This is a great place to start looking for the underlying problem: are there any if statements around that? What does that code do? Where does the data used in that method come from?

It is also possible for code to throw an Exception without an explicit throw statement, for example you can get:

NullPointerException if obj is null in code which calls obj.someMethod()
ArithmeticException if a division by zero happens in integer arithmetic, ie 1/0 - curiously there is no Exception if this is a floating-point calculation though, 1.0/0.0 returns infinity just fine!
NullPointerException if a null Integer is unboxed to an int in code like this: Integer a=null; a++;
There are some other examples in the Java Language Specification, so it’s important to be aware that Exceptions can arise without being explicitly thrown.

Dealing with Exceptions Thrown by Libraries

One of the great strengths of Java is the huge number of libraries available. Any popular library will be well tested so generally when faced with an Exception from a library, it’s best to check first whether the error is caused by how our code uses it.

For example, if we're using the Fraction class from Apache Commons Lang and pass it some input like this:

  Fraction.getFraction(numberOfFoos, numberOfBars);

If numberOfBars is zero, then the Stacktrace will be like this:

Exception in thread "main" java.lang.ArithmeticException: The denominator must not be zero
  at org.apache.commons.lang3.math.Fraction.getFraction(Fraction.java:143)   
  at com.project.module.MyProject.anotherMethod(MyProject.java:17)
  at com.project.module.MyProject.someMethod(MyProject.java:13)
  at com.project.module.MyProject.main(MyProject.java:9)

Many good libraries provide Javadoc which includes information about what kinds of Exceptions may be thrown and why. In this case Fraction.getFraction has documented will throw an ArithmeticException if a Fraction has a zero denominator. Here it's also clear from the message, but in more complicated or ambiguous situations the docs can be a great help.

To read this Stacktrace, start at the top with the Exception's type - ArithmeticException and message The denominator must not be zero. This gives an idea of what went wrong, but to discover what code caused the Exception, skip down the Stacktrace looking for something in the package com.myproject (it’s on the 3rd line here), then scan to the end of the line to see where the code is (MyProject.java:17). That line will contain some code that calls Fraction.getFraction. This is the starting point for investigation: What is passed to getFraction? Where did it come from?

In big projects with many libraries, Stacktraces can be hundreds of lines long so if you see a big Stacktrace, practise scanning the list of at ... at ... at ... looking for your own code - it’s a useful skill to develop.

Best Practice: Catching and Rethrowing Exceptions

Let's say we are working on a big project that deals with fictional FooBars, and our code is going to be used by others. We might decide to catch the ArithmeticException from Fraction and re-throw it as something project-specific, which looks like this:

  try {
    ....
    Fraction.getFraction(x,y);
    ....
  } catch ( ArithmeticException e ){ 
    throw new MyProjectFooBarException("The number of FooBars cannot be zero", e);
  }

Catching the ArithmeticException and rethrowing it has a few benefits:

Our users are shielded from having to care about the ArithmeticException - giving us flexibility to change how commons-lang is used.
More context can be added, eg stating that it’s the number of FooBars that is causing the problem.
It can make Stacktraces easier to read, too, as we’ll see below.

It isn't necessary to catch-and-rethrow on every Exception, but where there seems to be a jump in the layers of your code, like calling into a library, it often makes sense.

Notice that the constructor for MyProjectFooBarException takes 2 arguments: a message and the Exception which caused it. Every Exception in Java has a cause field, and when doing a catch-and-rethrow like this then you should always set that to help people debug errors. A Stacktrace might now look something like this:

Exception in thread "main" com.myproject.module.MyProjectFooBarException: The number of FooBars cannot be zero
  at com.myproject.module.MyProject.anotherMethod(MyProject.java:19)
  at com.myproject.module.MyProject.someMethod(MyProject.java:12)
  at com.myproject.module.MyProject.main(MyProject.java:8)
Caused by: java.lang.ArithmeticException: The denominator must not be zero
  at org.apache.commons.lang3.math.Fraction.getFraction(Fraction.java:143)
  at com.myproject.module.MyProject.anotherMethod(MyProject.java:17)
  ... 2 more

The most recently thrown Exception is on the first line, and the location where it was thrown is still on line 2. However, this type of Stacktrace can cause confusion because the catch-and-rethrow has changed the order of method calls compared to the Stacktraces we saw before. The main method is no longer at the bottom, and the code which first threw an Exception is no longer at the top. When you have multiple stages of catch-and-rethrow then it gets bigger but the pattern is the same:

Check the sections from first to last looking for your code, then read relevant sections from bottom to top.

Libraries vs Frameworks

The difference between a Library and a Framework in Java is:

Your code calls methods in a Library
Your code is called by methods in a Framework

A common type of Framework is a web application server, like SparkJava or Spring Boot. Using SparkJava and Commons-Lang with our code we might see a Stacktrace like this:

com.framework.FrameworkException: Error in web request
    at com.framework.ApplicationStarter.lambda$start$0(ApplicationStarter.java:15)
    at spark.RouteImpl$1.handle(RouteImpl.java:72)
    at spark.http.matching.Routes.execute(Routes.java:61)
    at spark.http.matching.MatcherFilter.doFilter(MatcherFilter.java:134)
    at spark.embeddedserver.jetty.JettyHandler.doHandle(JettyHandler.java:50)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1568)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:503)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.project.module.MyProjectFooBarException: The number of FooBars cannot be zero
    at com.project.module.MyProject.anotherMethod(MyProject.java:20)
    at com.project.module.MyProject.someMethod(MyProject.java:12)
    at com.framework.ApplicationStarter.lambda$start$0(ApplicationStarter.java:13)
    ... 16 more
Caused by: java.lang.ArithmeticException: The denominator must not be zero
    at org.apache.commons.lang3.math.Fraction.getFraction(Fraction.java:143)
    at com.project.module.MyProject.anotherMethod(MyProject.java:18)
    ... 18 more

OK that is getting quite long now. As before we should suspect our own code first, but it's getting harder and harder to find where that is. At the top there is the Framework's Exception, at the bottom the Library's and right in the middle is our own code. Phew!

A complex Framework and Library can create a dozen or more Caused by: sections, so a good strategy is to jump down those looking for your own code: Caused by: com.myproject... Then read that section in detail to isolate the problem.

In Summary

Learning how to understand Stacktraces and read them quickly will let you home in on problems and makes debugging much less painful. It's a skill which improves with practise, so next time you see a big Stacktrace don't be intimidated - there is a lot of useful information there if you know how to extract it.

If you have any tips or tricks about dealing with Java Stacktraces I'd love to hear about them so get in touch with me and let's share what we know.

mgilliard@twilio.com
@MaximumGilliard