DEV Community: Sam Rose

A Logical Way to Split Long Lines

Sam Rose — Mon, 27 May 2019 00:00:00 +0000

Splitting long lines is something we do every day as programmers, but rarely do I hear discussion about how best to do it. Considering our industry-wide obsession with “best practices,” line breaks have managed to stay relatively free from scrutiny.

A few years ago, I learned a method for splitting lines that is logical, language-independent and, most importantly, produces good results.

The Rectangle Method

The core principal of this method is to always make sure you can draw a rectangle around an element and all of its children, without having to overlap with any unrelated elements. The outcome is that related things stay closer together, and our eyes rarely have to dart between distant locations.

Confused? So was I. Let’s walk through an example.

JavacParser parser = parserFactory.newParser(javaInput.getText(), /*keepDocComments=*/ true, /*keepEndPos=*/ true, /*keepLineMap=*/ true);

This line is 139 characters long and was taken from the source code of google-java-format. It is composed of a number of elements:

A variable declaration. This encompasses the entire line.
The variable declaration splits in to two halves: the type and name on the left hand side, and the expression on the right hand side.
The expression is a single method call, which could be split in to the receiver, method name, and its arguments.
Lastly, each method argument is its own element. Comments included.

It’s easy to draw a rectangle around this, it’s just one line. But if we say that our maximum allowed line length is 80, this line is a bit too long and needs to be split.

Deciding where to make the first split

What do we mean when we say “an element and all of its children?” Programming language syntax can usually be represented as a tree. Our example would look something like this:

        Variable declaration                                                                                                         
       /                    \                                                                                                        
 Type + name             Expression                                                                                                  
                        /          \                                                                                                 
                   Receiver     Method call                                                                                          
                               /           \                                                                                         
                            Name         Arguments

We want to split such that you can draw a rectangle around each subtree, without touching any other subtree.

A natural place to make this first split would be just after the =:

JavacParser parser =
    parserFactory.newParser(javaInput.getText(), /*keepDocComments=*/ true, /*keepEndPos=*/ true, /*keepLineMap=*/ true);

This passes the rectangle test because we can draw a rectangle around every element and its children without overlapping with unrelated elements:

┌──────────────────────────────────────────────────────┐
│┌──────────────────────┐                              │
││ JavacParser parser = │                              │
│└─┬────────────────────┴─────────────────────────────┐│
│  │ parserFactory.newParser(javaInput.getText(), ... ││
│  └──────────────────────────────────────────────────┘│
└──────────────────────────────────────────────────────┘

One big rectangle around the whole thing, and two smaller rectangles around each the declaration and assignment. Note that no rectangle overlaps any other rectangle.

It’s good progress, but the second line is still 118 characters long so needs splitting again.

Before doing this, I want to show how this would look if we split a little differently:

JavacParser parser = parserFactory.newParser(
  javaInput.getText(), /*keepDocComments=*/ true, /*keepEndPos=*/ true, /*keepLineMap=*/ true);

This doesn’t pass the rectangle test:

                     ┌────────────────────────────────┐     
JavacParser parser = │ parserFactory.newParser(       │
┌────────────────────┘                                │
│ javaInput.getText(), /*keepDocComments=*/ true, ... │
└─────────────────────────────────────────────────────┘

It’s not possible to draw a rectangle around the right hand side of the = and catch all of its children without also catching the left hand side of the =. It’s correct that the rectangle method flags this as a bad split. There’s an awful long way to travel from newParser to its first argument, which might result in your eyes having to dart back and forth more than necessary.

Deciding where to make the second split

There are a couple of ways to make the second split, but the one I would go with is this:

JavacParser parser =
    parserFactory.newParser(
        javaInput.getText(), /*keepDocComments=*/ true, /*keepEndPos=*/ true, /*keepLineMap=*/ true);

Let’s see how that looks with some rectangles around it:

┌────────────────────────────────────────────────────────────┐
│┌──────────────────────┐                                    │
││ JavacParser parser = │                                    │
│└─┬────────────────────┴─────────────────────────────┐      │
│  │ parserFactory.newParser(                         │      │
│  └─┬────────────────────────────────────────────────┴────┐ │
│    │ javaInput.getText(), /*keepDocComments=*/ true, ... │ │
│    └─────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────┘

We’re still good! Note that the above doesn’t draw a rectangle around every element we could, mostly due to space limitations. You can also draw a rectangle from parserFactory.newParser around everything else after it.

Another way of doing the split that would also pass the rectangle test is:

JavacParser parser =
    parserFactory
        .newParser(
            javaInput.getText(), /*keepDocComments=*/ true, /*keepEndPos=*/ true, /*keepLineMap=*/ true);

But splitting at the . feels a little too eager to me. You can use less vertical space and lose no clarity by leaving that line as one.

Sadly our third line is still 94 characters long, and needs to be split yet again.

Deciding where to make the third split

Again, there are multiple routes for this one but I would go for the following:

JavacParser parser =
    parserFactory.newParser(
        javaInput.getText(),
        /*keepDocComments=*/ true,
        /*keepEndPos=*/ true,
        /*keepLineMap=*/ true);

With rectangles:

┌──────────────────────────────────┐
│┌──────────────────────┐          │
││ JavacParser parser = │          │
│└─┬────────────────────┴─────┐    │
│  │ parserFactory.newParser( │    │
│  └─┬────────────────────────┴───┐│
│    │ javaInput.getText(),       ││
│    ├────────────────────────────┤│
│    │ /*keepDocComments=*/ true, ││
│    ├────────────────────────────┤│
│    │ /*keepEndPos=*/ true,      ││
│    ├────────────────────────────┤│
│    │ /*keepLineMap=*/ true);    ││
│    └────────────────────────────┘│
└──────────────────────────────────┘

Again, for space reasons, not all possible rectangles have been drawn.

We could have also had multiple arguments per line:

JavacParser parser =
    parserFactory.newParser(
        javaInput.getText(), /*keepDocComments=*/ true,
        /*keepEndPos=*/ true, /*keepLineMap=*/ true);

This passes the rectangle test and none of the lines go past the 80 character limit. However, I usually avoid this as a matter of personal preference.

Conclusion

Most of my code follows this style, and I feel it’s easier to read as a result. I’m sure this is just one of many approaches, and I would love to hear about them and how they compare to this one!

API Design: In The Wild (part 2)

Sam Rose — Sun, 19 May 2019 00:00:00 +0000

In a previous post we looked at some real-world APIs, highlighting the good and the bad, and in this post we’re going to do the same!

Python’s datetime.datetime
Java’s URL.equals method
Go’s standard library
The word “filter”
Conclusion

Python’s `datetime.datetime`

Most experienced Pythonistas have written something like this at some point in their career:

import datetime
now = datetime.datetime.now()
print(now)

While not incorrect, the repeated naming is jarring.

Go specifically calls this out in a blog post on package naming, and I don’t think I could possibly improve on it so I’ll quote it verbatim:

Avoid stutter. Since client code uses the package name as a prefix when referring to the package contents, the names for those contents need not repeat the package name. The HTTP server provided by the http package is called Server, not HTTPServer. Client code refers to this type as http.Server, so there is no ambiguity.

You can combat the stuttering somewhat using Python’s from x import y syntax:

from datetime import datetime
now = datetime.now()
print(now)

As a fun little extra, a friend sent me the following bit of Ruby:

files = Dir.entries(Dir.pwd)
files.select! { |file| File.file?(file) }

Java’s `URL.equals` method

If you were going to test two URLs for equality, how might you do it?

It’s completely reasonable to do a comparison of the string representations. ”https://google.com” != “https://facebook.com”. This isn’t the road the original authors of Java’s URL class took, though.

import java.net.URL;

public final class Main {
    public static void main(String... args) throws Exception {
        URL a = new URL("https://google.com");
        URL b = new URL("https://google.com");
        System.out.println(a.equals(b));
    }
}

Does this always return true?

Sadly not.

java.net.URL.equals, through a long chain of indirection, ends up calling java.net.InetAddress.getByName for both of the URLs, which performs a DNS lookup in order to check that they resolve to the same IP address. It does have a cache, but if you’re super unlucky and the cache expires between the two lookups, it’s possible for you to get two different IP addresses for the same hostname if that hostname uses DNS round robin, which is common in 2019.

Similar to java.io.File.exists, this method doesn’t quite do what it advertises to do. Because of this, and the fact it makes a blocking network call, it’s flagged by static analysis tools.

Go’s standard library

I hesitated to add this section, as it’s fairly controversial, but I do think there’s a real problem here.

Go’s standard library is missing some key things I’d expect to find in a standard library. The desire to keep the language implementation and usage simple has shifted burden from the language implementor to the language user.

For example, checking that a list has a specific element is not something the language provides for you. If you were to look it up, the attitude you find is that methods like this are trivial to write. While true, I find myself frequently looking up and copy-pasting trivial methods on a weekly basis.

Efforts to add common operations to the standard library are appreciated, but often clunky. Getting an absolute value requires you to cast to and from a float64. Sorting an array requires you to sacrifice type safety and supply your own swapping function. Creating a big int has a method per type that you can create them from, appending the type to the method name.

The addition of generics and method overloading, while complicating the language and its implementation, would make life easier for users of the language.

The word “filter”

How would you define the word “filter” in general use? Most people, myself included, thinking of removing things. Impurities from metal, dirt from gold, water from pasta.

So what does this return?

list(filter(lambda n: n > 5, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))

If you guessed “a list containing all numbers greater than 5”, you would be correct.

I find this confusing. I have to check which way filter works almost every time I use it, even though in the research I did for this post I learned that filter works the same way in every language I checked.

I like the approach Ruby takes to this problem:

list = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
list.select { |n| n > 5 } # [7, 8, 8, 9]
list.reject { |n| n <= 5 } # [0, 1, 2, 3, 4, 5]

select and reject are more immediately obvious, and good general words for these use cases. At least they’re better than filterfalse.

Conclusion

Another post, another set of real-world API examples and suggestions on improving them.

I’d love to hear your thoughts, and if you have any APIs you love or hate I’d love to hear about them as well!

API Design: In The Wild

Sam Rose — Sun, 07 Apr 2019 00:00:00 +0000

We’ve explored some guiding principles in previous posts, but we’re yet to use our new-found skills. Let’s take a break and look at some examples from real-world code you can find in use today, and how we might improve them.

Go’s `math/big` package

Most languages have a library for representing really big numbers, and Go is no exception.

Here’s an example of it in action:

package main

import (
  "fmt"
  "math"
  "math/big"
  "os"
)

func main() {
  r := &big.Float{}
  _, ok := r.SetString("2.5")
  if !ok {
    fmt.Fprintln(os.Stderr, "Could not parse float")
    return
  }

  r2 := &big.Float{}
  r2.Mul(r, r)

  pi := big.NewFloat(math.Pi)

  result := &big.Float{}
  result.Mul(pi, r2)

  fmt.Printf("%v\n", result)
}

This prints out the area of a circle with radius 2.5.

A couple of things about this API bother me.

`SetString` returns a boolean, not an error

It’s conventional in Go to return two values from a function: the result, and an error. The caller then checks first if there’s an error, and if there isn’t they use the result. Go’s math/big returns a result and a boolean.

Returning a boolean instead of an error isn’t entirely without precedent. Indexing in to a map uses this pattern, for example.

m := make(map[string]string)
m["hello"] = "world"
s, ok := m["world"]
// `ok` will be false to represent that the key was not found

Everywhere other than map indexing, I assume an error is returned and spend a few seconds reading a compile-time error before realising my mistake.

In SetString, I’d prefer to see an error with some information about what went wrong in it. The error could say if the string passed in was empty, or not a floating point number, or anything that isn’t “computer says no.”

Confusing use of function receivers

Operations in math/big use the function receiver as the result of the computation. In a.Add(b, c), a is set to the result of b + c. The library justifies this in the documentation:

By always passing in a result value via the receiver, memory use can be much better controlled. Instead of having to allocate new memory for each result, an operation can reuse the space allocated for the result value, and overwrite that value with the new result in the process.

But I would argue that this was a misguided trade-off. It makes a niche use-case easier at the expense of the common use-case. Imagine if the receiver was instead used as an argument, and the return value were the result.

package main

import (
  "fmt"
  "math"
  "math/big"
  "os"
)

func main() {
  r := &big.Float{}
  _, ok := r.SetString("2.5")
  if !ok {
    fmt.Fprintln(os.Stderr, "Could not parse float")
    return
  }

  result := r.Mul(r).Mul(big.NewFloat(math.Pi))
  fmt.Printf("%v\n", result)
}

Less mutation, less visual noise, less confusing use of a function receiver.

If it turns out there’s still a real need to control the number of allocations, that could be a different set of functions on the package. Something like this:

package main

import (
  "fmt"
  "math"
  "math/big"
  "os"
)

func main() {
  result := &big.Float{}
  _, ok := result.SetString("2.5")
  if !ok {
    fmt.Fprintln(os.Stderr, "Could not parse float")
    return
  }

  big.MulFloat(result, result, result)
  big.MulFloat(result, result, big.NewFloat(math.Pi))

  fmt.Printf("%v\n", result)
}

The first argument is the result, and the second and third arguments are the things being operated on. This keeps the common use-case simple while still offering tools for more niche use-cases.

Java’s old `File.exists()` method

This is one of my favourites. What’s wrong with this?

import java.io.File;

public final class Exists {
    public static void main(String... args) {
        if (args.length != 1) {
            System.err.println("usage: java Exists filepath");
            System.exit(1);
        }

        if (new File(args[0]).exists()) {
            System.out.println("File \"" + args[0] + "\" exists!");
        } else {
            System.out.println("File \"" + args[0] + "\" does not exist.");
        }
    }
}

You might guess that it’s something to do with the .exists() method, and you would be right. I mentioned in the title that this is an older Java API, but why was it changed?

.exists() returns a boolean, and throws no exceptions. That last point is key. On Linux, Java issues a stat system call to figure out if a file exists or not. If it doesn’t, stat fails with error code ENOENT. stat can also fail for a variety of other reasons.

Save the above code in a file called Exists.java and run the following commands:

$ javac Exists.java
$ java Exists Exists.java
File "Exists.java" exists!
$ java Exists DoesntExist.java
File "DoesntExist.java" does not exist.

So far, so good. Now let’s use a tool called strace to simulate stat failing.

$ strace -f -qq -P Exists.java -e fault=stat:error=ENOMEM -- java Exists Exists.java
[pid 7199] stat("Exists.java", 0x7f7227c2b780) = -1 ENOMEM (Cannot allocate memory) (INJECTED)
File "Exists.java" does not exist.

We make it seem like out computer has run out of memory, and in this case .exists() erroneously reports that the file does not exist!

This is actually a bug and in the new Files.exists() it is more clear that false doesn’t always mean the file does not exist. There’s also readAttributes(), which will throw an IOException if stat fails, and this is what you should use if you need certainty.

The API design problem here was trying to represent dozens of possible outcomes in a data type that can only represent two. Whenever you’re considering returning a boolean, stop and think if an enum would be better. Though, if you do this, try and avoid the classic tri-state boolean.

As an aside: Java’s Math.abs() function has a similar design problem, points if you can spot it.

Apache’s `EntityUtils.consume()`

It’s not so much EntityUtils.consume() that’s the problem here, it’s the fact that it was considered necessary at all. Let’s look at an example and then talk through what’s wrong.

import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.HttpClients;

public final class App {
  public final static void main(String[] args) throws Exception {
    HttpClient client = HttpClients.createDefault();

    while (true) {
      HttpGet req = new HttpGet("http://example.com/");
      HttpResponse res = client.execute(req);
      System.out.println(res.getStatusLine());
    }
  }
}

If you were to run this, you wouldn’t get more than a couple of successful responses back before the program hangs indefinitely. Why? Because we’ve forgotten to close our HTTP responses, of course. We need to add the following line after our System.out.println() call:

EntityUtils.consume(res.getEntity());

I’ve had to track this one down in production code more than once, and it always gets me thinking: why does this keep happening? Is the language at fault for not giving us better tools to manage closeable resources? Could the library have a better interface? Could it output a helpful message or exception when it detects we’ve made this mistake?

The answer is a mix of these. Java has an interface called Closeable, which is the best way for library authors to signal to a user that a thing needs closing. The thing that needs closing in an HttpResponse is buried inside of it. This means there’s no way for an IDE to statically analyse the code and alert the user to the problem.

The authors of the library noticed this, and this is the code they now recommend you write:

import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;

public final class App {
    public final static void main(String[] args) throws Exception {
        try (CloseableHttpClient client = HttpClients.createDefault()) {
            while (true) {
                HttpGet req = new HttpGet("http://example.com/");
                try (CloseableHttpResponse res = client.execute(req)) {
                    System.out.println(res.getStatusLine());
                }
            }
        }
    }
}

This surfaces the Closeable interface, and now IDEs can alert you when you have forgotten to close the response. This is better, but does nothing to help the poor sod debugging their hanging process. What I’d love to see is a log message that’s output before the program hangs saying that the user may have forgotten to close their responses.

Go handles resource reclamation well with the defer keyword.

package main

import (
  "os"
)

func main() {
  f, err := os.Create("myfile")
  if err != nil {
    panic(err)
  }
  defer f.Close()

  // do other stuff with `f`
}

The execution of f.Close() won’t happen until the end of the function. This feature keeps creation and clean up close to each other, instead of screens apart. It’s not perfect, but it does help.

Assert equal in most languages

Lastly, the classic equality assertion in most language’s unit testing libraries. I’ll pick on Ruby, but this is present in almost all of them:

require "minitest/autorun"

class MyTest < Minitest::Test
  def test_equality
    a = 1
    b = 2
    assert_equal a, b
  end
end

assert_equal takes two parameters here: expected, and actual. But which is which? It’s usually expected and then actual, but there are examples where that isn’t true.

Ruby MiniTest: expected, actual
Kotlin: expected, actual
Rust: left, right
JavaScript chai: actual, expected
Python: first, second
Go: expected, actual

I don’t know about you, but I always have to take a second to remember which way around it should be. “Does it matter, though?” you might ask. Yes, it often does. Each testing library will generate test failure output of the form: “expected X to be equal to Y.” It’s not always obvious which argument is X and which is Y, especially if you’re comparing the output of two function calls.

Can we do better? Yes!

“Fluent” APIs do a really good job here, and is the way a lot of unit testing frameworks are going.

Here’s an example in Ruby using the rspec library:

require 'spec_helper'

describe "a test" do
  it "should be equal" do
    a = 1
    b = 2
    a.should eql(b)
  end
end

And in Java using Google’s Truth library:

int a = 1;
int b = 2;

assertThat(a).isEqualTo(b);
assertThat(a).isLessThan(b);

Fluent APIs read better and make it more obvious that actual always comes first. In API design terms, the interface does work to make it more difficult for the user to make mistakes. This is a good API design principle that we’ll explore in a future post.

Conclusion

Not much to conclude here. I hope you got some insight from looking at real examples, and I hope my suggested improvements make sense. If you don’t agree, or you think things could be even better, I’m extremely excited to hear about it in the comments. :)

Post planning: how do you do it?

Sam Rose — Fri, 05 Apr 2019 17:13:08 +0000

I tend to jot thoughts down in Evernote whenever they come to me. Just bullet points, nothing fancy.

How do you do it? 😀

dev.to Discord server?

Sam Rose — Wed, 03 Apr 2019 21:56:19 +0000

Has this idea been explored? I think it could be good fun. :)

API Design: Optional Parameters

Sam Rose — Tue, 02 Apr 2019 15:54:21 +0000

When we write functions, it's common to want to give the user options to suit a range of use-cases. There are good ways and bad ways of doing it, and this post is going to explore them.

Guiding principles

API design is hard. A good API needs to be:

Easy to change without breaking user code.
Simple for the common use-cases.
Easy to understand without having to read documentation.
Helpful when things go wrong.

We'll use these principles to help us understand good and bad design decisions in our case study.

The case study

Let's imagine we're writing an HTTP client library in Go.

package http

struct Response {
  StatusCode int
  Headers map[string]string
  Body string
}

func Get(url string) (Response, error) {
  // ...
}

The body of the function isn't super important. We're aiming to provide a simple API to the user, and what we've got so far is great but it suffers from a lack of flexibility. We can't set headers, we have no concept of timeouts, it'd be nice to be able to follow redirects easily. In short: we're missing a lot of vital functionality.

Go: adding parameters the wrong way

Let's go about this the naive way.

func Get(url string, followRedirects bool, headers map[string]string) (Response, error) {
  // ...
}

This isn't the most pleasant function to call:

rsp, err := http.Get("http://example.com", false, nil)

If you come across that without knowing the function signature, you'll be wondering what the false and nil represent. If you have to call the function, you'll feel resentful having to pass parameters you don't care about. The common use-case gets more difficult the more parameters you need to pass.

It's a nightmare to change. Adding or removing parameters will break user code. Adding more functions to achieve the same thing but with different sets of parameters will get confusing.

This is, in short, a mess.

Go: adding parameters a good way

What if we put the parameters in to a struct?

struct Options {
  FollowRedirects bool
  Headers map[string]string
}

func Get(url string, options Options) (Response, error) {
  // ...
}

rsp, err := Get("http://example.com", Options{
  FollowRedirects: true,
  Header: map[string]string{
    "Accept-Encoding": "gzip",
  },
})

Pros

Easy to add new parameters.
Easy for a casual reader to know what's going on.
User only has to set the parameters they care about.

Cons

If you don't want to set any parameters, you still have to pass in an empty struct. The common use-case is still more difficult than it needs to be.
If you want to set lots of parameters, it can get unwieldy.
In Go, it's hard to distinguish between something that's unset or has been specifically set to a zero-value. Other languages have the same problem with null values.

Go: adding parameters a better way

This one is a little more complex, but bear with it.

type options struct {
  followRedirects bool
  headers map[string]string
}

type Option = func(*options)

var (
  FollowRedirects = func(o *options) {
    o.followRedirects = true
  }

  Header = func(key, value string) func(*options) {
    return func(o *options) {
      o.headers[key] = value
    }
  }
)

func Get(url string, os ...Option) (Response, error) {
  o := options{}
  for _, option := range os {
    option(&o)
  }

  // ...
}

Some examples of using this:

rsp, err := http.Get("http://example.com")

If you want to specify parameters:

rsp, err := http.Get("http://example.com",
  http.FollowRedirects,
  http.Header("Accept-Encoding", "gzip"))

Pros

Easy to add new parameters.
Easy to deprecate old parameters by outputting a warning when they're used.
Easy for a casual reader to know what's going on.
Functions can do anything, so parameters could go beyond specifying values. For example, you could load configuration from disk.

Cons

If the API author isn't careful, they can create parameters that interfere with each other or do unsafe things like change global state.
A little more complicated for the API author to set up than plain old function arguments.

The pros outweigh the cons. The flexibility and clean API surface will pay for themselves, provided you keep check on what you're doing inside of Option functions.

This is my go-to for optional parameters in languages that don't have first-class support for them.

"But what about other languages?"

You may be thinking: "but Java has method overloading, wouldn't that work perfectly for optional parameters?"

It's a good question, we can get around the need to specify a default Options struct when we don't want to change anything. Let's explore how this might look using method overloading in Java.

The case study in Java

Here's the original case study again, but restated in Java.

public final class Http {
  public static final class Response {
    public final int statusCode;
    public final InputStream body;
    public final Map<String, String> headers;
  }

  public static Response get(String url) throws IOException {
    // ...
  }
}

Calling it would look something like this:

try {
  Response res = Http.get("https://example.com");
} catch (IOException e) {
  // ...
}

Java: the method overloading approach (bad)

Let's now use method overloading to get around having to specify a default Options struct.

public final class Http {
  public static Response get(String url) throws IOException {
    return get(url, null);
  }

  public static Response get(String url, Map<String, String> headers) throws IOException {
    return get(url, headers, false);
  }

  public static Response get(String url, boolean followRedirects, Map<String, String> headers) throws IOException {
    // ...
  }
}

This is commonly referred to as a "telescopic function," because as you add new parameters the whole thing gets longer, like extending a telescope.

Here's an example of using it:

public final class Main {
  public static final void main(String... args) throws IOException {
    Response res;
    res = Http.get("https://example.com");
    res = Http.get("https://example.com", true);
    res = Http.get("https://example.com", true, Map.of("Accept-Encoding", "gzip"));
  }
}

Pros

No need for the user to worry about specifying all parameters. The common use-case is simple.
Easy to add new parameters.

Cons

A lot of boilerplate to set up.
While the common use-case is simple, if you want to tweak one parameter you may have to specify loads of other ones you don't care about.

This isn't an ideal solution. It improves on the previous, but isn't what I would recommend you use in your own code.

Java: the Good Solution

public final class Http {
  private static final class Options {
    boolean followRedirects;
    Map<String, String> headers;

    Options() {
      this.followRedirects = false;
      this.headers = new HashMap<>();
    }
  }

  public static Consumer<Options> followRedirects() {
    return o -> o.followRedirects = true;
  }

  public static Consumer<Options> header(String key, String value) {
    return o -> o.headers.put(key, value);
  }

  public static Response get(String url, Consumer<Options>... os) throws IOException {
    Options options = new Options();
    for (Consumer<Options> o : os) {
      o.accept(options);
    }

    // ...
  }
}

And an example of using this API:

public final class Main {
  public static final void main(String... args) throws IOException {
    Response res;
    res = Http.get("https://example.com");
    res = Http.get("https://example.com", Http.followRedirects());
    res = Http.get("https://example.com", Http.followRedirects(), Http.header("Accept-Encoding", "gzip"));
  }
}

This has all of the pros of when we saw it in Go, and still manages to look nice in Java. It also gets around the problem we saw in the previous section of the user that may only want to specify one parameter.

While the above is nice, it's not what you're going to see in the wild when using Java. It's far more likely you'll see "builders."

Java: the idiomatic solution using builders

public final class HttpClient {
  private final Map<String, String> headers;
  private final boolean followRedirects;

  private HttpClient(Map<String, String> headers, boolean followRedirects) {
    this.headers = headers;
    this.followRedirects = followRedirects;
  }

  public static final class Builder {
    private Map<String, String> headers = new HashMap<>();
    private boolean followRedirects = false;

    private Builder() {}

    public Builder withHeader(String key, String value) {
      this.headers.put(key, value);
      return this;
    }

    public Builder followRedirects() {
      this.followRedirects = true;
      return this;
    }

    public Client build() {
      return new Client(headers, followRedirects);
    }
  }

  public static Builder builder() {
    return new Builder();
  }

  public Response get(String url) throws IOException {
    // ...
  }
}

And an example of using it:

public final class Main {
  public static final void main(String... args) {
    HttpClient client =
      HttpClient.builder()
        .withHeader("Accept-Encoding", "gzip")
        .followRedirects()
        .build();

    Response res = client.get("https://example.com");
  }
}

This might feel like a bit of a departure from passing in optional parameters to a method. Creating new objects in Java is cheap and preferred over long method signatures.

Pros

Can add new parameters without breaking user code.
Easy for a casual reader to know what's going on.
Function completion on the builder tells you what parameters are available.

Cons

Lots and lots of boilerplate for the library author.

This is the way I would recommend you support optional parameters in Java. It may feel like a lot of work, but there are libraries that can help reduce the boilerplate. Also IDEs like IntelliJ have tools to help you generate most of the boring stuff.

Why is this more idiomatic than the other method?

Builders predate lambdas in Java.

"But what about languages that support optional parameters?"

Another great question. In these cases, it makes the most sense to use what the language offers you.

Python

class Http:
  @staticmethod
  def get(url, follow_redirects=False, headers=None):
    pass

res = Http.get("https://example.com")
res = Http.get("https://example.com", follow_redirects=True)
res = Http.get("https://example.com", headers={"Accept-Encoding": "gzip"})

Ruby

class Http
  def self.get(string, follow_redirects: false, headers: nil)
  end
end

res = Http.get("https://example.com")
res = Http.get("https://example.com", headers: {"Accept-Encoding": "gzip"})
res = Http.get("https://example.com", follow_redirects: true, headers: {"Accept-Encoding": "gzip"})

Pros

Easy to add new parameters.
Easy for a casual reader to know what's going on.
It has little to no boilerplate for the API author.

Cons

Some oddities around setting default parameter values that we'll touch on later.

Cautionary words for dynamic languages

Beware **kwargs

Both Ruby and Python have the ability to "glob" their named arguments in to dictionaries:

class Http
  def self.get(string, **kwargs)
    # kwargs[:headers]
    # kwargs[:follow_redirects]
  end
end

res = Http.get("https://example.com")
res = Http.get("https://example.com", headers: {"Accept-Encoding": "gzip"})
res = Http.get("https://example.com", follow_redirects: true, headers: {"Accept-Encoding": "gzip"})

class Http:
  @staticmethod
  def get(url, **kwargs):
    # kwargs["headers"]
    # kwargs["follow_redirects"]
    pass

res = Http.get("https://example.com")
res = Http.get("https://example.com", follow_redirects=True)
res = Http.get("https://example.com", headers={"Accept-Encoding": "gzip"})

I would recommend against using these. Being explicit makes it more clear to the reader what they can pass in, and it also gives you the author an opportunity to set sensible defaults.

Beware mutable default values

You may have wanted to write this a few sections ago:

class Http:
  @staticmethod
  def get(url, follow_redirects=False, headers={}):
    pass

Note the difference in headers={} from above, where we wrote headers=None in Python and headers=nil in Ruby.

The problem with this in Python is that the empty dictionary isn't created every time the method gets called. It's created once when the class is defined, and so is shared between invocations. This isn't true in Ruby.

Here's an example:

class Http:
  @staticmethod
  def get(url, follow_redirects=False, headers={}):
    if "counter" not in headers:
        headers["counter"] = 0
    headers["counter"] += 1
    print(headers)

Http.get("https://example.com")
Http.get("https://example.com", headers={"counter": 100})
Http.get("https://example.com")

This outputs:

{'counter': 1}
{'counter': 101}
{'counter': 2}

Equivalent in Ruby:

class Http
  def self.get(url, follow_redirects: false, headers: {})
    headers[:counter] ||= 0
    headers[:counter] += 1
    puts headers
  end
end

Http.get("https://example.com")
Http.get("https://example.com", headers: { counter: 100 })
Http.get("https://example.com")

Outputs:

{:counter=>1}
{:counter=>101}
{:counter=>1}

Even though the problem isn't present in Ruby, I like to avoid it regardless.

Conclusion

I hope this post has given you some food for thought, and shown you some nice techniques you hadn't considered and why they're nice.

If you know other methods to achieve this goal that I haven't explored here, I'd love to read about it.

API Design: Errors

Sam Rose — Tue, 02 Apr 2019 00:00:00 +0000

Errors are one of the easiest things to overlook when creating an API. Your users will have problems from time to time, and an error is the first thing they’re going to see when they do. It’s worth spending time on them to make using your API a more pleasant experience.

Guiding Principles

A good error message should do the following:

Explain what went wrong.
Explain what you can do about it.
Be easy to isolate and handle if it’s a recoverable error.

Case study

We’re going to re-use our HTTP client case study from the previous post, the API surface of which looks like this:

package http

import (
  "io"
)

type Response struct {
  StatusCode int
  Headers map[string]string
  Body io.ReadCloser
}

type options struct {}
type Option = func(*options)

var (
  FollowRedirects = func(o *options) {}
  Header = func(key, value string) func(*options) {}
)

func Get(url string, options ...Option) (Response, error) {}

And here’s a realistic example of what calling it would look like:

package main

import (
  "fmt"
  "http"
  "io"
  "os"
)

func main() {
  res, err := http.Get("https://example.com", http.FollowRedirects, http.Header("Accept-Encoding", "gzip"))
  if err != nil {
    fmt.Fprintln(os.Stderr, err.Error())
    os.Exit(1)
  }

  if res.StatusCode != 200 {
    fmt.Fprintf(os.Stderr, "non-200 status code: %v\n", res.StatusCode)
    os.Exit(1)
  }

  _, err := io.Copy(os.Stdout, res.Body)
  if err != nil {
    fmt.Fprintln(os.Stderr, err.Error())
    os.Exit(1)
  }

  if err := res.Body.Close(); err != nil {
    fmt.Fprintln(os.Stderr, err.Error())
    os.Exit(1)
    }
}

I want to make clear that choosing Go for this is irrelevant. The principles we’re going to talk about apply to most languages.

Helpful error messages

One of the first distinctions you need to make when returning an error is whether the caller can do anything about it.

Take network errors as an example. The following are all part of normal operation for network-connected programs:

The destination process has crashed and is starting back up.
A node between you and the destination has gone bad and isn’t forwarding any packets.
The destination process is overloaded and is rate limiting clients to aid recovery.

Here’s what I would like to see when one of the above happens:

HTTP GET request to https://example.com failed with error: connection refused, ECONNREFUSED. Run man 2 connect for more information. You can pass http.NumRetries(int) as an option, but keep in mind that retrying too much can get you rate limited or blacklisted from some sites.

This saves me some digging online, a trip to the documentation, and a slap on the wrist by an angry webmaster or automated rate limiter. It hits points 1 and 2 from our guiding principles really well.

This isn’t what you would want to show to the end-user, though. We’ll need to give the API user the tools to either handle the error (like offering them the option to retry), or show the end-user a nice error message.

Different types of error

Different languages have different best practices for separating out types of error. We’ll look at Go, Java, Ruby and Python.

Go

The idiomatic way of doing this in Go is to export functions in your API that can check properties of the error thrown.

package http

type httpError struct {
  retryable bool
}

func IsRetryable(err error) bool {
  httpError, ok := err.(httpError)
  return ok && httpError.retryable
}

And using it:

package main

func main() {
  res, err := http.Get("https://example.com")
  if err != nil {
    if http.IsRetryable(err) {
      // retry
    } else {
      // bail
    }
  }

This idea extends to any property the error might have. Anything you think the API user might want to make a decision about should be exposed in this way.

Java

The mechanism for doing this in Java is a little more clear: custom exception types.

public class HttpException extends Exception {
  private final boolean retryable;

  private HttpException(String msg, Throwable cause, boolean retryable) {
    super(msg, cause);
    this.retryable = retryable;
  }

  public boolean isRetryable() {
    return this.retryable;
  }
}

And using it:

public final class Main {
  public static void main(String... args) {
    Response res;
    try {
      res = Http.get("https://example.com");
    } catch (HttpException e) {
      if (e.isRetryable()) {
        // retry
      } else {
        // bail
      }
    }
  }
}

Python

The story is similar in Python.

class Error(Exception):
  pass

class HttpError(Error):
  def __init__ (self, message, retryable):
    self.message = message
    self.retryable = retryable

  def is_retryable(self):
    return self.retryable

And using it:

try:
  res = Http.get("https://example.com")
except HttpError as err:
  if err.is_retryable():
    # retry
  else:
    # bail

Writing a generic Error class that extends from Exception is common practice when writing Python libraries. It allows users of your API to write catch-all error handling should they wish.

Ruby

And again in Ruby.

class HttpError < StandardError
  def initialize message, retryable
    @retryable = retryable
    super(message)
  end

  def is_retryable
    @retryable
  end
end

And using it:

begin
  res = Http.get("https://example.com")
rescue HttpError => e
  if e.is_retryable
    # retry
  else
    # bail

Pretty much identical to Python.

Conclusion

Don’t neglect your error messages. They’re often the first contact users have with your writing, and if it sucks they’re going to get frustrated.

You want to give users of your code the flexibility to handle errors in whatever way makes the most sense to them. Give them as much information about the situation as you can using the methods above.

I wanted to, but didn’t, touch on Rust and functional languages. Their methods of error handling are significantly different to the above. If you know of good patterns in other languages, I’ve love to hear about them in the comments.

Get the number of days between two dates in Go

Sam Rose — Fri, 29 Mar 2019 00:49:30 +0000

Recently I needed to find the number of days between two dates in Go and kept coming across implementations that didn't quite suit my needs.

So I rolled my own, and I'm pretty proud of how robust it is! :)

package main

import (
    "fmt"
    "time"
)

func main() {
    fmt.Println(daysBetween(date("2012-01-01"), date("2012-01-07"))) // 6
    fmt.Println(daysBetween(date("2016-01-01"), date("2017-01-01"))) // 366 because leap year
    fmt.Println(daysBetween(date("2017-01-01"), date("2018-01-01"))) // 365
    fmt.Println(daysBetween(date("2016-01-01"), date("2016-01-01"))) // 0
}

func daysBetween(a, b time.Time) int {
    if a.After(b) {
        a, b = b, a
    }

    days := -a.YearDay()
    for year := a.Year(); year < b.Year(); year++ {
        days += time.Date(year, time.December, 31, 0, 0, 0, 0, time.UTC).YearDay()
    }
    days += b.YearDay()

    return days
}

func date(s string) time.Time {
    d, _ := time.Parse("2006-01-02", s)
    return d
}

What's your favourite library to use and why?

Sam Rose — Fri, 29 Mar 2019 00:31:41 +0000

Looking for examples of libraries that do what they do really well and are a delight to use. Not restricted to any specific language. :)

The dos and don’ts of large, online communities

Sam Rose — Thu, 28 Feb 2019 00:00:00 +0000

One of the ways I spend my spare time is moderating the Programming Discussions Discord server. It’s an online community of almost 20,000 people, focussed around helping people with their programming problems through real-time chat.

A frequent mistake I see people make in the server is to treat it like a small group. How you should behave in front of tens of thousands of people you don’t know is very different to how you behave in front of a small number of people you do know.

Here are some do’s and don’ts of interacting with thousands of people.

Don’t ping everyone

Online group chat tools often have a way to notify individuals or groups that you have messaged them. Both Slack and Discord allow you to type @everyone to notify everybody in the server that you have sent a message. This is useful when you’re letting your team know you’re heading out for lunch, but when the audience is thousands of people you’re going to cause annoyance.

We find people typically do this to bring attention to their question, and that’s totally understandable. It will work, too, but we’ve disabled the feature in Programming Discussions. Hang tight, re-post your question if it has been a few hours and it has scrolled off the screen.

Do ask your whole question

Often phrased as “don’t ask to ask” or “no hello”, this is polite in real-life but doesn’t translate well to large online communities.

In real-life it would be strange if you blurted out a question about the possible modifiers you can prefix a Java method with, but online this is what we prefer.

Bad:

bob : does anyone here know about Java methods?

Good:

bob : I’m struggling to understand why you would make a Java method final. At first I thought it had something to do with the return value being constant, but that doesn’t seem to be the case. What use-case do final methods have in Java?

In the first example, the reader will be hesitant to say yes for fear that they don’t know the answer, or the question ends up taking a long time. It’s also possible for hours to pass between asking and answering, by which time bob could be asleep. The next day he’ll check back and see: “yes, what’s the problem?” and the cycle continues¹.

In the second example, anyone that knows the answer can contribute. Even if bob leaves and comes back the next day, the answer will be there.

Do ask your question well

A good question has the following components:

What you’re trying to achieve.
What you’ve done.
What you expected to happen.
What actually happened.

Including relevant code and error messages if there are any.

When it comes to posting code, dumping a multi-hundred line script into the chat isn’t going to make you any friends. It wastes vertical space and often only a dozen lines of it will be relevant. Spend time coming up with a small, self-contained, runnable example.

It also helps to have put some effort into solving your problem on your own before asking for help. If you fix it and it makes sense, you’ve learnt something. If you fix it and it doesn’t make sense, you have a great question to ask. If you can’t fix it, well, at least you tried. People respect the effort. They don’t respect people wanting to be spoon-fed.

Do be mindful of others

Behind your screen are thousands of people, collectively tens of thousands of years of life experience. You don’t know everyone, so you need to be mindful what you say.

The following are all bad form:

bob : I can’t believe I didn’t understand final in Java, I’m such a retard.

alice : If I miss another semi-colon, I’m going to kill myself.

jason : JavaScript is cancer.

This happens a lot. It can be easy to forget that usernames are people, and everyone is different. While none of the above sentences may bother one set of people, they could all cause real discomfort in others.

Consider instead:

Gratitude over self-deprecation.

bob : final makes so much more sense now! Thanks for taking the time to explain it to me.

Seeking help over getting frustrated.

alice : How do you all make a habit of remembering to end lines with semi-colons? I keep forgetting them.

Discussion over fire.

jason : I find JavaScript’s type coercion causes more harm than good. Does anyone else feel the same way or am I misunderstanding something?

Don’t use enter as punctuation

sandra : what would happen

sandra : if I used modulo

sandra : on

sandra : two negative numbers

sandra : ?

This pushes others’ messages off the screen faster than if the entire question were one message. It’s not that bad in small groups, but in a large community this can be frustrating to those trying to keep their questions “above the fold.”

Train yourself to hit space instead of enter, then read what you’ve written before hitting enter for good measure. A good 50% of what I type has mistakes first time around, and I bet I’m not the only one.

Don’t private message people

If you’re giving an answer, try not to disappear into private messages. A lot of people will first try searching their question before asking it, and if the answer is hiding in a private conversation they’ll never find it.

We also worry about shady things happening behind closed doors. On more than one occasion, we’ve had reports of abuse or illegal activity happening in private messages. As moderators, the more that happens behind closed doors, the more difficult our job becomes.

If you do need to ask a question privately for whatever reason, seek permission first. It could be that the person you thought would have the answer doesn’t, or they’re not at their keyboard. Asking to private message in the open gives another person a chance to offer to help.

Do say thank you

Lastly, it’s great to show gratitude to the people that help you. It doesn’t matter if it’s a little “thank you” or an Oscar acceptance speech. This is what makes the whole thing feel worthwhile to those who give their evenings and weekends to being a part of the community.

Conclusion

The points again:

Don’t ping everyone.
Do ask your whole question.
Do ask your question well.
Do be mindful of others.
Don’t use enter as punctuation.
Don’t private message people.
Do say thank you.

If you follow this advice, at least in the servers I’m a part of, you’ll have a much easier time getting help and will make a lot more friends along the way.

final in Java is to stop inheriting classes overriding a method. Any method marked final will generate a compile-time error if a child attempts to override it. ↩ ↩

The Birth and Death of a Running Program

Sam Rose — Mon, 14 Jan 2019 14:06:53 +0000

This is a cross-post from http://samwho.co.uk/blog/2013/04/13/the-birth-and-death-of-a-running-program/ and was originally written years ago. I wanted to import it from my site's RSS feed but dev.to only appears to scrape the last 10 posts.

I've been on a quest over the last year or so to understand fully how a program ends up going from your brain into code, from code into an executable and from an executable into an executing program on your processor. I like the point I've got to in this pursuit, so I'm going to brain dump here :)

Prerequisite Knowledge: Some knowledge of assembler will help. Some knowledge of processors will also help. I wouldn't call either of these necessary, though, I'll try my best to explain what needs explaining. What you will need, though, is a toolchain. If you're on Ubuntu, hopefully this article will help. If you're on another system, Google for "[your os] build essentials", e.g. "arch linux build essentials".

The Birth of a Program

You have an idea for a program. It's the best program idea you've ever had so you quickly prototype something in C:

#include <stdio.h>

int main(int argc, char* argv[]) {
    printf("Hello, world!\n");
    return 0;
}

A work of genius. You quickly compile and run it to make sure all is good:

$ gcc hello.c -o hello
$ ./hello
Hello, world!

Boom!

But wait... What has happened? How has it gone from being quite an understandable high level program into being something that your processor can understand and run. Let's go through what's happening step by step.

GCC is doing a tonne of things behind the scenes in the gcc hello.c -o hello command. It is compiling your C code into assembly, optimising lots in the process, then it is creating "object files" out of your assembly (usually in a format called ELF on Linux platforms), then it is linking those object files together into an executable file (again, executable ELF format). At this point we have the hello executable and it is in a well-known format with lots of cross-machine considerations baked in.

After we run the executable, the "loader" comes into play. The loader figures out where in memory to put your code, it figures out whether it needs to mess about with any of the pointers in the file, it figures out of the file needs any dynamic libraries linked to it at runtime and all sorts of mental shit like that. Don't worry if none of this makes sense, we're going to go into it in good time.

Compiling from C to assembly

This is a difficult bit of the process and it's why compilers used to cost you an arm and a leg before Stallman came along with the Gnu Compiler Collection (GCC). Commercial compilers do still exist but the free world has standardised on GCC or LLVM, it seems. I won't go into a discussion as to which is better because I honestly don't know enough to comment :)

If you want to see the assembly output of the hello.c program, you can run the following command:

$ gcc -S hello.c

This command will create a file called hello.s, which contains assembly code. If you've never worked with assembly code before, this step is going to be a bit of an eye opener. The file generated will be long, difficult to read and probably different to mine depending on your platform.

Now is not the time or place to teach assembly. If you want to learn, this book is a brilliant place to start. I will, however, point out a little bit of
weirdness in the file. Do you see stuff like this?

EH_frame0:
Lsection_eh_frame:
Leh_frame_common:
Lset0 = Leh_frame_common_end-Leh_frame_common_begin
    .long   Lset0
Leh_frame_common_begin:
    .long   0
    .byte   1
    .asciz   "zR"
    .byte   1
    .byte   120
    .byte   16
    .byte   1
    .byte   16
    .byte   12
    .byte   7
    .byte   8
    .byte   144
    .byte   1
    .align  3

I was initially curious as to what this was as well, so I checked out stack overflow and came across a really great explanation of what this bit means, which you can read here.

Also, notice the following:

callq   _puts

The assembly program is calling puts instead of printf. This is an example of the kind of optimisation GCC will do for you, even on the default level of "no optimisation" (-O0 flag on the command line). printf is a really heavy function, due to having to deal with a large range of format codes. puts is far less heavy. I could only find the NetBSD version of it. puts itself is very small and it delegates to __sfvwrite, the code of which is here. If you want more information on how GCC will optimise printf, this is a great article.

Also, if assembler is a bit new to you, a few things to note is that this post is using GAS (Gnu Assembler) syntax. There are different assemblers out there, a lot of people like the Netwide Assembler (NASM) which has a more human friendly syntax.

GAS suffixes its commands with a letter that describes what "word size" we're dealing with. Above, you'll see we used callq. The q stands for "quad", which is a 64bit value. Here are other suffixes you may run in to:

b = byte (8 bit)
s = short (16 bit integer) or single (32-bit floating point)
w = word (16 bit)
l = long (32 bit integer or 64-bit floating point)
q = quad (64 bit)
t = ten bytes (80-bit floating point)

Assembling into machine code

By comparison, turning assembly instructions into machine code is pretty simple. Compiling is a much more difficult step than assembling is. Assembly instructions are often a 1 to 1 mapping into machine code.

At the end of the assembling stage, you would expect to have a file that just contained binary instructions right? Sadly that's not quite the case. The processor needs to know a lot more about your code than just the instructions. To facilitate passing this required meta-information there are a variety of binary file formats. A very common one in *nix systems is ELF: executable linkable format.

Your program will be broken up into lots of sections. For example, a section called .text contains your program code. A section called .bss stores statically initialised variables (globals, essentially), that are not given a starting value, thus get zeroed. A section called .strtab contains a list of all of the strings you plan on using in your program. If you statically initialise a string anywhere, it'll go into the .strtab section. In our hello.c example, the string "Hello, world!\n" will go into the .strtab.

This article, from issue 13 of Linux Journal in 1995, gives a really good overview of the ELF format from one of the people who created it. It's quite in depth and I didn't understand everything he said (still not sure on relocations), but it's very interesting to see the motivations behind the format.

Linking into an executable

Coming back from the previous tangent, let's think about linking. When you compile multiple files, the .c files get compiled into .o files. When I first started doing C code, one thing that continuously baffled me was how a .c file referenced a function in another .c file. You only reference .h files in a .c file, so how did it know what code to run?

The way it works is by creating a symbol table. There are a multitude of types of symbols in an executable file, but the general gist is that a symbol is a named reference to something. The nm utility allows you to inspect an executable file's symbol table. Here's some example output:

$ nm hello
0000000100001048 B _NXArgc
0000000100001050 B _NXArgv
0000000100001060 B ___progname
0000000100000000 A __mh_execute_header
0000000100001058 B _environ
                 U _exit
0000000100000ef0 T _main
                 U _puts
0000000100001000 d _pvars
                 U dyld_stub_binder
0000000100000eb0 T start

Look at the symbols labelled with the letter U. We have _exit, _puts and dyld_stub_binder. The _exit symbol is operating system specific and will be the routine that knows how to return control back to the OS once your program has finished, the _puts symbol is very important for our program and exists in whatever libc we have, and dyld_stub_binder is an entry point for resolving dynamic loads. All of these symbols are "unresolved", which means if you try and run the program and no suitable match is found for them, your program will fail.

So when you create an object file, the reason you include the header is because everything in that header file will become an unresolved symbol. The process of linking multiple object files together will do the job of finding the appropriate function that matches your symbol and link them together for the final executable created.

To demonstrate this, consider the following C file:

#include <stdio.h>

extern void test(void);

int main(int argc, char* argv[]) {
    printf("Hello, world!\n");
    return 0;
}

Compiling this file into an object file and then inspecting the contents will show you the following:

$ gcc -c hello.c
$ nm hello.o
0000000000000050 r EH_frame0
000000000000003b r L_.str
0000000000000000 T _main
0000000000000068 R _main.eh
                 U _puts
                 U _test

We now have an unresolved symbol called _test! The linker will expect to find that somewhere else and, if it does not, will throw a bit of a hissy fit. Trying to link this file on its own complains about 2 unresolved symbols, _test and _puts. Linking it against libc complains about one unresolved symbol, _test.

Unfortunately, because we don't actually have a definition for test() we can't use it. This may sound confusing, seeing as we defer the linking of puts() until runtime. Why can't we just do the same with test()? Build an executable file and let the loader/linker try and figure it out at runtime?

In the linking process you need to specify where the linker will be able to find things on the target system. Let's step through the original hello.c example, doing each of the compilation steps ourself:

$ gcc -c hello.c

This creates hello.o with an unresolved _puts symbol.

$ ld hello.o

This craps out. We need to give it more information. At this point I'm going to mention that I'm on a Mac system and am about to reference libraries that have different names on a Linux system. As a general rule here, you can replace the .dylib extension with .so:

$ ld hello.o /usr/lib/libc.dylib

This still craps out. Check out this error message:

ld: entry point (start) undefined.  Usually in crt1.o for inferred
architecture x86_64

What the hell? This is a really good error to come across and learn about, though. It leads us nicely into the next section.

Running the program

Wait, didn't we finish the last section with an object file that wouldn't link for some arcane reason? Yes, we did. But getting to a point where we can successfully link it requires us to know a little bit more about how our program starts running when it's loaded into memory.

Before every program starts, the operating system needs to set things up for it. Things such as a stack, a heap, a set of page tables for accessing virtual memory and so on. We need to "bootstrap" our process and set up a good environment for it to run in. This setup is usually done in a file called crt0.o.

When you started learning programming and you used a language that got compiled, one of the first things you learned was that your program's entry point is main() right? The true story is that your program doesn't start in main, it starts in start. This detail is abstracted away from you by the OS and the toolchain, though, in the form of the crt0.o file.

The osdev wiki shows a great example of a simple crt0.o file that I'll copy here:

.section .text

.global _start
_start:
    # Set up end of the stack frame linked list.
    movq $0, %rbp
    pushq %rbp
    movq %rsp, %rbp

    # We need those in a moment when we call main.
    pushq %rsi
    pushq %rdi

    # Prepare signals, memory allocation, stdio and such.
    call initialize_standard_library

    # Run the global constructors.
    call _init

    # Restore argc and argv.
    popq %rdi
    popq %rsi

    # Run main
    call main

    # Terminate the process with the exit code.
    movl %eax, %edi
    call exit

07/08/2013 UPDATE: In a previous version of this post I got this bit totally wrong, confusing the 32bit x86 calling convention with the x86-64 calling convention. Thanks to Craig in the comments for pointing it out :) The below should now be correct.

The line that's probably most interesting there is where main is called. This is the entry point into your code. Before it happens, there is a lot of setup. Also notice that argc and argv handling is done in this file, but it assumes that the loader has pushed the values into registers beforehand.

Why, you might ask, do argc and argv live in %rsi and %rdi before being passed to your main function? Why are those registers so special?

The reason is something called a "calling convention". This convention details how arguments should be passed to a function call before it happens. The calling convention in x86-64 C is a little bit tricky but the explanation (taken from here) is as follows:

Once arguments are classified, the registers get assigned (in left-to-right order) for passing as follows:

If the class is MEMORY, pass the argument on the stack.

If the class is INTEGER, the next available register of the sequence %rdi, %rsi, %rdx, %rcx, %r8 and %r9 is used

For example, take this C code:

void add(int a, int b) {
    return a + b;
}

int main(int argc, char* argv[]) {
    add(1, 12);

    return 0;
}

The assembler that would call that function goes something like this:

movq $1,  %rdi
movq $12, %rsi
call add

The $12 and $1 there are the literal, decimal values being passed to the function. Easy peasy :) The convention isn't something that needs to be followed in your own assembly code. You're free to put arguments wherever you want, but if you want to interact with existing library functions then you need to do as the Romans do.

With all of this said and done, how do we correctly link and run our hello.o file? Like so:

$ ld hello.o /usr/lib/libc.dylib /usr/lib/crt1.o -o hello
$ ./hello
Hello, world!

Hey! I thought you said it was crt0.o? It can be... crt1.o is a file with exactly the same purpose but it has more in it. crt0.o didn't exist on my system, only crt1.o did. I guess it's an OS decision. Here's a short mailing list post that talks about it.

Interestingly, inspecting the symbol table of the executable we just linked together shows this:

$ nm hello
0000000000002058 B _NXArgc
0000000000002060 B _NXArgv
                 U ___keymgr_dwarf2_register_sections
0000000000002070 B ___progname
                 U __cthread_init_routine
0000000000001eb0 T __dyld_func_lookup
0000000000001000 A __mh_execute_header
0000000000001d9a T __start
                 U _atexit
0000000000002068 B _environ
                 U _errno
                 U _exit
                 U _mach_init_routine
0000000000001d40 T _main
                 U _puts
                 U dyld_stub_binder
0000000000001e9c T dyld_stub_binding_helper
0000000000001d78 T start

The reason is that .dylib and .so files (they have the same job, but on Mac they have the .dylib extension and probably a different internal format) are dynamic or "shared" libraries. They will tell the linker that they are to be linked dynamically, at runtime, rather than statically, at compile time. The crt*.o files are normal objects, and link statically which is why the start symbol has an address in the above symbol table.

The Death of a Running Program

You return a number from main() and then your program is done, right? Not quite. There is still a lot of work to be done. For starters, your exit code needs to be propagated up to any parent processes that may be anticipating your death. The exit code tells them something about how your program finished. Exactly what it tells them is entirely up to you, but the standard is that 0 means everything was okay, anything non-zero (up to a max of 255) signifies that an error occurred.

There is also a lot of OS cleanup that happens when your program dies. Things like tidying up file descriptors and deallocating any heap memory you may have forgotten to free() before you returned. You should totally get into the habit of cleaning up yourself, though!

Wrapping up

So that's about the extent of my knowledge on how your code gets turned into a running program. I know I missed some bits out, oversimplified some things and I was probably wrong in places. If you can correct me on any point, or have anything illuminating about how non-x86 or non-ELF systems do the above tasks, I would love to have a discussion about it in the comments :)

Language Interoperability From the Ground Up

Sam Rose — Sun, 29 Apr 2018 00:00:00 +0000

How does a function in Ruby call a function in C? How does a function in Kotlin call a function in Java?

It’s no secret that you can call functions in one programming language from another. Not only is it possible, it can be done in multiple ways. But how does it actually work?

This post is going to be a bottom-up, in-depth look at interoperability. Each section will build upon the previous until we have an understanding of the mechanisms that make language interoperability possible.

Who should read this post?

If you have been programming for a little while and have heard people talking about whether a library has “Java bindings” or “Ruby bindings.”

If you have been using a language like Kotlin for a while and called Java functions from Kotlin and wondered how on earth that works under the hood.

If you have a keen interest in the gory details of complicated systems, but want an explanation that doesn’t assume much and keeps the examples as simple and hands-on as possible.

Additionally, you will need to have the following programs installed on your computer and accessible from the command line if you would like to follow along with the examples: nm, gcc, nasm, gdb, javac javap, kotlinc,kotlin, ruby. All of the examples in this post were written and run on an Arch Linux installation, they’re not likely to work as shown here when run on a Mac or Windows machine.

Why is language interoperability important?

The majority of language interoperability involves higher level languages calling libraries written in lower level languages. A Java library that offers you the ability to call in to the native OpenSSL libraries, for example. Or a Rust library that provides a more idiomatic Rust API in to the cURL library written in C.

Duplication is bad. When you’ve written a complicated library in one language, reimplementing it in another is just one more thing to forget to maintain.

Additionally, it’s a good idea for young languages to be able to make the most of existing work. If you create a new language and include a mechanism for using existing native libraries, you can immediately draw upon decades of hard work.

The benefits of purity

Sometimes you might see a library advertised as a “pure Ruby” implementation, or a “pure Java” implementation of some other library. These libraries will be full reimplementations of their target technology.

For example, this is a pure Java implementation ofLevelDB. This would have been a lot of work for the author, but the advantage is that people will be able to use LevelDB from Java without having to install the native LevelDB library on their system and package that up with their Java code for deployment.

While a pure reimplementation can be a lot of up-front effort and maintenance, they can be easier to use.

Why start at the bottom and work up?

The key to interoperability is finding common ground. With computers, the common ground is a set of standards and conventions in the lower levels of how programs work that allow languages to speak to each other. Understanding these conventions is key to understanding when languages can interoperate, and when they can’t.

Rock bottom: assembly language

mov eax, 10

That there is a line of “assembly code.” It is a single “instruction,” and it consists of a “mnemonic”, mov, and two “operands,” eax and 10.

This is the “move” instruction, and it instructs our CPU to move the value 10in to the register eax. It is part of an instruction set called “x86,” which is used in 32-bit Intel and AMD CPUs¹.

What is a register?

Registers are small but fast bits of storage connected directly to your CPU. If you want to do something to some data, be it addition, subtraction, or some other operation, the data needs to first be loaded in to a register.

If you’re not used to writing code at this level, the concept of registers might seem silly. Why can’t we just store 10 in memory and operate on it there?

Because that isn’t physically possible. The CPU isn’t connected to memory directly. It is connected through a series of caches, and all memory access must go through the Memory Management Unit, or MMU. The MMU can’t process any operations, this happens in the Arithmetic Logic Unit, or ALU², and the ALU is also directly connected to the CPU. It can only get at data if it is in a register.

This is not to scale. In reality, there are around a kilobyte of registers, a few hundred kilobytes of L1 cache, a megabyte or two of L2 cache, low tens of megabytes of L3 cache, and then memory is often tens of gigabytes.

Program flow

Doing mathematical operations is great and all, but for us to write code that does anything we need to be able to compare things and make decisions based on the outcome.

mov     eax, 10
sub     eax, 5
mov     ebx, 5
test    eax, ebx
je      equal

notequal:
jmp     exit

equal:
; do something here

exit:
; end of program

We start this snippet with some moves and a subtract (sub). We introduce a new register, ebx. Then we do a test. The test instruction compares two values, they can either both be registers or one of them could be an “immediate” value, e.g. 10. The result of the test is stored in a special “flags” register that we don’t touch directly. Instead, we use a family of “jump” instructions that read the flags register and decide which instruction to run next.

The je instruction will jump to a point in our code, denoted by the “label”equal, if the result of the test was that both values are equal. Otherwise, the code falls through to the next assembly instruction, which will be whatever we decide to put below our notequal label. For now, we just do a jmp, which is an unconditional jump, to the end of our program at exit.

“Labels” are a new concept in this snippet. They aren’t instructions and the CPU doesn’t attempt to run them. Instead, they’re used to mark points in the code that can be jumped to, which is what we’ve used them for here. You can spot labels by looking for the trailing colon.

Accessing memory

So far so good, but we’re only touching registers at the moment. Eventually we will run out of space in registers and have to fall back to using memory. How does that work in assembly?

mov         eax, 0x7f00000
mov dword [eax], 123

The first bit should be familiar, we’re storing the hexidecimal value 0x7f00000in to eax, but then we do something funky with the next instruction.

The square brackets around [eax] mean that we want to store a value in the memory address inside of eax. The dword keyword signifies that we want to store a 4-byte value (“double word”). We need the dword keyword in there because without it there’s no way to infer how large the value 123 should be in memory.

If you’re familiar with pointers in C, this is essentially how pointers work. You store the memory address you want to point to in a register and then access that memory by wrapping the register name in square brackets.

We can take this a little further:

mov             eax, 0x7f00000
mov dword [eax + 1], 123

This will store the value 123 at address 0x7f00001, one byte higher than before. The value of eax isn’t modified by doing this, it’s just letting us access the value at a register plus an offset. This is commonly seen in real-world assembly code, and we’ll see why later on.

The stack

Fundamentals out of the way, this is our first recognisable concept from higher level languages. The stack is where languages typically store short-lived variables, but how does it work?

When your program starts, the value of a special register called esp is set to a memory location that represents the top of the stack space you are allowed to access, which is itself near the top of your program’s “address space.”

You can examine this phenomenon by writing a simple C program and running it in a debugger:

int main() {
    return 42;
}

Compile with gcc and run using gdb, the “GNU Debugger”:

$ gcc main.c -o main -m32
$ gdb main
gdb> start
gdb> info register esp
esp 0xffffd468  0xffffd468

When we say start in the GDB prompt, we’re asking it to load our program in to memory, start running it, but pause it as soon as it starts. Then we inspect the contents of esp with info register esp.

A brief detour into memory organisation

Why is the stack near the top of the address space? What even is an address space?

When your program gets loaded in to memory on a 32-bit system, it is given its own address space that is 4GB in size: 0x00000000 to 0xffffffff. This happens no matter how much physical RAM your machine has installed in it. Mapping this “virtual” address space is another one of the jobs that the MMU performs, and the details of it are beyond the scope of this post.

This simplified view of memory shows that our program is loaded somewhere fairly low down in the address space, our stack is up near the top and grows down, then we have this mysterious place called the “heap” that lives just above our program. It’s not important for language interoperability, but the heap stores long-lived data, for example things you’ve allocated usingmalloc() in C or new in C++.

This allows for a fairly efficient use of the available space. In reality, the stack is limited in size to a handful of megabytes and exceeding that will cause your program to crash. The heap can grow all the way up to the base of the stack but no further. If it ever attempts to overlap with the stack, yourmalloc() calls will start to fail, which causes many programs to crash.

Back to the stack

The x86 instruction set gives us some instructions for adding and removing values from the stack.

push    1
push    2
push    3
pop     eax
pop     ebx
pop     ecx

At the end of this sequence of instructions, the value of eax will be 3,ebx will be 2 and ecx will be 1. Don’t trust me? We can verify this for ourselves with a couple of small modifications.

global main
main:
        push    1
        push    2
        push    3
        pop     eax
        pop     ebx
        pop     ecx
        ret

Save this in to a file called stack.s and run the following commands:

$ nasm -f elf stack.s
$ gcc -m32 stack.o -o stack

If you don’t have nasm you’ll need to install it. It’s a type of program called an “assembler” and it can take assembly instructions and compile them down to machine code.

We now have an executable file in our working directory called “stack”. It’s a bonafide program you can run like any other program. The only modifications we had to make to it were giving it a main label, and making sure it correctly returns control back to the operating system with the ret instruction. We’ll explore ret in more detail later.

Running this program doesn’t really do anything. It will run, but exit silently³. To see what’s going on inside of it, we will once again need a debugger.

$ gdb stack
gdb> break *&main+9
gdb> run
gdb> info registers eax ebx ecx

The output of this sequence of commands should verify what I said earlier about the contents of those registers. gdb allows us to load up a program, run it until a certain point (break *&main+9 is us telling gdb to stop just before the ret instruction) and then examine the program state.

So what do `push` and `pop` actually do?

The push and pop instructions are shorthand and can be expanded to the following:

; push
sub     esp, 4
mov     [esp], operand

; pop
mov     operand, [esp]
add     esp, 4

All of which should be familiar to you from previous sections, and neatly demonstrates how the stack grown downwards and shrinks upwards as things are added and removed.

Functions in assembly

Language interoperability is, at its most fundamental, the ability to call a function written in one language from a different language. So how do we define and call functions in assembly?

With the knowledge we have so far, you might be tempted to think that function calls look like this:

main:
        jmp myfunction

myfunction:
        ; do things here

A label to denote where your function is in memory, and a jump instruction to call it. This approach has two critical problems: it doesn’t handle passing arguments to or returning values from the function and it doesn’t handle returning control to the caller when your function ends.

We could solve the first problem by putting arguments and return values on the stack:

main:
        push    1
        push    2
        jmp     add
        pop     eax

add:
        pop     eax
        pop     ebx
        add     eax, ebx
        push    eax

This would work really well if only our add function were able to jump back to the caller when it was finished. At the moment, when add ends, the program ends. In an ideal world it would return back to just after the jmp in main.

What if we saved where we were when we called the function and jumped back to that location when the function was finished?

The eip register holds the location of the currently executing instruction. Using this knowledge, we could do this:

main:
        push    1
        push    2
        push    eip
        jmp     add
        pop     eax

add:
        pop     edx ; store the return address for later
        pop     eax ; 2
        pop     ebx ; 1
        add     eax, ebx
        push    eax
        mov     eip, edx

We’re getting there. This approach has a couple of problems, though: we modifyesp a lot more than we really have to and x86 doesn’t let you mov things in to eip.

Here’s what a compiler would actually generate for our example above:

main:
        push    ebp
        mov     ebp, esp
        push    2
        push    1
        call    add
        add     esp, 8
        pop     ebp
        ret

add:
        push    ebp
        mov     ebp, esp
        mov     edx, dword [ebp + 8]
        mov     eax, dword [ebp + 12]
        add     eax, edx
        pop     ebp
        ret

This is a lot to take in, so let’s go through it line by line.

We’re introducing a new special register: ebp. This is the “base pointer” register, and its purpose is to act as a pointer to the top of the stack at the moment a function is called. Every function starts with saving the old value of the base pointer on the stack, and then moving the new top of the stack in toebp.

Next we do the familiar pushing of arguments on to the stack. At least we got that right. Then we use an instruction we haven’t seen before called call. call can be expanded to the following:

; call
push    eip + 2
jmp     operand

With eip + 2 meaning the instruction after the jmp below it. It doesn’t matter what the value is, just think of it as pushing the address of the instruction after the call instruction on to the stack so we can refer to it later.

Then control is passed to add, which follows a similar pattern. The base pointer is saved, the stack pointer becomes the new base pointer, and then we get to see the base pointer in action.

mov     edx, dword [ebp + 8]
mov     eax, dword [ebp + 12]

This code is pulling the two arguments to add off of the stack and in to registers so that we can operate on them. But why are they 8 bytes and 12 bytes away?

Remember we pushed the arguments on to the stack, and then call pushed the address to return to. This means that the first 4 bytes of stack are a return address, then then 8 bytes after that are our arguments. To get to the first argument, you need to move 8 bytes up from the stack pointer (because it grows downwards), and to get to the second argument you need to move 12 bytes up.

This has the added benefit of not requiring us to modify esp with every singlepush and pop instruction. It’s a small saving, but when you consider that this has to happen for every argument to every function called, it adds up.

When we’ve done what we need to do in our add function, we perform the steps we did at the start but in reverse order.

pop     ebp
ret

The ret instruction is special because it allows us to set the value of eip. It pops the return value that call pushed for us and jumps to it, returning control of the program to the calling function.

The same thing happens in main, except there’s a subtle difference. The add esp, 8 is necessary to “free” the arguments to add we pushed on to the stack. If we don’t do this, the pop ebp will not correctly restore the base pointer and we’ll likely try to refer to memory we never intended to, crashing our program.

Lastly, you’ll notice that add doesn’t push its result back on to the stack when it’s done. It leaves the result in eax. This is because it’s conventional for a function’s return value to be stored in eax.

Conventions

We’ve just done the deepest possible dive on how function calls work in x86, now let’s put names to each of the things we have learnt.

Saving the base pointer and moving the stack pointer prior to calling a function is called the function prologue.

Restoring the stack pointing and base pointer after a function call is called the function epilogue.

Those two concepts, along with returning values in eax and storing your function arguments on the stack make up what’s called a calling convention , and calling conventions are part of a larger concept known as an application binary interface , or ABI. Specifically, all of what I have described so far it part of the System V ABI , which is used by almost all Unix and Unix-like operating systems.

Before we start calling functions written in one language from functions written in another, there’s one last thing we need to be aware of.

Object files

When you compile a C program, quite a lot of things happen under the hood. The most important concept to understand for language interoperability is “object files.”

If your program consists of 2 .c files, the first thing a compiler does is compile each of those .c files in to a .o file. There is typically a 1-to-1 mapping between.c files and .o files. The same is true of assembly, or .s files.

Object files, on Linux and lots of other Unix-like operating systems, are in a binary format called the “executable and linkable format,” or ELF for short. If you’re interested in the full range of data found inside of an ELF file, you can use the readelf -a <elffile> command to find out. The output is likely to be quite dizzying, though, and we’re only really interested in one of its features here.

Symbol tables

To explain symbol tables, let’s split out our assembly from earlier in to two.s files:

add.s:

add:
        push    ebp
        mov     ebp, esp
        mov     edx, dword [ebp + 8]
        mov     eax, dword [ebp + 12]
        add     eax, edx
        pop     ebp
        ret

main.s:

main:
    push    ebp
    mov     ebp, esp
    push    2
    push    1
    call    add
    add     esp, 8
    pop     ebp
    ret

Assemble both of the files:

$ nasm -f elf add.s
$ nasm -f elf main.s
main.s:6: error symbol `add' undefined

Whoops. Our assembler isn’t happy about us calling an undefined function. This is sticky, because we want to define that function elsewhere and call it in main.s, but it seems here like the assembler doesn’t allow that.

The problem is that there is no add symbol defined in this file. If we want to tell the assembler that we intend to find this symbol in another file, we need to say so. Add this line to the top of main.s:

extern add

And now it should assemble without complaint. Before we go further, have a look in your working directory. You should have two .o files: main.o and add.o. We can look at the contents of their symbol tables with a tool called nm:

$ nm add.o
00000000 t add

$ nm main.o
         U add
00000000 t main

The first column is the address of the symbol, the second column is the type of the symbol, and the third column is the name of the symbol. Notice that the symbol names match up with our label names. Also note that main.o has anadd symbol of type U. U means “undefined”, which means we need to find a definition for it when we link these object files together later.

Both of our defined functions have a symbol type of t. This means that the symbol points to some code.

To create an executable out of these object files, we run:

$ gcc -m32 main.o add.o -o main

This will shout at you, claiming that it cannot find either main or add. What gives?

Unless we explicitly say so, the symbols in an object file cannot be used by other object files when the compiler links them together. To fix this, we need to add:

global add

To the top of add.s and:

global main

To the top of main.s. This allows the symbols to be linked and the result is that the compiler now takes our object files and creates an executable out of them without complaint.

$ nm main

This will produce a lot of output now, because the compiler has to link in a lot of administrative stuff, like the libc constructor and destructor handlers,__libc_csu_init and __ libc_csu_fini respectively. Don’t worry about them, the important thing is that both main and add are defined and the program runs without complaint.

Calling a C function from C++

Let’s go up a level and look at some C and C++.

main.cpp:

extern int inc(int i);

int main() {
    return inc(2);
}

The first line here is us telling the compiler to expect to find a function calledinc in another file.

inc.c:

int inc(int i) {
    return i + 1;
}

Here’s the set of steps we need to follow to compile them both separately and then link them together:

$ gcc -c main.cpp -o main.o
$ gcc -c inc.c -o inc.o
$ gcc inc.o main.o

Unfortunately, this fails with a seemingly unfathomable error:

main.o: In function `main'
main.cpp:(.text+0xa): undefined reference to `inc(int)'
collect2: error: ld return 1 exit status

But we supplied an object file with a definition of inc(int) in it, we made sure to tell the compiler to expect to find a function called inc(int), why can’t it find it?

Name mangling

Sadly C++ wasn’t able to provide all of the features it wanted to without diverging from the System V ABI a little bit. When you compile a function in C, that function gets given a symbol with the name you gave it so that others can call it by that name.

C++ does not do this by default. As well as the name you give it, C++ also tacks on information about the return type and argument types of that function. If the function is in a class, information about the class is also included. It does this to allow you to overload the name of a function, so you can define multiple variations of a function that takes different arguments. This is called name mangling.

As a result, when we compiled our main.cpp file, it was told to look for a function called _Z3inci instead of plain old inc. Our inc.c file provides a function called inc, and as such the two languages cannot interoperate without a little bit of help.

Fortunately, the problem is easily solved by adding 4 characters to our main.cpp:

extern "C" int inc(int i);

This addition of "C" tells C++ to compile this file in search of a function with plain old C-style calling conventions, and this includes using the plain old C name of inc. Attempting to compile this code should now work as expected.

Calling a C++ function from C

This relationship works similarly in the other direction. If we want to write a function in C++ but expose it in a way that a C program could call it, we would need to use extern "C" on that function:

extern "C" int inc(int i) {
    return i + 1;
}

What about Java?

What we’ve discussed up until now is the fundamental basis for how language interoperability works between native languages, but what about a language that runs on a virtual machine, such as Java?

The process is a little more involved, but doable. The problem arises because Java runs on a thing called the Java Virtual Machine, or JVM, which acts as a layer of indirection between Java code and the machine on which it runs. Because of this, Java cannot link directly to native libraries in the same way that C and C++ can. We have to introduce a layer that translates between the native world and the JVM world.

Fortunately, the people behind Java gave this a lot of thought and they came up with the “Java Native Interface,” or JNI. It’s the accepted way to get Java code to talk to native code, and here’s how it works:

public class Interop {
  static {
    System.loadLibrary("inc");
  }

  public static native int inc(int i);

  public static void main(String... args) {
    System.out.println(inc(2));
  }
}

Notice the use of the native keyword. This tells Java that the implementation for this function is defined in some native code somewhere. TheSystem.loadLibrary("inc") line will search Java’s library path for a library called libinc.so and, when it finds it, we will be able to use the Java function inc to call our native code!

But how do we do that?

Step 1: Generate the JNI header file from our code.

$ javac -h . Interop.java

The -h . tells javac to generate a file called Interop.h in the current directory. This will define the function we have to to implement. The resulting file looks like this:

/* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class Interop */

#ifndef _Included_Interop
#define _Included_Interop
#ifdef __cplusplus
extern "C" {
#endif
/*
 * Class: Interop
 * Method: inc
 * Signature: (I)I
 */
JNIEXPORT jint JNICALL Java_Interop_inc
  (JNIEnv *, jclass, jint);

#ifdef __cplusplus
}
#endif
#endif

This looks a lot scarier than it is. The lines containing _Included_Interopare just “header guards” that make sure we can’t accidentally include this file twice and the __cplusplus bit checks if we’re compiling as C++ code and, if we are, wraps everything in an extern "C" block, which you’ll remember from earlier in this post.

The rest is the definition of the JNI function we have to implement:

JNIEXPORT jint JNICALL Java_Interop_inc
  (JNIEnv *, jclass, jint);

It might not look like one, but this is indeed a function declaration. We implement it like so:

inc-jni.c:

#include <jni.h>
#include "Interop.h"

JNIEXPORT jint JNICALL Java_Interop_inc
  (JNIEnv* env, jclass class, jint i) {
    return (jint)(i + 1);
}

The jint cast is to make sure our integer is the size that Java is expecting it to be. The jni.h include is required for all of the Java-specific things we’re seeing, such as jint and jclass.

To compile this we need to execute a pretty gnarly gcc call:

$ gcc -I"$JAVA_HOME\include" -I"$JAVA_HOME\include\linux" -fPIC -shared -o libinc.so inc-jni.c

The -I flags tell gcc where to find header files, which we need for the#include <jni.h> lines to work. The -fPIC -shared flags create a special type of object file called a “shared object.” What’s special about this type of object file is that it’s possible to, instead of compiling directly against it, load it in to your process at runtime. Shared object files are, by convention, named lib<something>.so.

Now we can run the Java code like so:

$ java -Djava.library.path=. Interop
3

Voila! We successfully called our native inc implementation from Java! How cool is that?

What about Ruby?

Ruby has a similar approach to Java. It exposes an API called “Ruby native extensions” that you can hook in to to call native functions in Ruby. Given that we have explored Java’s way of doing this, and Ruby’s is not too dissimilar, I want to use Ruby to focus on a different and more convenient way of calling native code.

ffi

ffi stands for “foreign function interface,” and it’s a Ruby gem that allows us to call functions in existing shared object files with very little setup. First, we install the ffi gem:

$ gem install ffi

Then we need to compile our original inc.c file in to a shared object:

$ gcc -fPIC -shared -o libinc.so inc.c

And then we can write the following short snippet of Ruby code and call ourinc function:

require 'ffi'

module Native
  extend FFI::Library
  ffi_lib './libinc.so'
  attach_function :inc, [:int], :int
end

puts Native.inc(2)

This method makes us jump through far fewer hoops than Java’s JNI does, and has the benefit of allowing us to call existing shared objects without modifying them. For example, you can call directly in to libc.so:

require 'ffi'

module Libc
  extend FFI::Library
  ffi_lib 'c'
  attach_function :puts, [:string], :int
end

Libc.puts "Hello, libc!"

Notice that we only had to specify 'c' as the library name. This is because Unix-like systems have standardised paths to find libraries in, often including/lib and /usr/lib. By default ffi will look for libraries with the naming scheme lib<name>.so. If you do ls /usr/lib/libc.so you should find a file exists at that path.

Why doesn’t Java have FFI?

It does! There’s a library called JNA that does the same job that Ruby’s FFI library does.

import com.sun.jna.Library;
import com.sun.jna.Native;

class JNA {
  public interface Libc extends Library {
    Libc INSTANCE = (Libc)Native.loadLibrary("c", Libc.class);
    public int puts(String s);
  }

  public static void main(String... args) {
    Libc.INSTANCE.puts("Hello, libc!");
  }
}

We have to download the JNA library, which you can do so from here. Then we compile and run the Java code including the JNA library:

$ javac -classpath jna-4.1.0.jar JNA.java
$ java -classpath jna-4.1.0.jar:. JNA
Hello, libc!

Why would anyone use a language’s native interface if FFI is so much more convenient?

Lots of languages have some form of FFI library, and they’re very convenient for calling in to existing native libraries. The problem, though, is that it’s one-way communication. The library you’re calling in to can’t modify, for example, a Java object directly. It can’t call Java code. The only way to do that is to use Java’s JNI or Ruby’s native extensions, because they expose to you an API for doing that.

If you don’t need to have two-way communication between languages, though, and all you want to do is call existing native code, FFI is the way to go.

Hold up, so how does Kotlin call functions written in Java?

Kotlin is a relatively new language that runs on the JVM.

“Wait, doesn’t Java run on the JVM?”

That’s true! But the JVM isn’t just for Java, it’s a generic virtual machine that compilers can generate “machine code” for just like native machines. The JVM machine code is more commonly referred to as “bytecode,” because every instruction is one byte in length. Yes, this limits the instruction set to 256 instructions. Far fewer than the 2,034 instructions you’ll find in x86.

In this respect, you could consider the JVM to be a “native” machine. That is the abstraction it aims to present to compilers and users. The only difference is that the native machine you’re running code on is being emulated by a piece of software called the JVM.

Let’s look at some example Java and Kotlin code.

Hello.java:

public class Hello {
  public static void world() {
    System.out.println("Hello, world!");
  }
}

Main.kt:

fun main(args: Array<String>) {
  Hello.world()
}

Let’s compile and run this code.

$ javac Hello.java
$ kotlinc -classpath . Main.kt
$ kotlin MainKt
Hello, world!

And now we disassemble the code to see what’s going on under the hood:

$ javap -c MainKt.class
Compiled from "Main.kt"
public final class MainKt {
  public static final void main(java.lang.String[]);
    Code:
      0: aload_0
      1: ldc #9 // String args
      3: invokestatic #15 // Method kotlin/jvm/internal/Intrinsics.checkParameterIsNotNull:(Ljava/lang/Object;Ljava/lang/String;)V
      6: invokestatic #21 // Method Hello.world:()V
      9: return
}

This looks a lot different to disassembled native code. The bit underneath the “Code:” heading is the actual bytecode that gets run. The numbers on the left hand side are the byte offset in to the code, the next column is the “opcode” and anything after the opcode are the operands.

“But you said JVM bytecodes were one byte in length, why does the byte index on the left increment more than 1 byte per instruction?”

The opcodes are one byte in length. The operands make up the rest of the space. For example, the invokestatic opcode takes two additional bytes in operands: one byte to reference the class being called and another to reference the method on that class.

“How does the number 21 reference our Hello.world() method?”

A fundamental concept in JVM class files is the “constant pool.” It’s not output by default by javap, but we can ask for it:

$ javap -verbose MainKt.class
Classfile /tmp/MainKt.class
  Last modified May 7, 2018; size 735 bytes
  MD5 checksum f3ce23c2362512e852a5c91a1053c198
  Compiled from "Main.kt"
public final class MainKt
  minor version: 0
  major version: 50
  flags: ACC_PUBLIC, ACC_FINAL, ACC_SUPER
Constant pool:
  ...
  #16 = Utf8 Hello
  #17 = Class #16
  #18 = Utf8 world
  #19 = Utf8 ()V
  #20 = NameAndType #18:#19
  #21 = Methodref #17.#20
  ...

I’ve trimmed a lot of the output, but left the relevant entried in the constant pool. 16, 18 and 19 are the raw UTF-8 strings that give the names “Hello”, “world” and “()V”, which is a way of saying it’s a function that returns void. The invokestatic opcode specifically takes operands that are indexes in to the constant pool, and entries in the constant pool can reference other entries in the constant pool.

If we tried to compile Main.kt on its own, without adding Hello.class in to the classpath, we get an error very similar to the one we got when we tried to link an object files without all of the symbol definitions it required:

$ kotlinc Main.kt
Main.kt:2:2: error: unresolved reference: Hello

So because these languages, Java and Kotlin, both run using the same ABI and on the same instruction set, we are able to use that ABI’s calling conventions to make function calls across language boundaries.

Conclusion

We’ve come a long way. From raw, native assembly code up to calling functions between languages running on the same virtual machine.

With the knowledge you have built up of calling conventions, instruction sets, object files and FFI libraries, you should now be well equipped to explore how languages not mentioned in this post would call functions written in other languages.

Don’t worry too much about these words if they don’t mean anything to you. They’re just terms you may see in the wild and, when you do, you’ll know that the things in this post are what they are referring to. ↩
With exceptions including floating point calculations, which happen on the Floating Point Unit, or FPU. ↩
It’s not 100% silent. Check the exit status of the program when it finishes running. Why do you think it exits with the status it does? ↩

DEV Community: Sam Rose

A Logical Way to Split Long Lines

The Rectangle Method

Deciding where to make the first split

Deciding where to make the second split

Deciding where to make the third split

Conclusion

API Design: In The Wild (part 2)

Python’s datetime.datetime

Java’s URL.equals method

Go’s standard library

The word “filter”

Conclusion

API Design: In The Wild

Go’s math/big package

SetString returns a boolean, not an error

Confusing use of function receivers

Java’s old File.exists() method

Apache’s EntityUtils.consume()

Assert equal in most languages

Conclusion

Post planning: how do you do it?

dev.to Discord server?

API Design: Optional Parameters

Guiding principles

The case study

Go: adding parameters the wrong way

Go: adding parameters a good way

Pros

Cons

Go: adding parameters a better way

Pros

Cons

"But what about other languages?"

The case study in Java

Java: the method overloading approach (bad)

Pros

Cons

Java: the Good Solution

Java: the idiomatic solution using builders

Pros

Cons

Why is this more idiomatic than the other method?

"But what about languages that support optional parameters?"

Python

Ruby

Pros

Cons

Cautionary words for dynamic languages

Beware **kwargs

Beware mutable default values

Conclusion

API Design: Errors

Guiding Principles

Case study

Helpful error messages

Different types of error

Go

Java

Python

Ruby

Conclusion

Get the number of days between two dates in Go

What's your favourite library to use and why?

The dos and don’ts of large, online communities

Don’t ping everyone

Do ask your whole question

Do ask your question well

Do be mindful of others

Don’t use enter as punctuation

Don’t private message people

Do say thank you

Conclusion

The Birth and Death of a Running Program

The Birth of a Program

Compiling from C to assembly

Assembling into machine code

Linking into an executable

Running the program

The Death of a Running Program

Python’s `datetime.datetime`

Java’s `URL.equals` method

Go’s `math/big` package

`SetString` returns a boolean, not an error

Java’s old `File.exists()` method

Apache’s `EntityUtils.consume()`

So what do `push` and `pop` actually do?